Control plane
Liftr receiver, QaaS control plane and Key Vaults in sub-anq-service-prod: the Service layer that creates, updates and deletes customer clusters.
Liftr receiver · last 15 min
OK
414 requests · 1 failed · 0 HTTP 5xx
p50 181 ms · p95 646 ms
QaaS control plane
Degraded
3 operations in progress · 1 stuck
2 failed in the last 24 hours
Key Vault
Healthy
3 vaults · 0 throttled (HTTP 429) in 15 min
Azure Monitor platform metrics
PagerDuty · ANQ SRE Automation
1 open
1 not acknowledged
func-anq-sre-maintenance-prod: 7 failed runs in 24h
Liftr receiver · last 24 hours
Requests
30,966
Success rate
99.75%
Failed
77
HTTP 5xx
17
p50
182 ms
p95
653 ms
Requests
Requests30,966
App Insights requests · 15-minute buckets · UTC
Failures and HTTP 5xx
Failed requests77HTTP 5xx17
Dashed line: alert at 3 failures per 15 minutes; any 5xx also alerts · 15-minute buckets · UTC
Latency p50 / p95
p50181 msp95646 ms
App Insights request duration · latest bucket shown · UTC
Resolved · Oct 5, 05:26 UTC – 05:38 UTC (shaded on the charts): HTTP 503 on create and patch while kv-anq-cp-prod-eus2 throttled secret reads (429). Recovered without intervention.
Cluster operations · last 24 hours
By region
| Region | In progress | Succeeded | Failed |
|---|---|---|---|
| East US 2 | 2 (1 stuck) | 4 | 0 |
| East US | 0 | 1 | 1 |
| South Central US | 0 | 1 | 1 |
| West Europe | 1 | 1 | 0 |
| UK South | 0 | 3 | 0 |
| Central US | 0 | 2 | 0 |
| East Asia | 0 | 2 | 0 |
| Southeast Asia | 0 | 2 | 0 |
| Sweden Central | 0 | 2 | 0 |
| West US 3 | 0 | 2 | 0 |
| Australia Southeast | 0 | 1 | 0 |
| Brazil South | 0 | 1 | 0 |
| Canada Central | 0 | 1 | 0 |
| France Central | 0 | 1 | 0 |
| Germany West Central | 0 | 1 | 0 |
| Japan East | 0 | 1 | 0 |
| North Central US | 0 | 1 | 0 |
| West US 2 | 0 | 1 | 0 |
| All regions | 3 | 28 | 2 |
Recent operations
| Operation | Type | Cluster | Region | Started | Duration | Status |
|---|---|---|---|---|---|---|
| op-cc2c46bf | CreateLiftr | ANQ-0388 | East US 2 | Oct 5, 13:43 UTC | 47 min | In progress for 47 min Stuck: past the 40 min limit for a create |
| op-dd29eb1a | PatchLiftr | ANQ-0217 | East US 2 | Oct 5, 14:29 UTC | 1 min | In progress · 1 min |
| op-abd2fdd0 | UpgradeQaaS | ANQ-0734 | West Europe | Oct 5, 14:19 UTC | 11 min | In progress · 11 min |
| op-54c013c2 | CreateLiftr | ANQ-2209 | Germany West Central | Oct 5, 14:01 UTC | 18 min | Succeeded |
| op-53c0122f | PatchLiftr | ANQ-1956 | West US 2 | Oct 5, 13:59 UTC | 3 min | Succeeded |
| op-2ad08d9f | CreateLiftr | ANQ-2451 | Sweden Central | Oct 5, 12:15 UTC | 17 min | Succeeded |
| op-179d8db0 | CreateLiftr | ANQ-1640 | East US | Oct 5, 11:18 UTC | 21 min | Failed AllocationFailed: Standard_L16s_v3 capacity unavailable in zone 2 after 3 retries |
| op-77368b46 | PatchLiftr | ANQ-1450 | East US 2 | Oct 5, 07:15 UTC | 3 min | Succeeded |
| op-75368820 | UpgradeQaaS | ANQ-1835 | Central US | Oct 5, 06:59 UTC | 55 min | Succeeded |
| op-78368cd9 | ScaleQaaS | ANQ-2224 | Canada Central | Oct 5, 06:34 UTC | 9 min | Succeeded |
| op-7dc43a2d | PatchLiftr | ANQ-0523 | Southeast Asia | Oct 5, 06:15 UTC | 34 s | Succeeded |
| op-7cc4389a | PatchLiftr | ANQ-2060 | UK South | Oct 5, 05:34 UTC | 1 min | Succeeded |
| op-9f926c62 | PatchLiftr | ANQ-1857 | South Central US | Oct 5, 05:29 UTC | 9 s | Failed HTTP 503 from Liftr receiver: Key Vault kv-anq-cp-prod-eus2 throttled secret read (429) |
| op-64c29077 | PatchLiftr | ANQ-1480 | East US 2 | Oct 5, 05:02 UTC | 3 min | Succeeded |
| op-87e734c6 | ScaleQaaS | ANQ-1517 | East US 2 | Oct 5, 02:05 UTC | 22 min | Succeeded |
Dependencies
Key Vaults · last 24 hours
| Vault | Purpose | Region | Availability | ServiceApiLatency | Throttled (429) | Status |
|---|---|---|---|---|---|---|
| kv-anq-cp-prod-eus2 | Control-plane secrets: Liftr receiver, executor, invoker | East US 2 | 100% | 24.0 ms | 46 | Healthy |
| kv-anq-cp-prod-weu | Control-plane secrets: EU executor and autoscaler | West Europe | 100% | 25.5 ms | 0 | Healthy |
| kv-anq-clusters-prod | Per-cluster admin credentials and encryption keys | East US 2 | 100% | 30.2 ms | 0 | Healthy |
Function Apps · last 24 hours
| Function App | Component | Availability | Cold starts | Executions | Failures | p95 duration | Last run | Status |
|---|---|---|---|---|---|---|---|---|
func-anq-liftr-receiver-prod Marketplace / Liftr create, patch, delete and SaaS webhooks | Liftr receiver | 99.95% | 0 | 30,966 | 77 | 653 ms | running | Healthy |
func-anq-cp-executor-prod Runs deployment, scaling and upgrade steps | QaaS control plane | 99.95% | 3 | 1,486 | 4 | 5 min | running | Degraded |
func-anq-cp-invoker-prod Dispatches queued operations to the executor | QaaS control plane | 100% | 2 | 2,904 | 0 | 1.8 s | running | Healthy |
func-anq-cp-autoscaler-prod Evaluates capacity and schedules scale operations | QaaS control plane | 100% | 6 | 288 | 0 | 14 s | 3 min ago | Healthy |
func-anq-cp-canary-prod Hourly synthetic create, patch and delete in the canary region | QaaS control plane | 100% | 11 | 24 | 0 | 27 min | 30 min ago | Healthy |
func-anq-sre-provision-prod Cluster create and expand | SRE automation | 100% | 9 | 46 | 0 | 7 min | 18 min ago | Healthy |
func-anq-sre-maintenance-prod Node patching and rolling restarts | SRE automation | 99.62% | 4 | 212 | 7 | 5 min | 4 min ago | Degraded |
func-anq-sre-upgrade-prod Qumulo Core version upgrades | SRE automation | 100% | 7 | 12 | 0 | 31 min | 1 h ago | Healthy |
func-anq-sre-healthcheck-prod Read-only cluster health checks | SRE automation | 99.98% | 1 | 7,200 | 3 | 6.0 s | running | Healthy |
func-anq-sre-decommission-prod Cluster teardown and data-disk release | SRE automation | 100% | 2 | 2 | 0 | 11 min | 9 h ago | Healthy |
PagerDuty · ANQ SRE Automation
- func-anq-sre-maintenance-prod: 7 failed runs in 24h#Q5VN9KC · opened Oct 5, 12:55 UTC · 1 h 35 min agoLowTriggered
Receiver triage runbook (Confluence)
Runbook link not configured
Receiver telemetry: Application Insights for func-anq-liftr-receiver-prod in rg-anq-service-prod-eus2. Alert rule: 3 failures in 15 minutes or any HTTP 5xx pages the on-call engineer (Sev 1).