ANQ-0142Degraded
Media & Entertainment · East US 2 · cluster detail
Customer
Media & Entertainment
Subscription
sub-anq-clusters-prod
Region
East US 2
Zone layout
Zonal · Zone 1
Blob SKU
Hot · ZRS
Nodes
6
VM SKU
Standard_L32s_v3
Qumulo version
7.2.2
Environment
Prod
Resource group
Risk86· Lsv3 family · generation v2 (tiers to Blob) · local time America/New_York
Active alerts
5
2 paging
Read latency p95
26.6ms
Alert above 20 ms
Capacity used
78%
225 of 288 TiB
Busiest data disk
100%
disk-anq-c0142-data-07
Nodes online
6/6
0 evictions in 24 h · fully protected
Blob availability
99.99%
Lowest of 5 accounts
Likely Azure-causedAzure
Symptom at Oct 5, 13:49 UTC: File system read latency p95 rising: 9 → 26.6 ms. Azure Monitor preceded it by 4 min on disk-anq-c0142-data-07: Data disk IOPS consumed 100%: throttling at provisioned IOPS. 2 more Azure signals nearby. No control plane operation in the 2 hours before.
Shared timeline
Azure (Service Health, Resource Health, Scheduled Events, Activity Log, metrics), Qumulo events and symptoms, and control plane operations on one axis.
- Oct 5, 13:49 UTCQumuloSymptomFile system read latency p95 rising: 9 → 26.6 ms
- Oct 5, 13:46 UTCQumuloNode driven03: Drive disk-anq-c0142-data-07 (TW) write latency above 30 ms; flagged slow, no data at risk
- Oct 5, 13:45 UTCAzureAzure MonitorData disk IOPS consumed 100%: throttling at provisioned IOPS
- Oct 5, 13:38 UTCAzureService HealthService issue 7KT4-9QZ: Degraded performance for Premium SSD v2 managed disks (East US 2)
Alerts · 5 active
| Rule | Severity | Since | Pages |
|---|---|---|---|
File system p99 latency > threshold per SKU for ≥ 10 min rg-anq-prod-eus2-c0142 p99 58.5 ms vs 40 ms for L32s_v3 | Sev 2 | Oct 5, 13:59 UTC Active 31 min | Yes |
Resource Health = Unavailable / Degraded disk-anq-c0142-data-07 Resource Health Degraded | Sev 2 | Oct 5, 13:46 UTC Active 44 min | Yes |
Service Health incident in a region with ANQ clusters East US 2 7KT4-9QZ: Degraded performance for Premium SSD v2 managed disks | Sev 2 | Oct 5, 13:38 UTC Active 52 min | No |
Disk or VM IOPS/bandwidth consumed > 95% for ≥ 5 min disk-anq-c0142-data-07 IOPS consumed 100%, bandwidth 81% | Sev 3 | Oct 5, 13:50 UTC Active 40 min | No |
Data Disk Latency > 2× 7-day baseline for ≥ 5 min 18 data disks · worst disk-anq-c0142-data-07 p95 42 ms vs 7-day baseline 13.5 ms | Sev 3 | Oct 5, 13:50 UTC Active 40 min | No |
PagerDuty
- Q3K7ZP1 ANQ-0142 read latency p95 above 20 msHigh · Acknowledged · 13:52 UTC
File system latency
Qumulo telemetry · read and write p95, all-op p99
Read p9526.6 msWrite p9535.4 msp9958.5 msp99 alert · L32s_v3 40 ms
Data disk consumed
Azure Monitor · 18 Premium SSD v2 data disks, % of provisioned
Busiest disk IOPS100%Busiest disk bandwidth81%Average IOPS90.7%Alert 95%
Nodes · 6
| Node | VM | State | VM availability | CPU | CPU steal | Disk throttling | Floating IPs | Last Scheduled Event |
|---|---|---|---|---|---|---|---|---|
| n01 | vm-anq-c0142-n01 | VM running Provisioning succeeded | 1 | 40% | 0.5% | 90.9% | 3 | None in 30 days |
| n02 | vm-anq-c0142-n02 | VM running Provisioning succeeded | 1 | 63% | 0.4% | 92.5% | 3 | None in 30 days |
| n03 | vm-anq-c0142-n03 | VM running Provisioning succeeded | 1 | 54% | 0.7% | 100% | 3 | None in 30 days |
| n04 | vm-anq-c0142-n04 | VM running Provisioning succeeded | 1 | 60% | 0.7% | 94.1% | 3 | None in 30 days |
| n05 | vm-anq-c0142-n05 | VM running Provisioning succeeded | 1 | 44% | 0.5% | 90.7% | 3 | None in 30 days |
| n06 | vm-anq-c0142-n06 | VM running Provisioning succeeded | 1 | 56% | 0.7% | 91.7% | 3 | None in 30 days |