ANQ-0217Degraded
Life Sciences · East US 2 · cluster detail
Customer
Life Sciences
Subscription
sub-anq-clusters-prod
Region
East US 2
Zone layout
Zonal · Zone 1
Blob SKU
Hot · ZRS
Nodes
4
VM SKU
Standard_L16s_v3
Qumulo version
7.2.2
Environment
Prod
Resource group
Risk72· Lsv3 family · generation v2 (tiers to Blob) · local time America/New_York
Active alerts
4
2 paging
Read latency p95
20.8ms
Alert above 20 ms
Capacity used
64%
61 of 96 TiB
Busiest data disk
77%
disk-anq-c0217-data-01
Nodes online
4/4
0 evictions in 24 h · fully protected
Blob availability
98.29%
Lowest of 5 accounts
Likely Azure-causedAzure
Symptom at Oct 5, 14:16 UTC: File system read latency p95 rising: 9.3 → 20.8 ms. Scheduled Event preceded it by 6 min on vm-anq-c0217-n02: Freeze (platform): Host update: memory-preserving maintenance, VM paused 9 s. 3 more Azure signals nearby. No control plane operation in the 2 hours before.
Shared timeline
Azure (Service Health, Resource Health, Scheduled Events, Activity Log, metrics), Qumulo events and symptoms, and control plane operations on one axis.
- Oct 5, 14:18 UTCAzureAzure MonitorBlob ServerBusyError and ServerTimeoutError, 1.8% of transactions
- Oct 5, 14:16 UTCQumuloSymptomFile system read latency p95 rising: 9.3 → 20.8 ms
- Oct 5, 14:10 UTCQumuloNode heartbeatn02: Node unresponsive for 9 s while the VM was paused; below the 30 s eviction threshold, not evicted
- Oct 5, 14:10 UTCAzureScheduled EventFreeze (platform): Host update: memory-preserving maintenance, VM paused 9 s
- Oct 5, 14:10 UTCAzureActivity LogPlatform maintenance (memory-preserving) (platform-initiated)
- Oct 5, 13:38 UTCAzureService HealthService issue 7KT4-9QZ: Degraded performance for Premium SSD v2 managed disks (East US 2)
Alerts · 4 active, 2 resolved
| Rule | Severity | Since | Pages |
|---|---|---|---|
File system p99 latency > threshold per SKU for ≥ 10 min rg-anq-prod-eus2-c0217 p99 45.8 ms vs 45 ms for L16s_v3 | Sev 2 | Oct 5, 14:26 UTC Active 4 min | Yes |
Blob Availability < 99.9% over 5 min stanq0217data02 Availability 98.29% | Sev 2 | Oct 5, 14:23 UTC Active 7 min | Yes |
Service Health incident in a region with ANQ clusters East US 2 7KT4-9QZ: Degraded performance for Premium SSD v2 managed disks | Sev 2 | Oct 5, 13:38 UTC Active 52 min | No |
Blob ServerBusyError / ServerTimeoutError > 0 for ≥ 5 min stanq0217data02 ServerBusyError 13.3/min, ServerTimeoutError 2/min | Sev 3 | Oct 5, 14:23 UTC Active 7 min | No |
VmAvailabilityMetric < 1 for ≥ 1 min vm-anq-c0217-n02 VmAvailabilityMetric 0.85 during Freeze | Sev 2 | Oct 5, 14:11 UTC Resolved 14:12 UTC | Yes |
Any Scheduled Event received vm-anq-c0217-n02 Freeze · Host update: memory-preserving maintenance, VM paused 9 s | Sev 3 | Oct 5, 14:09 UTC Resolved 14:11 UTC | No |
PagerDuty
- Q1M4RX8 ANQ-0217 read latency p95 above 20 msHigh · Triggered · 14:18 UTC
File system latency
Qumulo telemetry · read and write p95, all-op p99
Read p9520.8 msWrite p9527.9 msp9945.8 msp99 alert · L16s_v3 45 ms
Data disk consumed
Azure Monitor · 12 Premium SSD v2 data disks, % of provisioned
Busiest disk IOPS77%Busiest disk bandwidth61.2%Average IOPS64.4%Alert 95%
Nodes · 4
| Node | VM | State | VM availability | CPU | CPU steal | Disk throttling | Floating IPs | Last Scheduled Event |
|---|---|---|---|---|---|---|---|---|
| n01 | vm-anq-c0217-n01 | VM running Provisioning succeeded | 1 | 39% | 0.8% | 77% | 3 | None in 30 days |
| n02 | vm-anq-c0217-n02 | VM running Provisioning succeeded | min 0.85 | 61% | 21.4% | 71.4% | 3 | Freeze · Completed Oct 5, 14:10 UTC |
| n03 | vm-anq-c0217-n03 | VM running Provisioning succeeded | 1 | 55% | 0.4% | 75.5% | 3 | Freeze · Completed Sep 12, 06:04 UTC |
| n04 | vm-anq-c0217-n04 | VM running Provisioning succeeded | 1 | 31% | 0.5% | 69.7% | 3 | None in 30 days |