ANQ-1236Degraded

Pharmaceuticals · West Europe · cluster detail

Customer
Pharmaceuticals
Subscription
sub-anq-clusters-prod
Region
West Europe
Zone layout
Zonal · Zone 1
Blob SKU
Hot · LRS
Nodes
4
VM SKU
Standard_D16s_v5
Qumulo version
7.3.0
Environment
Prod
Resource group
Risk22· Dsv5 family · generation v1 (managed disks only) · local time Europe/Amsterdam
Active alerts
1
0 paging
Read latency p95
8.9ms
Alert above 20 ms
Capacity used
87%
84 of 96 TiB
Busiest data disk
41%
disk-anq-c1236-data-04
Nodes online
4/4
0 evictions in 24 h · fully protected
Blob availability
100.00%
Lowest of 5 accounts
No Azure or control plane signal: Qumulo-sideQumulo

Symptom at Oct 5, 08:30 UTC: Capacity used crossed 85% (now 87%). No Azure Service Health, Resource Health, Scheduled Event or throttling signal within 10 minutes. No control plane operation in the 2 hours before.

Shared timeline

Azure (Service Health, Resource Health, Scheduled Events, Activity Log, metrics), Qumulo events and symptoms, and control plane operations on one axis.

  1. Oct 5, 08:30 UTCQumuloSymptom
    Capacity used crossed 85% (now 87%)

Alerts · 1 active

RuleSeveritySincePages
Capacity > 85% / > 95%
rg-anq-prod-weu-c1236
Capacity used 87%
Sev 3
Oct 5, 08:30 UTC
Active 6 h 0 min
No
PagerDuty
  • Q8TD2LW ANQ-1236 storage capacity used above 85%Low · Acknowledged · 08:30 UTC

File system latency

Qumulo telemetry · read and write p95, all-op p99

Read p958.9 msWrite p9512.4 msp9919.7 msp99 alert · D16s_v5 60 ms

Data disk consumed

Azure Monitor · 12 Premium SSD v2 data disks, % of provisioned

Busiest disk IOPS41.1%Busiest disk bandwidth34.7%Average IOPS35.5%Alert 95%

Nodes · 4

NodeVMStateVM availabilityCPUCPU stealDisk throttlingFloating IPsLast Scheduled Event
n01vm-anq-c1236-n01
VM running
Provisioning succeeded
138%0.6%38.7%2None in 30 days
n02vm-anq-c1236-n02
VM running
Provisioning succeeded
136%0.4%41.1%2None in 30 days
n03vm-anq-c1236-n03
VM running
Provisioning succeeded
159%0.4%36.4%2None in 30 days
n04vm-anq-c1236-n04
VM running
Provisioning succeeded
156%0.4%40.6%2None in 30 days