Settings
Azure connection, signal sources, risk bands and notification routing.
Azure connection
Checking the Azure connection…
Required secrets
AZURE_TENANT_IDDirectory (tenant) ID of the app registrationAZURE_CLIENT_IDApplication (client) IDAZURE_CLIENT_SECRETClient secret valueAZURE_SUBSCRIPTION_IDSSubscription IDs to read, comma-separated
Optional settings
AZURE_CLUSTER_RG_PATTERNResource groups treated as clusters (default ^rg-anq-)AZURE_CUSTOMER_TAGResource-group tag holding the customer name (default customer)EARLY_WARNING_DATA_SOURCESet to builtin to keep built-in data while connected
The app registration needs only Reader and Monitoring Reader on each subscription. It never writes to Azure.
Alert rules
Starting thresholds, tuned from incident history. The dashboard shows alert state; paging stays in PagerDuty and Azure Monitor alert rules.
| Condition | Severity | Pages |
|---|---|---|
| Cluster lost quorum or file system unavailable | Sev 1 | Yes |
| Node offline > 2 min | Sev 2 | Yes |
| VmAvailabilityMetric < 1 for ≥ 1 min | Sev 2 | Yes |
| Resource Health = Unavailable / Degraded | Sev 2 | Yes |
| Any Scheduled Event received | Sev 3 | No; shown in timeline |
| Data protection degraded | Sev 2 | Yes |
| Disk or VM IOPS/bandwidth consumed > 95% for ≥ 5 min | Sev 3 | No |
| Data Disk Latency > 2× 7-day baseline for ≥ 5 min | Sev 3 | No |
| Blob ServerBusyError / ServerTimeoutError > 0 for ≥ 5 min | Sev 3 | No |
| Blob Availability < 99.9% over 5 min | Sev 2 | Yes |
| File system p99 latency > threshold per SKU for ≥ 10 min | Sev 2 | Yes |
| Capacity > 85% / > 95% | Sev 3 | No / Yes (Sev 2 above 95%) |
| Receiver failures ≥ 3 in 15 min, or HTTP 5xx | Sev 1 | Yes; existing alerts |
| Service Health incident in a region with ANQ clusters | Sev 2 | Notify only |
Data sources
| Source | Feeds | Scope | Interval | Last sync | Status |
|---|---|---|---|---|---|
Azure Service Health Resource Graph · ServiceHealthResources | Azure Platform Events | 6 subscriptions | 5 min | 2 min ago | Healthy |
Azure Resource Health Microsoft.ResourceHealth/availabilityStatuses | Azure Platform Events, Storage / Data-Path | 4,852 resources | 5 min | 3 min ago | Healthy |
Scheduled Events Instance Metadata Service via node agent | Azure Platform Events | 642 VMs | 1 min | 1 min ago | Healthy |
Azure Monitor metrics Managed disks, VMs, storage accounts | Storage / Data-Path, Compute Capacity & Provisioning | 4,852 resources | 1 min | 1 min ago | Healthy |
Compute quota and usage Microsoft.Compute/locations/usages | Compute Capacity & Provisioning | 25 regions | 15 min | 9 min ago | Healthy |
Azure Activity Log Event Hub export | Azure Platform Events, Compute Capacity & Provisioning | 6 subscriptions | Streaming | 14 min ago | Delayed |
Qumulo cluster telemetry Qumulo REST API · Prometheus exporter | Storage / Data-Path, Customer Impact (Cross-Cluster) | 200 clusters · 642 nodes | 1 min | 1 min ago | Healthy |
Network probes Read-only RTT, DNS, TCP and SMB/NFS connect checks | Network / Connectivity | 25 regions · 75 zones | 1 min | 1 min ago | Healthy |
Risk bands
| Band | Score | Response |
|---|---|---|
| Low | 0–25 | No action |
| Elevated | 26–50 | Shown on dashboard |
| High | 51–75 | Teams notification |
| Critical | 76–100 | Page on-call and open ticket |
Signal weights are recalibrated monthly against confirmed incidents and false positives.
Notifications
PagerDuty
Qumulo Azure SRE · Primary · Critical and above
Microsoft Teams
#azure-sre-early-warning · High and above
Jira
Auto-create issue · SRE project · Critical and above
Email digest
sre-leads distribution list · daily 08:00 UTC · Elevated and above