Skip to content
About Me
----

Prometheus Alertmanager Architecture & Multi-Burn-Rate Alerting

Architect Key Takeaways

Master production alerting hygiene: Google SRE multi-window multi-burn-rate alerting, Prometheus Alertmanager routing trees, deduplication, and on-call escalation.

Prometheus Alertmanager Architecture & Multi-Burn-Rate Alerting

Section titled “Prometheus Alertmanager Architecture & Multi-Burn-Rate Alerting”

In production cloud engineering, bad alerting is worse than no alerting. An on-call rotation inundated with hundreds of noisy, transient alerts develops “alert fatigue”—leading engineers to ignore notifications until a catastrophic outage takes down customer traffic.

Modern SRE practices eliminate alert fatigue by implementing Multi-Window Multi-Burn-Rate Alerting based on Error Budget consumption rather than arbitrary CPU or memory thresholds.


The journey of an alert travels from metric evaluation in Prometheus through Alertmanager’s routing tree to on-call responders and automated runbooks:

Prometheus Alerting Pipeline: Multi-Window Multi-Burn-Rate & Alertmanager

2. Multi-Window Multi-Burn-Rate Alerting (Google SRE Standard)

Section titled “2. Multi-Window Multi-Burn-Rate Alerting (Google SRE Standard)”

Traditional threshold alerts (e.g. rate(http_requests_total{status=~"5.."}[5m]) > 0.05) suffer from two critical flaws:

  1. A small 30-second spike pages an engineer at 3 AM even though only 10 requests failed.
  2. A low-grade 1% error rate goes unnoticed for weeks, completely consuming the quarterly Error Budget.
  • A 1x Burn Rate consumes 100% of the Error Budget over exactly the SLO window (e.g. 30 days).
  • A 14.4x Burn Rate consumes 2% of the entire 30-day Error Budget in just 1 hour. This warrants an immediate page (severity: critical).
  • A 6x Burn Rate consumes 5% of the budget in 6 hours. This warrants an urgent business-hours ticket (severity: warning).

Production Prometheus Rule: Multi-Window Error Budget Burn

Section titled “Production Prometheus Rule: Multi-Window Error Budget Burn”
groups:
- name: api-slo-burn-rate-alerts
rules:
# Severe burn: 14.4x burn rate over 1h AND 5m (Immediate Page)
- alert: ErrorBudgetBurnRateSevere
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
) > (1 - 0.999) * 14.4
and
(
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
) > (1 - 0.999) * 14.4
for: 2m
labels:
severity: critical
tier: platform
annotations:
summary: "API Error Budget consuming at 14.4x rate (2% consumed in 1h)"
runbook_url: "https://devsecopsguru.in/observability/prometheus-alertmanager-runbooks/"
dashboard_url: "https://grafana.internal/d/api-slo"

3. Alertmanager Routing, Grouping & Inhibition Trees

Section titled “3. Alertmanager Routing, Grouping & Inhibition Trees”

Alertmanager decouples alert emission from notification dispatching, preventing alert storms through Grouping, Deduplication, and Inhibition.

Production alertmanager.yml Configuration:

Section titled “Production alertmanager.yml Configuration:”
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s # Wait 30s to buffer initial alerts together
group_interval: 5m # Send batch updates every 5m for new firing pods
repeat_interval: 4h # Re-notify every 4 hours if still broken
receiver: 'slack-warnings'
routes:
# Critical alerts route immediately to PagerDuty
- match:
severity: critical
receiver: 'pagerduty-urgent'
continue: true # Also continue down to post to Slack
- match:
severity: critical
receiver: 'slack-critical'
# Inhibit rules: suppress downstream symptoms if root cause is firing
inhibit_rules:
- source_match:
alertname: 'NodeNetworkDown'
target_match:
alertname: 'KubeletDown'
equal: ['node', 'instance']

4. Real-Time Production Troubleshooting Runbook

Section titled “4. Real-Time Production Troubleshooting Runbook”

Scenario: Paged for ErrorBudgetBurnRateSevere

Section titled “Scenario: Paged for ErrorBudgetBurnRateSevere”
  1. Acknowledge PagerDuty: Prevents automated escalation to the backup SRE.
  2. Inspect SLI Dashboard: Open the attached dashboard_url to identify which specific route (e.g. /api/v1/checkout) is returning 5xx status codes.
  3. Correlate with Recent Deployments: Check ArgoCD or GitHub Actions:
    Terminal window
    kubectl rollout history deployment/checkout-service -n production
  4. Instant Mitigation (Rollback): If a recent commit caused the burn, execute an automated rollback before debugging root cause:
    Terminal window
    kubectl rollout undo deployment/checkout-service -n production

Q1: Why must multi-burn-rate alerts evaluate both a short window (e.g. 5m) and a long window (e.g. 1h) simultaneously?

Evaluating only a 1-hour window means the alert would continue firing for 59 minutes after the incident is completely resolved, because the 1-hour rolling average still contains the historical error spike. Evaluating only a 5-minute window causes alerts to flap and resolve prematurely. Combining both windows guarantees the alert triggers quickly and resolves immediately when the system returns to normal.

Q2: What is the purpose of Alertmanager Inhibition Rules during a major data center network failure?

During an infrastructure network outage, hundreds of dependent services (Pods, API gateways, database connections, health checks) will fail simultaneously, triggering thousands of individual alerts. An Inhibition Rule identifies the root-cause alert (e.g. NetworkSwitchUnreachable) and automatically suppresses all secondary downstream alerts, ensuring the on-call engineer receives one actionable page instead of being overwhelmed by notifications.