Alerting

An alert is a request for a human's attention, often at 3 a.m. That makes it the most expensive signal in the system: it costs sleep, focus and, over time, trust. Two failure modes kill alerting. Too many alerts — "CPU above 80%" on every pod, every blip of every dependency — and the on-call engineer learns to ignore the pager; the real incident drowns in the noise. Too few, or the wrong ones — and the first person to notice the broken checkout is a fan on social media. This page is about alerts that fire when, and only when, users are being hurt enough that someone must act.

What deserves a page

A good page is urgent, actionable, and about a user-visible problem¹. If any of the three is missing, it should be something else:

Symptoms, not causes

Alert on symptoms — what users experience: errors, latency, freshness, correctness. Use causes — CPU, memory, a pod restart, a full queue — for dashboards and diagnosis, not for pages.

The exception is a leading indicator that will certainly become a symptom and gives time to act — a disk filling up, a certificate expiring in 7 days. Those are tickets, not pages, unless the deadline is hours away.

From the SLO to the alert

The SLO already says how much failure is acceptable: with 99.9% of checkouts succeeding over a rolling 28-day window (the window used throughout Reliability), the error budget is 0.1% of checkouts. The question an alert must answer is therefore not "are there errors?" — there always are — but "are we spending the budget fast enough to be in trouble?".

The natural measure is the burn rate:

A burn rate of 1 spends exactly the whole budget in the whole window; a burn rate of 14.4 spends it 14.4 times faster. If a burn rate \(b\) is sustained for a time \(t\), the fraction of the budget consumed is

A burn rate of 1000× is a total outage (100% errors against a 0.1% budget): the whole 28-day budget is gone in about 40 minutes. That is why fast alerts must fire within minutes — and why slow burns can wait for office hours.

Six ways to write the alert

The SRE Workbook² compares six progressively better designs for an SLO alert. Their quality is measured by four properties: precision (fraction of alerts that were real problems), recall (fraction of real problems that alerted), detection time and reset time (how long the alert keeps firing after the problem ends).

Multiwindow, multi-burn-rate

The recommended design combines, for each severity, a long window that proves the burn is significant and a short window (1/12 of the long one) that proves it is still happening — so the alert stops quickly once the problem is fixed:

The plot shows the detection time for a sudden failure of a given error ratio, for the two paging rules. Note the trade-off it encodes: the worse the incident, the faster the alert — and for a given rule, the budget spent at the moment it fires is always the same (2% and 5%).

A total outage (100% errors) fires the fast rule in under a minute; an incident failing 2% of checkouts takes about 43 minutes to fire the fast rule, and one failing 0.7% never fires it, but fires the 6-hour rule after about 5 hours. Below 0.6% — a burn rate under 6× — only the ticket rule catches it.

The rules in PromQL

Recording rules precompute the error ratio of the checkout journey over each window; the alert rules compare them with the thresholds. (Tools such as Sloth, Pyrra or the OpenSLO specification generate these rules from a short SLO definition, which avoids copy-paste mistakes.)

# prometheus-rules.yaml — checkout availability SLO: 99.9% over 28 days
groups:
  - name: slo-checkout-recordings
    rules:
      - record: slo:checkout_errors:ratio_rate5m
        expr: |
          sum(rate(http_server_request_duration_seconds_count{job="order", http_route="/checkout", http_response_status_code=~"5.."}[5m]))
          /
          sum(rate(http_server_request_duration_seconds_count{job="order", http_route="/checkout"}[5m]))
      - record: slo:checkout_errors:ratio_rate1h
        expr: |
          sum(rate(http_server_request_duration_seconds_count{job="order", http_route="/checkout", http_response_status_code=~"5.."}[1h]))
          /
          sum(rate(http_server_request_duration_seconds_count{job="order", http_route="/checkout"}[1h]))
      # ... the same expression for 30m, 6h and 3d

  - name: slo-checkout-alerts
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          slo:checkout_errors:ratio_rate1h > (14.4 * 0.001)
          and
          slo:checkout_errors:ratio_rate5m > (14.4 * 0.001)
        labels:
          severity: page
          team: checkout
        annotations:
          summary: "Checkout is burning its 28-day error budget at more than 14.4x"
          runbook_url: "runbooks/checkout-slo.md"
      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          slo:checkout_errors:ratio_rate6h > (6 * 0.001)
          and
          slo:checkout_errors:ratio_rate30m > (6 * 0.001)
        labels:
          severity: page
          team: checkout
      - alert: CheckoutErrorBudgetTicket
        expr: |
          slo:checkout_errors:ratio_rate3d > (1 * 0.001)
          and
          slo:checkout_errors:ratio_rate6h > (1 * 0.001)
        labels:
          severity: ticket
          team: checkout

A latency SLO works the same way: the "bad events" are the requests slower than the threshold, computed from the histogram bucket that ends at the threshold:

# fraction of checkouts slower than 2 s over 1 h (the "error ratio" of a latency SLO)
1 - (
  sum(rate(http_server_request_duration_seconds_bucket{job="order", http_route="/checkout", le="2"}[1h]))
  /
  sum(rate(http_server_request_duration_seconds_count{job="order", http_route="/checkout"}[1h]))
)

- uid: catalog-availability-fast-burn
  title: "catalog availability SLO: fast burn"
  condition: C
  data:
    - refId: A          # burn rate over the long window (1 h)
      relativeTimeRange: { from: 3600, to: 0 }
      datasourceUid: prometheus
      model:
        refId: A
        instant: true
        expr: |
          (
            sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*", http_response_status_code=~"5.."}[1h]))
            / sum(rate(http_server_request_duration_seconds_count{service_name="catalog", http_route!~"/actuator.*"}[1h]))
          ) / 0.005
    - refId: C
      datasourceUid: __expr__
      model:
        refId: C
        type: math
        expression: "$A > 14.4 && $B > 14.4"
  for: 1m
  noDataState: OK
  execErrState: Error
  labels:
    severity: page
    slo: catalog-availability

Routing: from a firing rule to a person

An alert rule only produces a firing alert. Something must decide who hears about it, how, and how often. In the Prometheus ecosystem this is Alertmanager³ (Grafana Alerting has the same concepts): it groups related alerts into one notification, deduplicates them, applies inhibitions and silences, and routes them by labels to receivers.

flowchart LR
    R["Rule evaluation<br/>Prometheus / Mimir ruler"] --> F["Firing alert<br/>severity=page, team=checkout"]
    F --> G["Alertmanager<br/>group · dedupe"]
    G --> S{"Silenced or<br/>inhibited?"}
    S -->|yes| X["Dropped<br/>(still visible in the UI)"]
    S -->|no| RT{"Route by labels"}
    RT -->|severity=page| PD["Pager<br/>on-call phone"]
    RT -->|severity=page| SL["Slack #incidents"]
    RT -->|severity=ticket| TK["Ticket queue"]
    PD --> ACK["On-call acknowledges<br/>or escalates after 5 min"]

# alertmanager.yaml (sketch)
route:
  receiver: slack-observability          # default for anything unmatched
  group_by: [alertname, team]
  group_wait: 30s                        # wait to batch related alerts into one notification
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: pager-checkout
      continue: true                     # also evaluate the next route
    - matchers: [severity="page"]
      receiver: slack-incidents
    - matchers: [severity="ticket"]
      receiver: ticket-queue

inhibit_rules:
  - source_matchers: [alertname="RegionDown"]
    target_matchers: [severity=~"page|ticket"]
    equal: [region]                      # one page for the region, not fifty for its services

receivers:
  - name: pager-checkout
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pager-checkout
  - name: slack-incidents
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-incidents
        channel: "#incidents"
  - name: slack-observability
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-observability
        channel: "#observability"
  - name: ticket-queue
    webhook_configs:
      - url: http://ticket-bridge:8080/alerts

Who watches the watcher?

If Prometheus or Alertmanager is down, nothing alerts — and silence looks exactly like health. Two standard protections:

• A dead man's switch: a rule that always fires (expr: vector(1), often named Watchdog), sent to an external service that pages if the heartbeat stops arriving.
• Absent data alerts: absent(up{job="order"}) or absent_over_time(...) fire when a metric disappears — a crashed exporter, a renamed metric, a job that stopped being scraped.

For journeys with little traffic at night, a ratio of zero requests is undefined; synthetic probes (a k6 or Blackbox Exporter check that buys a test ticket on a fake event every minute) keep the SLI meaningful at any hour.

Runbooks

Every page must link a runbook: a short document that tells a tired engineer what the alert means and what to try first. Runbooks reduce MTTR because the first 10 minutes stop being improvisation.

# CheckoutErrorBudgetFastBurn

**Meaning**: checkout is failing at more than 14.4x the rate the 99.9% SLO allows;
at this pace the 28-day budget is gone in about 2 days.

**Impact**: fans cannot buy tickets. During an on-sale, declare an incident immediately (SEV1).

**First checks (5 minutes)**
1. Checkout dashboard: which status codes? which route? since when? any deploy or flag change?
2. Payment provider panel: latency and error ratio of the provider call.
3. Open an exemplar trace of a failed checkout: which span fails or waits?

**Mitigations**
- Recent deploy → roll back (`argocd app rollback order`).
- Provider slow or failing → switch to the secondary provider (flag `payment.provider=secondary`).
- Database saturated → enable the waiting room (flag `waiting-room.enabled=true`).

**Escalate to**: payments team (secondary on-call), then the incident commander rotation.

Machines detect, people repair

An alert does not take a broken replica out of the load balancer: a health check does that in seconds, and wakes nobody. The page is for what a machine cannot do on its own — restore what was lost before the next failure finds BoxOffice with less redundancy than it was designed for. That is why the runbook above shortens MTTR, and why a replica that restarts and recovers by itself is not worth a page.

Try it. In observability-stack, one of the three catalog replicas dies at t = 50 s and stays dead for 80 s. Add Metrics (connected to the load balancer, the app and the database) and a Health Check connected to the load balancer, and run: errors last 3 s, and availability is 99.4%. Now add Alerting connected to Metrics and run again.

Trade-offs

How it connects

flowchart LR
    SLO["SLO 99.9% over 28 days<br/>→ SLIs, SLOs & error budgets"] --> BUD["error budget 0.1%"]
    SLI["SLI from metrics<br/>(RED, histograms)"] --> BR["burn rate =<br/>error ratio ÷ budget"]
    BUD --> BR
    BR --> MW["multiwindow rules:<br/>14.4× and 6× page, 1× ticket"]
    MW --> AM["Alertmanager: group,<br/>inhibit, silence, route"]
    AM --> PAGE["page + runbook"]
    AM --> TK["ticket"]
    PAGE --> IR["Incident Response"]
    DMS["dead man's switch,<br/>absent(), synthetic probes"] --> AM
    MW -.-> ONS["on-sale SLO,<br/>short windows"]

Palette components: alerting · metrics (the SLI and the recording rules) · payment (the slow provider that burns the on-sale budget) · kafka (a broker disk filling up: a ticket, not a page). Scenarios: observability-stack, flash-sale (a 30-minute on-sale needs its own SLO). Continues in: SLIs, SLOs & error budgets (SLI, SLO, error budget policy), Incident Response, Cloud-native — delivery (rollbacks named in the runbook), Hands-on 8 — SLO and burn-rate alert.

Key takeaways