Incident Response

10:03, Saturday. The phone of the on-call engineer vibrates: CheckoutErrorBudgetFastBurn. Two million fans are in the waiting room, the payment provider is answering in seconds, and the company's social media is already lighting up. What happens in the next thirty minutes decides whether this becomes a bad morning or a news story. Good telemetry tells you what is wrong; it does not decide who does what, who talks to the outside world, or when you stop investigating and start mitigating. That is incident response — a practice borrowed from firefighters and emergency services, adapted to software¹.

What counts as an incident

An incident is an unplanned event that degrades or threatens a service users depend on, and that needs a coordinated response. Not every alert is an incident; an incident is declared when the response needs more than one person working the normal way. Severity levels make that decision fast and consistent — nobody should debate the definition at 3 a.m.

On-call

On-call means a named person is responsible, for a period, for responding to pages for a set of services. It is a production requirement, not a heroic act, and it must be sustainable.

• Rotations: a primary and a secondary (escalation if the primary does not acknowledge within a few minutes), usually week-long shifts with a written handoff — open issues, risky deploys, silences in place.
• Follow-the-sun: teams in different time zones split the day so nobody is paged at night; it needs at least two sites.
• Sustainable load: Google's SRE practice caps on-call work so that engineers spend most of their time on engineering, and aims for at most about two incidents per 12-hour shift — an incident takes hours of follow-up, not just the response². More than that means the alerting or the system needs work, not more stamina.
• You build it, you run it: the team that owns a service carries its pager. It is the strongest incentive to build operable software.
• Preparation: runbooks, access already granted (never "I'm waiting for production access"), dashboards bookmarked, and practice in game days (Chaos engineering).

The incident lifecycle

stateDiagram-v2
    [*] --> Detected : alert, customer report, or engineer notices
    Detected --> Triaged : on-call acknowledges, assesses impact
    Triaged --> Declared : severity set, commander and channel assigned
    Triaged --> Closed : false alarm or no impact (tune the alert)
    Declared --> Mitigating : stop the bleeding
    Mitigating --> Monitoring : impact gone, SLIs recovering
    Monitoring --> Mitigating : impact returns
    Monitoring --> Resolved : stable, root fix scheduled
    Resolved --> Reviewed : blameless postmortem, action items
    Reviewed --> [*]
    Closed --> [*]

Each transition has a timestamp, and the gaps between them are the metrics that matter:

Users only feel impact: the time until mitigation is the number that shapes availability (recall from Reliability that availability = MTTF / (MTTF + MTTR)). The root cause can be fixed on Monday.

Roles

The most common way an incident goes wrong is not a missing tool — it is five engineers debugging the same thing in parallel, nobody deciding, and nobody telling customers what is going on. The Incident Command System (ICS), created for wildfire response in the 1970s, solved this with explicit roles; Google, PagerDuty and others adapted it to software¹³.

In a small team, one person can hold several roles; the IC and the operations lead should still be different people whenever possible. Roles are handed over explicitly ("I am now IC") when shifts change or the incident escalates.

Play the first fifteen minutes of the flash-sale incident:

sequenceDiagram
    autonumber
    participant AM as Alertmanager
    participant OC as On-call (primary)
    participant IC as Incident commander
    participant OPS as Ops lead (payments)
    participant COM as Comms lead
    participant ST as Status page
    AM->>OC: page: CheckoutErrorBudgetFastBurn
    OC->>OC: ack, open dashboard: 504s on /checkout, on-sale live
    OC->>IC: declare SEV1, open channel inc-onsale-0313
    IC->>OPS: you lead mitigation, payments on-call joins
    IC->>COM: you own updates, every 15 min
    COM->>ST: "Some fans see errors at checkout. We are working on it."
    OPS->>OPS: exemplar trace: 7.9 s in payment pool.acquire
    OPS->>IC: provider slow, propose switch to secondary provider
    IC->>OPS: approved, go
    OPS->>OPS: flag payment.provider=secondary, waiting room throttled
    OPS->>IC: error ratio back under 0.5%, p99 1.4 s
    IC->>COM: mitigated, monitoring
    COM->>ST: "Checkout is working again. Monitoring."

Mitigate first

During an incident the goal is to stop the impact, not to understand it. Understanding comes later, with the postmortem — and is much easier once the pressure is gone. The best mitigations are generic: they work regardless of the exact cause, and they are rehearsed in advance.

Communication

Silence during an incident is read as incompetence or indifference. Updates have a fixed cadence (for a SEV1, every 15–30 minutes) even when there is nothing new, and a fixed shape:

[SEV1] Checkout errors during the 10:00 on-sale — update #3, 10:21
Impact:   since 10:02, ~12% of checkout attempts failed; fans in the waiting room were not affected.
Status:   MITIGATED at 10:14 — traffic moved to the secondary payment provider.
Next:     monitoring until 10:45; the primary provider is investigating on their side.
Next update: 10:45 or earlier if anything changes.
IC: @ana   Ops: @bruno   Comms: @carla

Outside the company, the status page states impact and progress, never internal names, guesses or blame ("our provider failed").

The BoxOffice incident, by the numbers

The page came one minute after the impact started only because the on-sale has its own SLO, alerting on 5-minute and 1-minute windows (Alerting). At 12% failures, the general 28-day rule — 14.4× over a 1-hour window — would have needed about 60 min × 1.44% / 12% ≈ 7 minutes to fire: more than half of the time to mitigation, gone before anyone looked.

What did it cost in SLO terms? With illustrative numbers — 200 checkout attempts per second during the on-sale and an average of 12% failures for the 12 minutes of impact:

If BoxOffice handles about 2 million checkouts in a normal 28-day window (the SLO window used in Reliability), a 99.9% SLO allows 2,000 failures. This one incident spent more than eight 28-day budgets — over seven months' worth — in twelve minutes. The error budget policy now applies: feature work on checkout pauses until the reliability action items are done. That is the error budget doing its job — turning one bad morning into a funded engineering priority.

Postmortems

A postmortem (also called incident review or retrospective) is a written analysis of an incident, done after it is resolved, with one goal: reduce the chance and the impact of the next one. Write one for every SEV1/SEV2, any data loss, any significant budget spend — and for interesting near misses.

Blameless

A postmortem is blameless⁴: it assumes everyone acted reasonably given what they knew at the time, and it asks how the system made the mistake easy, not who made it. This is not about being nice. If people are punished for honest accounts, they stop giving them, and the organisation loses the only accurate source of information about how its systems really fail. "The engineer ran the wrong command" becomes "the tool allowed a destructive command against production without confirmation, and the runbook showed it next to a harmless one".

Root cause — or contributing factors?

Complex systems rarely fail for one reason; incidents need several factors lining up⁵. In the BoxOffice incident:

• the provider slowed down (external trigger, outside BoxOffice's control);
• the payment adapter had no timeout shorter than the provider's own, and a pool of 20 connections shared by every call;
• the secondary provider existed, but switching was a manual, undocumented step;
• the general 28-day SLO alerts were not designed for a 30-minute on-sale;
• the load test before the on-sale used a provider mock that always answered in 100 ms.

Asking "five whys" down a single chain would have stopped at "the provider was slow" — the one factor BoxOffice cannot fix. Looking for all contributing factors finds four that it can.

Template

# Postmortem: checkout errors during the 13 March on-sale (SEV1)

**Status**: final · **Authors**: … · **Reviewed**: …

## Summary
Two to four sentences: what happened, impact, how it was mitigated.

## Impact
Users affected, duration, failed checkouts, revenue and SLO budget consumed, support tickets.

## Timeline (UTC-3)
10:02 provider latency rises … 10:03 page … 10:14 mitigated … 10:40 resolved.

## Contributing factors
Technical, process and organisational factors — no names, no blame.

## What went well / what went poorly / where we got lucky

## Action items
| Action | Type (prevent / detect / mitigate) | Owner | Due | Ticket |
|---|---|---|---|---|
| Timeout 800 ms + bulkhead on the provider client | prevent | payments | 27/03 | PAY-312 |
| Automatic failover to the secondary provider | mitigate | payments | 10/04 | PAY-313 |
| On-sale SLO with 1 min / 5 min burn-rate alerts | detect | SRE | 20/03 | OBS-88 |
| Provider latency injected in the pre-sale load test | prevent | performance | 27/03 | PERF-41 |

Action items are the only part of a postmortem that changes anything. Each one is specific, has one owner and a date, and is tracked like any other work. A postmortem whose actions are never done is a story, not a fix.

Closing the loop

• Review meeting: the authors walk the team through the document; questions are about the system, not the people.
• Share widely: other teams have the same pool, the same missing timeout.
• Look for trends: many incidents from deploys → invest in canaries and rollbacks (Delivery); many from one dependency → bulkheads and fallbacks.
• Practice: turn the incident into a game day scenario — inject provider latency with Toxiproxy and check that the new timeout, breaker and alert behave (Hands-on 9).
• Measure delivery health: the DORA metrics change failure rate and failed deployment recovery time⁶ show whether incidents caused by changes are becoming rarer and shorter.

echo "$(date +%T) experiment $1"
case "$1" in
  1) toxic payment '{"name":"gd","type":"latency","stream":"downstream","attributes":{"latency":2000,"jitter":300}}' ;;
  2) toxic psp '{"name":"gd","type":"timeout","stream":"downstream","attributes":{"timeout":0}}' ;;
  3) toxic payment '{"name":"gd","type":"reset_peer","toxicity":0.4,"attributes":{"timeout":0}}' ;;
  4) curl -s -X POST "$API/proxies/payment" -d '{"enabled":false}' > /dev/null ;;
  5) docker compose pause db ;;
  6) toxic payment '{"name":"gd","type":"bandwidth","stream":"downstream","attributes":{"rate":1}}' ;;
  reset) curl -s -X POST "$API/reset" > /dev/null; docker compose unpause db 2> /dev/null || true ;;
  *) echo "unknown experiment"; exit 1 ;;

Trade-offs

How it connects

flowchart LR
    AL["burn-rate page<br/>→ Alerting"] --> TRI["triage: severity<br/>SEV1 to SEV4"]
    TRI --> ROLES["IC, ops lead,<br/>comms lead, scribe"]
    ROLES --> MIT["mitigate first"]
    MIT --> RB["rollback, flag off<br/>→ Cloud-native delivery"]
    MIT --> FO["failover, shed load,<br/>degrade → Reliability"]
    MIT --> BLK["block bad traffic<br/>→ Edge (WAF)"]
    MIT -->|"timestamps"| MTT["MTTD, MTTM, MTTR<br/>→ availability"]
    MTT --> BUD["error budget spent<br/>→ SLO policy"]
    MIT --> PM["blameless postmortem,<br/>action items"]
    PM --> GD["game day<br/>→ Chaos engineering"]

Palette components: alerting (the page that starts it) · tracing (the exemplar trace that found pool.acquire) · payment (failover to the secondary provider) · queue (the waiting room, throttled to shed load) · waf (blocking abusive traffic) · app (rollback or scale-out of a service). Scenarios: flash-sale (the on-sale incident), cascading-failure (a slow provider without timeouts). Continues in: Reliability (availability = MTTF / (MTTF + MTTR)), SLIs, SLOs & error budgets, Reliability — chaos engineering, Cloud-native — delivery (rollbacks, canaries, DORA), Project — game day, Interview preparation.

Key takeaways