3. Game Day

The patterns of step 2 were tested one at a time, by the person who wrote them, knowing what would break. Real incidents are not like that. A game day is a rehearsed incident: a team injects failures into a running system, without telling the operators which ones, and checks two hypotheses at once — that the system degrades the way the design says, and that the people notice, diagnose and mitigate in time. It is the practice side of chaos engineering.

Roles (groups of 3–4)

Role	Does	Must not
Game master	picks and injects the failures with Toxiproxy, keeps the clock	tell the others what was injected
Operator(s)	watches Grafana (lab 8) and the logs, diagnoses, mitigates	look at the game master's terminal
Scribe	writes the timeline: every observation and decision, with the time	fix anything

Rotate roles between rounds, so that everyone operates once.

Before you start

Steady state. Run load/pay-under-chaos.js with DURATION=30m and write down the normal values from the RED dashboard: order and payment rate, error %, p95. This is the baseline every experiment is compared with.
Hypotheses. For each experiment below, the team writes before injecting what it expects to see (e.g. "with payment +2 s, order p95 for pay stays under 1.5 s and GET /orders/{id} is unaffected").
Blast radius and abort. Only your laptop's BoxOffice. Abort condition: anything outside the experiment breaks (e.g. catalog errors). The abort button is one command: curl -s -X POST localhost:8474/reset (removes every toxic and re-enables every proxy).
Detection. The burn-rate alert from lab 8 watches only catalog. Add the same two rules for order (the lab 8 file with catalog replaced by order, shown below), then recreate otel-lgtm with docker compose up -d --force-recreate --wait otel-lgtm: Alerting → Alert rules now lists four rules in the folder BoxOffice, all Normal, so that the operators can be paged. Recreating (not just restarting) also empties the metrics: otherwise the errors of the step 2 experiments, still inside the 1-hour and 6-hour windows, make the order rules fire at once. The dashboard and the rules are files, so they come back.

infra/grafana/alerting/slo-order.yaml — the complete file

infra/grafana/alerting/slo-order.yaml

# Lab 9: slo-catalog.yaml of lab 8 with "catalog" replaced by "order", so the game day can page.
# Multiwindow, multi-burn-rate alerts for the order availability SLO (Google SRE workbook, ch. 5).
# SLI: share of order requests (health checks excluded) NOT answered with 5xx. SLO: 99.5% / 28 days.
apiVersion: 1

groups:
  - orgId: 1
    name: order-slo
    folder: BoxOffice
    interval: 30s
    rules:
      - uid: order-availability-fast-burn
        title: "order availability SLO: fast burn"
        condition: C
        data:
          - refId: A          # burn rate over the long window (1 h)
            relativeTimeRange: { from: 3600, to: 0 }
            datasourceUid: prometheus
            model:
              refId: A
              instant: true
              expr: |
                (
                  sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*", http_response_status_code=~"5.."}[1h]))
                  / sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*"}[1h]))
                ) / 0.005
          - refId: B          # burn rate over the short window (5 min): stops paging soon after recovery
            relativeTimeRange: { from: 300, to: 0 }
            datasourceUid: prometheus
            model:
              refId: B
              instant: true
              expr: |
                (
                  sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*", http_response_status_code=~"5.."}[5m]))
                  / sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*"}[5m]))
                ) / 0.005
          - refId: C
            datasourceUid: __expr__
            model:
              refId: C
              type: math
              expression: "$A > 14.4 && $B > 14.4"
        for: 1m
        noDataState: OK
        execErrState: Error
        labels:
          severity: page
          slo: order-availability
        annotations:
          summary: "order is burning its 28-day error budget 14.4x too fast"
          description: 'Burn rate {{ printf "%.1f" $values.A.Value }} (1 h) and {{ printf "%.1f" $values.B.Value }} (5 min): at this pace about 2% of the 28-day budget is gone in one hour.'

      - uid: order-availability-slow-burn
        title: "order availability SLO: slow burn"
        condition: C
        data:
          - refId: A          # 6 h window
            relativeTimeRange: { from: 21600, to: 0 }
            datasourceUid: prometheus
            model:
              refId: A
              instant: true
              expr: |
                (
                  sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*", http_response_status_code=~"5.."}[6h]))
                  / sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*"}[6h]))
                ) / 0.005
          - refId: B          # 30 min window
            relativeTimeRange: { from: 1800, to: 0 }
            datasourceUid: prometheus
            model:
              refId: B
              instant: true
              expr: |
                (
                  sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*", http_response_status_code=~"5.."}[30m]))
                  / sum(rate(http_server_request_duration_seconds_count{service_name="order", http_route!~"/actuator.*"}[30m]))
                ) / 0.005
          - refId: C
            datasourceUid: __expr__
            model:
              refId: C
              type: math
              expression: "$A > 6 && $B > 6"
        for: 5m
        noDataState: OK
        execErrState: Error
        labels:
          severity: ticket
          slo: order-availability
        annotations:
          summary: "order is burning its error budget 6x too fast"
          description: 'Burn rate {{ printf "%.1f" $values.A.Value }} (6 h) and {{ printf "%.1f" $values.B.Value }} (30 min): about 5% of the 28-day budget in six hours.'

The experiments

The game master picks three of these, in any order, 5–8 minutes each with recovery time in between (at least 5 minutes, until the fast-burn rule is back to Normal: its 5-minute window must forget the previous experiment, or the next one is "detected" by an alert that never stopped firing):

#	Injection (game master)	What a good system does	What a good operator does
1	`payment` +2 s latency	timeouts + breaker: fast `503`s on `pay`, `browse` unaffected	finds `payment` in the trace, sees the breaker open
2	`psp` black hole (`timeout` toxic, 0)	`payment` read timeout (2 s) fails calls; `order` breaker opens	follows the trace one hop further, to the provider
3	40% `reset_peer` on `payment`	retries hide most errors; error rate rises only a little	notices retries in the metrics, does not panic
4	`payment` proxy disabled	breaker open, fast `503`, orders stay `PENDING_PAYMENT`	communicates "payments down, orders safe"
5	`db` paused (`docker compose pause db`)	everything slows; the SLO alert fires	identifies the shared dependency, not the symptom
6	1 KB/s `bandwidth` on `payment`	`pay` about 0.5 s slower (a few hundred bytes at 1 KB/s), under the read timeout: no errors, only latency	distinguishes "slow" from "down"

Injection commands:

scripts/game-day.sh

#!/bin/sh
# Game master helper. Usage: sh scripts/game-day.sh <experiment 1..6 | reset>
API=localhost:8474
toxic() { # proxy, json
  curl -s -X POST "$API/proxies/$1/toxics" -d "$2" > /dev/null
}
echo "$(date +%T) experiment $1"
case "$1" in
  1) toxic payment '{"name":"gd","type":"latency","stream":"downstream","attributes":{"latency":2000,"jitter":300}}' ;;
  2) toxic psp '{"name":"gd","type":"timeout","stream":"downstream","attributes":{"timeout":0}}' ;;
  3) toxic payment '{"name":"gd","type":"reset_peer","toxicity":0.4,"attributes":{"timeout":0}}' ;;
  4) curl -s -X POST "$API/proxies/payment" -d '{"enabled":false}' > /dev/null ;;
  5) docker compose pause db ;;
  6) toxic payment '{"name":"gd","type":"bandwidth","stream":"downstream","attributes":{"rate":1}}' ;;
  reset) curl -s -X POST "$API/reset" > /dev/null; docker compose unpause db 2> /dev/null || true ;;
  *) echo "unknown experiment"; exit 1 ;;
esac

During the round

The operators work as in a real incident (see the incident response class): declare the incident when an alert fires or a symptom is seen, assign an incident commander, diagnose with the dashboard → trace → logs path, mitigate first (e.g. stop sending traffic to a broken path, raise a timeout, scale), investigate later. The scribe records times in this format:

Time	Who	Observation / decision
10:02:10	GM	(hidden) experiment 1 injected
10:04:30	Op	`order` p95 rising on the dashboard; no alert yet
10:05:05	Op	trace shows `payment` span ~2 s; breaker `OPEN` in actuator
…

From the timeline you get the two numbers every incident review asks for: time to detect (injection → first correct observation) and time to mitigate (injection → users no longer affected).

After: a blameless review

Write one review per round in results/lab9-gameday.md. Blameless means the review explains how the system and the process allowed the outcome — never who is to blame.

results/lab9-gameday.md (template)

# Game day — round <n>

**Experiment:** <what was injected, when> · **Duration:** <mm:ss>
**Hypothesis:** <what we expected>
**Result:** <confirmed | refuted> — <one sentence>

## Impact
- Users affected: <which requests, how many, for how long> (from the RED dashboard)
- SLO budget spent: <% of the 28-day budget>

## Timeline
| Time | Event |
|---|---|

## Detection and mitigation
- Time to detect: <min> — how: <alert | dashboard | trace>
- Time to mitigate: <min> — how: <...>

## What went well / what was hard

## Action items
| Action | Type (prevent / detect / mitigate) | Owner |
|---|---|---|

Pitfalls

No hypothesis, no experiment. Breaking things to "see what happens" teaches little; a refuted hypothesis is the valuable outcome.
Start small. Latency before outages, one dependency before two, a laptop before production.
The abort button must work before the first injection. Test reset first.
Fix the action items. A game day whose findings stay in a document will be repeated, with the same findings, by a real incident.

Next: Measure it