Chaos Engineering

BoxOffice now has timeouts, a retry with jitter, a circuit breaker, a bulkhead, a fallback, a warm standby region and a backup policy. Do they work? The unit tests pass. But nobody has ever seen the payment breaker open with real traffic, nobody has ever promoted the standby database, and nobody knows how long the regional failover really takes. The first time any of it matters will be during a real stadium on-sale, with two million fans watching. The only way to know is to cause the failures yourself, on purpose, while you are watching and ready.

Chaos engineering is the discipline of experimenting on a system in order to build confidence in its ability to withstand turbulent conditions in production¹. Instead of waiting for failures to happen, you inject them deliberately, in a controlled way, and use what you learn to fix weaknesses before real incidents find them.

"The best way to avoid failure is to fail constantly." — John Ciancutti, Netflix, 2010⁴

Where it came from

The idea is older than the name. At Amazon in the early 2000s, Jesse Robbins ran GameDay exercises that deliberately caused major failures in production to expose hidden weaknesses — in the software and in the teams operating it³.

Netflix gave the practice its name and its tools. After moving from its own data centre to AWS around 2010 — where any instance can disappear at any time — it built Chaos Monkey, a service that terminates random production instances during business hours, so that every team had to build services that survive instance loss, and engineers were at their desks to see what broke⁴. A whole Simian Army followed: Latency Monkey injected delays into calls, Chaos Gorilla simulated the loss of a whole availability zone. In 2016 Netflix engineers formalized the discipline in an IEEE Software article², from which the principles below come.

The insight: failure is not a special case to avoid; it is the environment production systems live in, and the only way to know a system survives it is to test it there.

Principles

The Principles of Chaos Engineering¹ define the discipline with five ideas:

1. Build a hypothesis around steady-state behaviour : Describe "normal" with measurable outputs users care about — the SLIs (SLOs): purchase success rate, seat-hold p99, e-mail freshness. The hypothesis: "the steady state continues even when X happens."

2. Vary real-world events : Inject failures that actually happen: instance and zone loss, latency, packet loss, a dependency returning errors, a full disk, clock skew, a traffic spike, an expired certificate.

3. Run experiments in production : Staging rarely has production's traffic shape, data volume or dependency behaviour. Production is the goal — reached gradually, not the first step.

4. Automate experiments to run continuously : Systems change every day; a weakness fixed today can return with next week's deploy. Repeat experiments automatically.

5. Minimize blast radius : Affect as few users as possible, have an abort switch, and widen the scope only after smaller experiments pass.

The experiment lifecycle

Each experiment follows the scientific method: predict, change one variable, measure, compare⁵.

flowchart LR
    H["1. Hypothesis<br/>steady state + expected behaviour"] --> D["2. Design<br/>fault, scope, abort conditions"]
    D --> I["3. Inject<br/>the controlled fault"]
    I --> O["4. Observe<br/>SLIs, traces, breaker state"]
    O --> A["5. Analyse<br/>expected vs actual"]
    A --> F["6. Improve<br/>fix, update runbooks"]
    F -->|"repeat with a wider scope"| H

An experiment that confirms the hypothesis is evidence that a defence works. One that falsifies it is a finding: a weakness discovered on your schedule, with the team ready, instead of at 10:00 on a sale day.

export const options = {
  scenarios: {
    pay: {
      executor: 'constant-arrival-rate',
      exec: 'pay',
      rate: Number(__ENV.PAY_RATE || 20),
      timeUnit: '1s',
      duration: DURATION,
      preAllocatedVUs: 50,
      maxVUs: 600,
    },
    browse: {
      executor: 'constant-arrival-rate',
      exec: 'browse',
      rate: 20,
      timeUnit: '1s',
      duration: DURATION,
      preAllocatedVUs: 20,
      maxVUs: 300,
    },
  },
  summaryTrendStats: ['avg', 'med', 'p(95)', 'p(99)', 'max'],
  thresholds: {
    'http_req_duration{name:pay}': ['p(95)<1500'],
    'http_req_duration{name:browse}': ['p(95)<200'], // the collateral damage
    'http_req_failed{name:browse}': ['rate<0.01'],
  },
};

Fault injection

Toxiproxy: chaos on your laptop

Toxiproxy is a TCP proxy with an API to add "toxics" — latency, timeouts, bandwidth limits, connection resets — to a connection. Put it between payment and the provider stub in Docker Compose, and point payment at the proxy:

# docker-compose.yaml (excerpt)
services:
  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.9.0
    ports:
      - "8474:8474"           # Toxiproxy API
  payment:
    environment:
      PROVIDER_URL: http://toxiproxy:8666   # instead of http://provider-stub:8080

# create the proxy: toxiproxy:8666 -> provider-stub:8080
toxiproxy-cli create -l 0.0.0.0:8666 -u provider-stub:8080 provider

# experiment 1: 3 s of latency (+/- 500 ms) on every response
toxiproxy-cli toxic add -t latency -a latency=3000 -a jitter=500 provider

# experiment 2: connections hang forever (a "black hole")
toxiproxy-cli toxic add -t timeout -a timeout=0 provider

# end of the experiment
toxiproxy-cli toxic remove -n latency_downstream provider
toxiproxy-cli toxic remove -n timeout_downstream provider

[
  { "name": "payment", "listen": "0.0.0.0:18085", "upstream": "payment:8085", "enabled": true },
  { "name": "psp",     "listen": "0.0.0.0:18095", "upstream": "psp:8080",     "enabled": true }
]

toxic() { # proxy, json
  curl -s -X POST "$API/proxies/$1/toxics" -d "$2" > /dev/null
}
echo "$(date +%T) experiment $1"
case "$1" in
  1) toxic payment '{"name":"gd","type":"latency","stream":"downstream","attributes":{"latency":2000,"jitter":300}}' ;;
  2) toxic psp '{"name":"gd","type":"timeout","stream":"downstream","attributes":{"timeout":0}}' ;;
  3) toxic payment '{"name":"gd","type":"reset_peer","toxicity":0.4,"attributes":{"timeout":0}}' ;;
  4) curl -s -X POST "$API/proxies/payment" -d '{"enabled":false}' > /dev/null ;;
  5) docker compose pause db ;;
  6) toxic payment '{"name":"gd","type":"bandwidth","stream":"downstream","attributes":{"rate":1}}' ;;
  reset) curl -s -X POST "$API/reset" > /dev/null; docker compose unpause db 2> /dev/null || true ;;
  *) echo "unknown experiment"; exit 1 ;;
esac

Run k6 before, during and after each toxic and compare the numbers — that is the "Measure it" step of Hands-on 9.

Chaos Mesh: chaos on Kubernetes

On Kubernetes, experiments are declared as resources, versioned in Git and bounded in time:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-latency
  namespace: boxoffice
spec:
  action: delay
  mode: one                  # affect a single pod: small blast radius
  selector:
    namespaces: [boxoffice]
    labelSelectors:
      app: payment
  delay:
    latency: "3s"
    jitter: "500ms"
  duration: "10m"            # the experiment ends by itself

Controlling the blast radius

The blast radius of an experiment is the set of users and systems it can affect. Controlling it is what separates engineering from recklessness.

flowchart LR
    S1["1. Local / Compose<br/>no users"] -->|"hypothesis holds"| S2["2. Staging under k6 load<br/>internal users"]
    S2 -->|"no findings"| S3["3. One canary pod in production<br/>a few % of traffic"]
    S3 -->|"SLIs hold"| S4["4. Wider production<br/>a zone, then a region"]

• Abort conditions tied to SLOs: stop automatically if the error-budget burn rate passes a threshold or a key SLI drops (AWS FIS and Chaos Mesh support stop conditions; at minimum, a human with a button).
• Time-boxed: every experiment has a duration and cleans up after itself.
• Business hours, never during an on-sale: the team must be present; the error budget policy decides whether you can afford it.
• Announce it: the on-call and the support team know an experiment is running.

Game days

A game day is a planned, team-wide exercise in which failures are injected and the people respond as in a real incident. Automated experiments test the software; game days also test the runbooks, the dashboards, the alerts and the communication.

Continuous verification

Once an experiment passes, keep it passing:

• In CI: integration tests with Toxiproxy (Testcontainers has a Toxiproxy module) assert that the client gives up after its timeout and that the breaker opens — cheap, every build.
• Scheduled in staging: Chaos Mesh or LitmusChaos workflows run the standard experiments nightly against the latest deploy.
• In production, small: a recurring pod-kill on a canary, a monthly backup restore, a twice-yearly regional failover drill (Disaster recovery).

Observability first

Chaos without observability is just breaking things. Before injecting anything you must be able to:

1. See the steady state — SLI dashboards with baselines.
2. Detect the deviation — alerts that fire within minutes (burn-rate alerts).
3. Explain it — traces that show which call got slow, metrics of breakers and pools.
4. Stop it — an abort switch that works in seconds.

Observability first, then resilience patterns, then chaos experiments to prove the patterns work.

It is also the order of the labs: lab 8 instruments BoxOffice before lab 9 breaks it on purpose, and the game-day operators work from the lab 8 dashboard and alerts.

How it connects

flowchart LR
    SS["steady state = SLIs<br/>→ SLOs"] --> HYP["hypothesis"]
    HYP --> INJ["inject: Toxiproxy,<br/>Chaos Mesh, AWS FIS"]
    LOAD["k6 load during the run<br/>→ Performance"] --> INJ
    INJ --> PAT["defences under test:<br/>breaker, bulkhead, fallback"]
    INJ --> DR["failover, restore<br/>→ Disaster recovery"]
    OBS["metrics, traces, burn-rate alerts<br/>→ Observability"] --> OA["observe and abort"]
    PAT --> OA
    DR --> OA
    OA --> BR["widen the blast radius:<br/>local → canary → zone"]
    OA --> GD["game day + blameless debrief<br/>→ Incident response"]
    GD --> HYP

Palette components: thirdparty (the provider stub behind Toxiproxy) · app (order pods killed during the game day) · kafka (a broker stopped) · sql (failover to the standby) · cache (Redis flushed: stampede protection) · metrics · tracing · alerting (you cannot run chaos without them). Scenarios: flash-sale (chaos profile), cascading-failure. Continues in: Observability — incident response, Performance — load testing, Cloud-native — Kubernetes, Hands-on 9 — payment and Toxiproxy, Project — game day.