Skip to content

9. Resilience & Chaos

Goal

Make the checkout survive a bad day at the payment provider. Build the payment adapter and a fake external provider, put Toxiproxy between the services, watch a slow dependency take down all of order (a cascading failure), and fix it with Resilience4j β€” timeout, retry with backoff and jitter, circuit breaker, bulkhead β€” measuring after each pattern. End with a game day: a rehearsed incident with hidden failures and a blameless review.

At 10:00 the flash sale opens and the payment provider, under the load of every ticketing site in the country, starts answering in three seconds instead of three hundred milliseconds. Nothing is "down". Yet within a minute order stops answering everything β€” even GET /orders/{id}, which never talks to the provider β€” because every one of its threads is waiting on payment. This lab reproduces that failure on your laptop and applies the patterns of the reliability class, with a number for each.

The patterns are those of the reliability class: the cascading failure and its defences β€” timeout, retry with jitter, circuit breaker and bulkhead β€” then a game day. It comes right after lab 8 on purpose: chaos needs observability first, so every experiment is watched on the lab 8 dashboard and paged by its burn-rate alert.

The story so far

Chapter 10 of the story of the course. After lab 8, BoxOffice can see itself: a dashboard per service, one trace per order from the edge through Kafka to the e-mail, and a burn-rate alert that paged someone about 440 s after the database stopped in the test run. Seeing is not surviving. At 10:02 the payment provider answers in 4 s instead of 200 ms: the trace puts the time in the payment call, the alert fires β€” and checkout still dies, because every order thread is waiting on the provider and none is left even for requests that never touch it (class 10 β€” Reliability). This lab builds the defences in PaymentClient.java β€” timeout, retry with jitter, circuit breaker, bulkhead β€” reproduces the slow provider with Toxiproxy adding 3 s to every call, and measures what each pattern buys, above all for GET /orders/{id}, a request that never calls payment.

Slides

Click the slides and use ← β†’ to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you start

Time about 4 hours, plus 1 hour for the game day
Tools the lab 8 ones: Docker with Compose v2, k6, curl, jq, a browser for Grafana; watch (macOS: brew install watch)
Docker resources at least 4 CPUs and 8 GB (three more containers: payment, psp, toxiproxy)
Ports free on your machine: 8085, 8474 (plus the lab 8 ones)

Every command runs from the root of your boxoffice repository, in the state lab 8 left it:

Start state (end of lab 8)
boxoffice/
β”œβ”€β”€ compose.yaml            # labs 1-8: ... kafka, order, notification, otel-lgtm (+ x-otel anchor)
β”œβ”€β”€ infra/                  # nginx/, postgres/, pgbouncer/, grafana/, otel/
β”œβ”€β”€ load/  scripts/
└── services/
    β”œβ”€β”€ auth/  catalog/  gateway/  inventory/  notification/
    └── order/              # outbox with traceparent (lab 8)

Catch up

Skipped or broke lab 8? Download boxoffice-after-lab-8.zip, unzip it, cd boxoffice, run sh scripts/get-otel-agent.sh, and continue from here.

Every file of this lab is shown in full on these pages and published under https://insper.github.io/platform/2027.1/hands-on/9/code/<path>.

Architecture

flowchart LR
    k6[k6: pay + browse] --> order["order (Resilience4j)"]
    order -->|"timeout Β· retry Β· breaker Β· bulkhead"| tp1["toxiproxy: payment"]
    tp1 --> payment[payment]
    payment -->|"timeouts, Idempotency-Key"| tp2["toxiproxy: psp"]
    tp2 --> psp["psp (WireMock)"]
    gm[game master] -.->|"latency, reset, outage"| tp1
    gm -.-> tp2
    order -.->|OTLP| lgtm[otel-lgtm]
    payment -.->|OTLP| lgtm

BoxOffice components touched

Palette component In this lab
Payment psp, a WireMock fake of an external provider
3rd Party API the provider behind an adapter, with timeouts and idempotency keys
App Server order and payment, protected by Resilience4j
Alerting the burn-rate alert of lab 8, used to detect the game-day failures

What you will measure

Question Metric Tool
Does a slow dependency hurt endpoints that do not use it? browse p95 and errors while payment is slow k6
How much does each pattern help, and what does it cost? pay p95, confirmed %, fast-fail % k6
How fast does the breaker open and recover? state transitions over time /actuator/circuitbreakers
How fast does the team detect and mitigate? time to detect, time to mitigate game-day timeline

Prerequisites

Steps

  1. Payment, provider and Toxiproxy β€” the idempotent payment adapter, a WireMock provider, Toxiproxy proxies and your first toxic.
  2. Resilience patterns β€” reproduce the cascade, then add timeout, retry, circuit breaker and bulkhead one at a time.
  3. Game day β€” roles, hypotheses, hidden failures, timeline and a blameless review.
  4. Measure it β€” the k6 script with a pay and a browse scenario, and the results table.

Try it in the playground

Try it: in cascading-failure, run the load profile and raise the latency of the 3rd-party node: watch the app server's utilization reach 100% and its errors spread to every request. Then set a timeout and enable the circuit breaker on the edge to the 3rd party, and run again. In retry-storm, compare retries: 3 with and without exp+jitter backoff during an outage.

The boxoffice-payment preset is this lab in the playground: pay-under-chaos.js's 20 payments/s and 20 browse requests/s against order's 50 Tomcat threads, and at t = 60 s Toxiproxy adds 3 s to every call to payment. Run it as it is and watch browsing fail too; then select the edge order β†’ payment, set the lab's timeout (1 000 ms), 3 attempts with jittered backoff, the circuit breaker and a bulkhead of 0.2 of the threads (10 of 50), and run again: payments fail fast during the incident and browsing keeps its threads.

The full simulator is in the playground.

Measure it

The Measure it step has the k6 script, the commands and the results table to keep in results/lab9.md; the game-day reviews go to results/lab9-gameday.md.

Troubleshooting

Symptom Likely cause Fix
curl localhost:8474/proxies fails toxiproxy not running, or infra/toxiproxy/toxiproxy.json missing docker compose ps toxiproxy; docker compose logs toxiproxy
every payment fails with I/O error … Connection refused the payment proxy was left disabled by an experiment curl -s -X POST localhost:8474/reset (removes toxics, re-enables proxies)
pay returns 503 even with no toxic the circuit breaker is still OPEN from the last experiment, or the first call was cold wait 10 s (wait-duration-in-open-state) and retry; check /actuator/circuitbreakers
psp answers 404 the WireMock stub was not mounted the file must be infra/psp/mappings/charge.json; docker compose restart psp
PAYMENT_* variables seem ignored they only reach the container through compose.yaml and a recreate PAYMENT_BULKHEAD=10 docker compose up -d order (not restart)
results vary a lot between runs the laptop runs k6 and 20 containers close other apps, run each experiment twice, compare ratios

Best practices in this lab

  • A timeout on every outbound call, shorter than the caller's β€” PaymentClient.java (500 ms connect, 1 s read). Why: Timeout.
  • Retries only with an idempotency key, with exponential backoff and jitter β€” PaymentClient.java sends Idempotency-Key: order-<id>; application.yaml allows 3 attempts, 200 ms Β± 50%. Why: Retry with backoff and jitter.
  • A bulkhead smaller than the thread pool, and fail fast when it is full β€” application.yaml: 10 of the 50 threads, max-wait-duration: 0. Why: Bulkhead.
  • No database transaction open while waiting on a remote call β€” OrderService.java calls payment first and updates the status in a short transaction. Why: Cascading failures (the wait would exhaust the connection pool instead).
  • Chaos as code, with a one-command abort β€” scripts/game-day.sh and infra/toxiproxy/toxiproxy.json; reset removes every toxic. Why: Game days.

Clean up

curl -s -X POST localhost:8474/reset    # no toxic left behind
docker compose stop                     # keep the stack for lab 10's images
docker compose down -v                  # or remove everything

End state

End state (what this lab added or changed)
boxoffice/
β”œβ”€β”€ compose.yaml            # + psp, toxiproxy, payment; order -> toxiproxy:18085 + PAYMENT_* knobs
β”œβ”€β”€ infra/
β”‚   β”œβ”€β”€ grafana/alerting/slo-order.yaml   # the lab 8 burn-rate alerts, for order (game day)
β”‚   β”œβ”€β”€ psp/mappings/charge.json
β”‚   └── toxiproxy/toxiproxy.json
β”œβ”€β”€ load/pay-under-chaos.js
β”œβ”€β”€ scripts/game-day.sh
β”œβ”€β”€ results/lab9.md, lab9-gameday.md
└── services/
    β”œβ”€β”€ order/              # pom (+ resilience4j, aspectj), application.yaml, OrderService,
    β”‚                       # OrderController (+ /pay), payment/ package
    └── payment/            # new: idempotent adapter to the provider (+ Dockerfile, .dockerignore)

Commit and tag: git add -A && git commit -m "lab 9: resilience" && git tag lab9.

Checklist

  • payment, psp and toxiproxy run in compose.yaml; order and payment reach their dependencies through Toxiproxy
  • POST /payments is idempotent by Idempotency-Key, and the key is forwarded to the provider
  • POST /orders/{id}/pay confirms the order (status CONFIRMED + OrderConfirmed in the outbox) and is protected by timeout, retry, circuit breaker and bulkhead
  • the unprotected run shows the cascade (browse degraded); each pattern's run shows its effect, in numbers
  • one game day with at least three hidden experiments, a timeline and a blameless review per round
  • results table filled in results/lab9.md and explained
  • self-check: curl -s -X POST localhost:8474/reset, then PAYMENT_RESILIENCE=true docker compose up -d --wait order and one pay call returns CONFIRMED; pushed with the tag lab9

What breaks next

The patterns contain the slow provider: in the test run, GET /orders/{id} waited 9 s at the median without them and a few milliseconds with them, and the game day timed how long your team took to detect and mitigate. But look at who mitigated: a person at a terminal, running docker compose and curl against Toxiproxy. Every defence of this lab assumes that something restarts a dead replica, adds replicas before 10:00 and rolls a bad release back β€” on one laptop with Compose, that something is you, and at 10:00, with 2 million fans in the line, nobody should be restarting containers by hand. Lab 10 β€” Kubernetes hands those jobs to control loops β€” probes that restart, an autoscaler that adds catalog pods, rolling updates that do not lose requests β€” the platform of class 11 β€” Cloud-Native.

Start: Payment, provider and Toxiproxy