9. Resilience & Chaos
Goal
Make the checkout survive a bad day at the payment provider. Build the payment adapter and a fake external provider, put Toxiproxy between the services, watch a slow dependency take down all of order (a cascading failure), and fix it with Resilience4j β timeout, retry with backoff and jitter, circuit breaker, bulkhead β measuring after each pattern. End with a game day: a rehearsed incident with hidden failures and a blameless review.
At 10:00 the flash sale opens and the payment provider, under the load of every ticketing site in the country, starts answering in three seconds instead of three hundred milliseconds. Nothing is "down". Yet within a minute order stops answering everything β even GET /orders/{id}, which never talks to the provider β because every one of its threads is waiting on payment. This lab reproduces that failure on your laptop and applies the patterns of the reliability class, with a number for each.
The patterns are those of the reliability class: the cascading failure and its defences β timeout, retry with jitter, circuit breaker and bulkhead β then a game day. It comes right after lab 8 on purpose: chaos needs observability first, so every experiment is watched on the lab 8 dashboard and paged by its burn-rate alert.
The story so far
Chapter 10 of the story of the course. After lab 8, BoxOffice can see itself: a dashboard per service, one trace per order from the edge through Kafka to the e-mail, and a burn-rate alert that paged someone about 440 s after the database stopped in the test run. Seeing is not surviving. At 10:02 the payment provider answers in 4 s instead of 200 ms: the trace puts the time in the payment call, the alert fires β and checkout still dies, because every order thread is waiting on the provider and none is left even for requests that never touch it (class 10 β Reliability). This lab builds the defences in PaymentClient.java β timeout, retry with jitter, circuit breaker, bulkhead β reproduces the slow provider with Toxiproxy adding 3 s to every call, and measures what each pattern buys, above all for GET /orders/{id}, a request that never calls payment.
Slides
Click the slides and use β β to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab
Before you start
| Time | about 4 hours, plus 1 hour for the game day |
| Tools | the lab 8 ones: Docker with Compose v2, k6, curl, jq, a browser for Grafana; watch (macOS: brew install watch) |
| Docker resources | at least 4 CPUs and 8 GB (three more containers: payment, psp, toxiproxy) |
| Ports | free on your machine: 8085, 8474 (plus the lab 8 ones) |
Every command runs from the root of your boxoffice repository, in the state lab 8 left it:
boxoffice/
βββ compose.yaml # labs 1-8: ... kafka, order, notification, otel-lgtm (+ x-otel anchor)
βββ infra/ # nginx/, postgres/, pgbouncer/, grafana/, otel/
βββ load/ scripts/
βββ services/
βββ auth/ catalog/ gateway/ inventory/ notification/
βββ order/ # outbox with traceparent (lab 8)
Catch up
Skipped or broke lab 8? Download boxoffice-after-lab-8.zip, unzip it, cd boxoffice, run sh scripts/get-otel-agent.sh, and continue from here.
Every file of this lab is shown in full on these pages and published under https://insper.github.io/platform/2027.1/hands-on/9/code/<path>.
Architecture
flowchart LR
k6[k6: pay + browse] --> order["order (Resilience4j)"]
order -->|"timeout Β· retry Β· breaker Β· bulkhead"| tp1["toxiproxy: payment"]
tp1 --> payment[payment]
payment -->|"timeouts, Idempotency-Key"| tp2["toxiproxy: psp"]
tp2 --> psp["psp (WireMock)"]
gm[game master] -.->|"latency, reset, outage"| tp1
gm -.-> tp2
order -.->|OTLP| lgtm[otel-lgtm]
payment -.->|OTLP| lgtm BoxOffice components touched
| Palette component | In this lab |
|---|---|
| Payment | psp, a WireMock fake of an external provider |
| 3rd Party API | the provider behind an adapter, with timeouts and idempotency keys |
| App Server | order and payment, protected by Resilience4j |
| Alerting | the burn-rate alert of lab 8, used to detect the game-day failures |
What you will measure
| Question | Metric | Tool |
|---|---|---|
| Does a slow dependency hurt endpoints that do not use it? | browse p95 and errors while payment is slow | k6 |
| How much does each pattern help, and what does it cost? | pay p95, confirmed %, fast-fail % | k6 |
| How fast does the breaker open and recover? | state transitions over time | /actuator/circuitbreakers |
| How fast does the team detect and mitigate? | time to detect, time to mitigate | game-day timeline |
Prerequisites
- Labs 7 (the
orderservice) and 8 (dashboards and alerts) done. - Read the reliability class: resilience patterns and chaos engineering.
Steps
- Payment, provider and Toxiproxy β the idempotent
paymentadapter, a WireMock provider, Toxiproxy proxies and your first toxic. - Resilience patterns β reproduce the cascade, then add timeout, retry, circuit breaker and bulkhead one at a time.
- Game day β roles, hypotheses, hidden failures, timeline and a blameless review.
- Measure it β the k6 script with a
payand abrowsescenario, and the results table.
Try it in the playground
Try it: in cascading-failure, run the load profile and raise the latency of the 3rd-party node: watch the app server's utilization reach 100% and its errors spread to every request. Then set a timeout and enable the circuit breaker on the edge to the 3rd party, and run again. In retry-storm, compare retries: 3 with and without exp+jitter backoff during an outage.
The boxoffice-payment preset is this lab in the playground: pay-under-chaos.js's 20 payments/s and 20 browse requests/s against order's 50 Tomcat threads, and at t = 60 s Toxiproxy adds 3 s to every call to payment. Run it as it is and watch browsing fail too; then select the edge order β payment, set the lab's timeout (1 000 ms), 3 attempts with jittered backoff, the circuit breaker and a bulkhead of 0.2 of the threads (10 of 50), and run again: payments fail fast during the incident and browsing keeps its threads.
The full simulator is in the playground.
Measure it
The Measure it step has the k6 script, the commands and the results table to keep in results/lab9.md; the game-day reviews go to results/lab9-gameday.md.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
curl localhost:8474/proxies fails | toxiproxy not running, or infra/toxiproxy/toxiproxy.json missing | docker compose ps toxiproxy; docker compose logs toxiproxy |
every payment fails with I/O error β¦ Connection refused | the payment proxy was left disabled by an experiment | curl -s -X POST localhost:8474/reset (removes toxics, re-enables proxies) |
pay returns 503 even with no toxic | the circuit breaker is still OPEN from the last experiment, or the first call was cold | wait 10 s (wait-duration-in-open-state) and retry; check /actuator/circuitbreakers |
psp answers 404 | the WireMock stub was not mounted | the file must be infra/psp/mappings/charge.json; docker compose restart psp |
PAYMENT_* variables seem ignored | they only reach the container through compose.yaml and a recreate | PAYMENT_BULKHEAD=10 docker compose up -d order (not restart) |
| results vary a lot between runs | the laptop runs k6 and 20 containers | close other apps, run each experiment twice, compare ratios |
Best practices in this lab
- A timeout on every outbound call, shorter than the caller's β
PaymentClient.java(500 ms connect, 1 s read). Why: Timeout. - Retries only with an idempotency key, with exponential backoff and jitter β
PaymentClient.javasendsIdempotency-Key: order-<id>;application.yamlallows 3 attempts, 200 ms Β± 50%. Why: Retry with backoff and jitter. - A bulkhead smaller than the thread pool, and fail fast when it is full β
application.yaml: 10 of the 50 threads,max-wait-duration: 0. Why: Bulkhead. - No database transaction open while waiting on a remote call β
OrderService.javacallspaymentfirst and updates the status in a short transaction. Why: Cascading failures (the wait would exhaust the connection pool instead). - Chaos as code, with a one-command abort β
scripts/game-day.shandinfra/toxiproxy/toxiproxy.json;resetremoves every toxic. Why: Game days.
Clean up
curl -s -X POST localhost:8474/reset # no toxic left behind
docker compose stop # keep the stack for lab 10's images
docker compose down -v # or remove everything
End state
boxoffice/
βββ compose.yaml # + psp, toxiproxy, payment; order -> toxiproxy:18085 + PAYMENT_* knobs
βββ infra/
β βββ grafana/alerting/slo-order.yaml # the lab 8 burn-rate alerts, for order (game day)
β βββ psp/mappings/charge.json
β βββ toxiproxy/toxiproxy.json
βββ load/pay-under-chaos.js
βββ scripts/game-day.sh
βββ results/lab9.md, lab9-gameday.md
βββ services/
βββ order/ # pom (+ resilience4j, aspectj), application.yaml, OrderService,
β # OrderController (+ /pay), payment/ package
βββ payment/ # new: idempotent adapter to the provider (+ Dockerfile, .dockerignore)
Commit and tag: git add -A && git commit -m "lab 9: resilience" && git tag lab9.
Checklist
-
payment,pspandtoxiproxyrun incompose.yaml;orderandpaymentreach their dependencies through Toxiproxy -
POST /paymentsis idempotent byIdempotency-Key, and the key is forwarded to the provider -
POST /orders/{id}/payconfirms the order (statusCONFIRMED+OrderConfirmedin the outbox) and is protected by timeout, retry, circuit breaker and bulkhead - the unprotected run shows the cascade (
browsedegraded); each pattern's run shows its effect, in numbers - one game day with at least three hidden experiments, a timeline and a blameless review per round
- results table filled in
results/lab9.mdand explained - self-check:
curl -s -X POST localhost:8474/reset, thenPAYMENT_RESILIENCE=true docker compose up -d --wait orderand onepaycall returnsCONFIRMED; pushed with the taglab9
Related
- Theory: Reliability Β· Service level objectives (SLOs) and error budgets Β· Resilience patterns Β· Chaos engineering Β· External integrations
- Case study: Payment system
- Project: Chaos game day
What breaks next
The patterns contain the slow provider: in the test run, GET /orders/{id} waited 9 s at the median without them and a few milliseconds with them, and the game day timed how long your team took to detect and mitigate. But look at who mitigated: a person at a terminal, running docker compose and curl against Toxiproxy. Every defence of this lab assumes that something restarts a dead replica, adds replicas before 10:00 and rolls a bad release back β on one laptop with Compose, that something is you, and at 10:00, with 2 million fans in the line, nobody should be restarting containers by hand. Lab 10 β Kubernetes hands those jobs to control loops β probes that restart, an autoscaler that adds catalog pods, rolling updates that do not lose requests β the platform of class 11 β Cloud-Native.