9. Resilience & Chaos

At 10:00 the flash sale opens and the payment provider, under the load of every ticketing site in the country, starts answering in three seconds instead of three hundred milliseconds. Nothing is "down". Yet within a minute order stops answering everything — even GET /orders/{id}, which never talks to the provider — because every one of its threads is waiting on payment. This lab reproduces that failure on your laptop and applies the patterns of the reliability class, with a number for each.

This lab builds chapter 10 of the story of the course: the cascading failure and its defences — timeout, retry with jitter, circuit breaker and bulkhead — then a game day. It comes right after lab 8 on purpose: chaos needs observability first, so every experiment is watched on the lab 8 dashboard and paged by its burn-rate alert.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you start

Every command runs from the root of your boxoffice repository, in the state lab 8 left it:

boxoffice/
├── docker-compose.yaml     # labs 1-8: ... kafka, order, notification, otel-lgtm (+ x-otel anchor)
├── infra/                  # nginx/, postgres/, pgbouncer/, grafana/, otel/
├── load/  scripts/
└── services/
    ├── auth/  catalog/  gateway/  inventory/  notification/
    └── order/              # outbox with traceparent (lab 8)

Every file of this lab is shown in full on these pages and published under https://insper.github.io/platform/2027.1/hands-on/9/code/<path>.

Architecture

flowchart LR
    k6[k6: pay + browse] --> order["order (Resilience4j)"]
    order -->|"timeout · retry · breaker · bulkhead"| tp1["toxiproxy: payment"]
    tp1 --> payment[payment]
    payment -->|"timeouts, Idempotency-Key"| tp2["toxiproxy: psp"]
    tp2 --> psp["psp (WireMock)"]
    gm[game master] -.->|"latency, reset, outage"| tp1
    gm -.-> tp2
    order -.->|OTLP| lgtm[otel-lgtm]
    payment -.->|OTLP| lgtm

What you will measure

Prerequisites

• Labs 7 (the order service) and 8 (dashboards and alerts) done.
• Read the reliability class: resilience patterns and chaos engineering.

Steps

1. Payment, provider and Toxiproxy — the idempotent payment adapter, a WireMock provider, Toxiproxy proxies and your first toxic.
2. Resilience patterns — reproduce the cascade, then add timeout, retry, circuit breaker and bulkhead one at a time.
3. Game day — roles, hypotheses, hidden failures, timeline and a blameless review.
4. Measure it — the k6 script with a pay and a browse scenario, and the results table.

Try it in the playground

Try it: in cascading-failure, run the load profile and raise the latency of the 3rd-party node: watch the app server's utilization reach 100% and its errors spread to every request. Then set a timeout and enable the circuit breaker on the edge to the 3rd party, and run again. In retry-storm, compare retries: 3 with and without exp+jitter backoff during an outage.

The full simulator is in the playground.

Measure it

The Measure it step has the k6 script, the commands and the results table to keep in results/lab9.md; the game-day reviews go to results/lab9-gameday.md.

Troubleshooting

Clean up

curl -s -X POST localhost:8474/reset    # no toxic left behind
docker compose stop                     # keep the stack for lab 10's images
docker compose down -v                  # or remove everything

End state

boxoffice/
├── docker-compose.yaml     # + psp, toxiproxy, payment; order -> toxiproxy:18085 + PAYMENT_* knobs
├── infra/
│   ├── grafana/alerting/slo-order.yaml   # the lab 8 burn-rate alerts, for order (game day)
│   ├── psp/mappings/charge.json
│   └── toxiproxy/toxiproxy.json
├── load/pay-under-chaos.js
├── scripts/game-day.sh
├── results/lab9.md, lab9-gameday.md
└── services/
    ├── order/              # pom (+ resilience4j, aspectj), application.yaml, OrderService,
    │                       # OrderController (+ /pay), payment/ package
    └── payment/            # new: idempotent adapter to the provider (+ Dockerfile, .dockerignore)

Commit and tag: git add -A && git commit -m "lab 9: resilience" && git tag lab9.

Checklist

• payment, psp and toxiproxy run in docker-compose.yaml; order and payment reach their dependencies through Toxiproxy
• POST /payments is idempotent by Idempotency-Key, and the key is forwarded to the provider
• POST /orders/{id}/pay confirms the order (status CONFIRMED + OrderConfirmed in the outbox) and is protected by timeout, retry, circuit breaker and bulkhead
• the unprotected run shows the cascade (browse degraded); each pattern's run shows its effect, in numbers
• one game day with at least three hidden experiments, a timeline and a blameless review per round
• results table filled in results/lab9.md and explained
• self-check: curl -s -X POST localhost:8474/reset, then PAYMENT_RESILIENCE=true docker compose up -d --wait order and one pay call returns CONFIRMED; pushed with the tag lab9

Related

• Theory: Reliability · SLOs and error budgets · Resilience patterns · Chaos engineering · External integrations
• Case study: Payment system
• Project: Chaos game day
• Next lab: 10. Kubernetes adds self-healing and autoscaling.

Start: Payment, provider and Toxiproxy