System Architecture · Insper

Hands-on 9 — Resilience & Chaos

One slow payment provider must not take down the checkout

The story so far

chapter 9 · lab 8dashboards, traces, a burn-rate alert: it paged ~440 s after db stopped
→
chapter 10 · lab 9timeout, retry, breaker, bulkhead, game day
→
next · lab 10Kubernetes heals and scales

10:02: the payment provider answers in 4 s instead of 200 ms. The lab 8 dashboard shows it, the alert fires — and checkout still dies. Is seeing a failure enough to survive it?

  • In this lab, Toxiproxy adds 3 s to every call to payment.

Nothing is down. Everything is broken.

10:00, flash sale. The payment provider answers in 3 s instead of 300 ms. Within a minute order
stops answering everything — even GET /orders/{id}. Why?

  • 20 payments/s × 3 s = 60 threads just waiting on payment.
  • order has 50 → every request queues: a cascading failure.
  • Little's Law: concurrency = rate × time held.

The lab setup

k6pay 20/s + browse 20/s
→
order50 threads · Resilience4j
:18085 →
toxiproxyproxy payment
→
payment8085 · idempotent
:18095 →
toxiproxyproxy psp
→
pspWireMock
  • payment: adapter to the payment service provider (PSP), POST /payments idempotent by Idempotency-Key.
  • toxiproxy (API on 8474): Transmission Control Protocol (TCP) proxies — latency, resets, outages, two places to break.
  • New endpoint: POST /orders/{id}/pay → CONFIRMED + OrderConfirmed in the outbox.

Toxics

Toxic Simulates
latency (+ jitter) slow dependency
timeout (0) black hole
reset_peer connection reset
bandwidth thin pipe
proxy enabled: false dependency down
curl -X POST localhost:8474/proxies/payment/toxics \
  -d '{"name":"slow","type":"latency","attributes":{"latency":3000}}'

toxicity 0–1: the probability per connection. POST /reset is the abort button.

1 · Timeout

factory.setConnectTimeout(Duration.ofMillis(500));
factory.setReadTimeout(Duration.ofSeconds(1)); // 0 = forever (the default!)
  • A thread is held at most 1 s: 20/s × 1 s = 20 threads → the pool survives.
  • Slow failure → fast failure; fast failures do not cascade.

Budgets shrink at each hop: edge 10 s > gateway 5 s > order→payment 1 s × attempts.

2 · Retry, backoff, jitter

max-attempts: ${PAYMENT_RETRY_ATTEMPTS:3}
wait-duration: 200ms
enable-exponential-backoff: true      # 200 ms, 400 ms
enable-randomized-wait: true
randomized-wait-factor: 0.5           # ± 50% jitter
retry-exceptions: [ResourceAccessException, HttpServerErrorException, RestClientException]
ignore-exceptions: [HttpClientErrorException, CallNotPermittedException, BulkheadFullException]
  • Per attempt p = 0.3 → three attempts fail with p³ = 2.7%. Measured: 0.5% → 0.1% — the toxic hits only new connections.
  • Costs: up to 3× load, more latency. Only on idempotent calls.

Idempotency end to end

orderIdempotency-Key: order-<id>, same on every attempt
→
paymentkey UNIQUE · known key = first result (200)
→
providergets the same key

A timeout does not mean the call failed — the response may have been lost. The key makes the retry harmless.

3 · Circuit breaker

CLOSEDwindow of 20, min 10
proxy disabled · 10 calls fail →
OPENCallNotPermitted · 503 in µs
10 s →
HALF_OPEN3 probes
proxy back →
CLOSED
  • Opens at ≥ 50% failed or ≥ 50% slower than 800 ms — no network, no retry, no waiting.
  • Protects the caller's threads and gives the dependency room to recover.
  • Time from re-enable to CLOSED: up to 10 s plus the probes — it depends on where the wait was.

Watch it: /actuator/circuitbreakers, /actuator/circuitbreakerevents.

4 · Bulkhead

Calls succeed in 700 ms — below the timeout and the slow-call threshold. At 70/s, GET /orders/{id} slows down anyway. Why, and what fixes it?

  • 70/s × 0.7 s ≈ 49 of 50 threads wait on payment.
  • Bulkhead max-concurrent-calls: 10, max-wait-duration: 0 → the 11th call gets 503 now.
order50 threads
→
bulkhead: 10wait 700 ms on payment
+
the other 40GET /orders/{id} in ms
+
the rest503 at once (~85%)

How the annotations stack

@Retry(name = "payment", fallbackMethod = "unavailable")
@CircuitBreaker(name = "payment")
@Bulkhead(name = "payment")
public PaymentResponse charge(UUID orderId, BigDecimal amount) { ... }
Retry3 attempts
→
CircuitBreakeropen: fail fast
→
Bulkheadfull: fail fast
→
call
any refusal →
fallback: 503
  • Retry ignores CallNotPermittedException and BulkheadFullException; the breaker ignores BulkheadFullException.

Never wait inside a transaction

  • pay() calls payment with no transaction open.
  • Then a short transaction: UPDATE … WHERE status = 'PENDING_PAYMENT' + outbox event.
  • Waiting 3 s inside a transaction moves the exhaustion to the connection pool (≈ 10).

Simulator: the five runs, without the stack

timeout, beforebrowse med 4.75 s · 65 iterations dropped
→
after (1 s)pay fails in 1.01 s · browse p95 8 ms
retry0.3% → 0.0% failed
breakeropens 0.3 s · closes 61.1 s
bulkhead58% fail fast, browse p95 759 ms → 81% rejected, 8 ms
  • The patterns page's pay-under-chaos simulator: pick a step, before or after its pattern, and read k6's summary.
  • The model's numbers are not yours — compare the shape with your own runs.

Measure it

Experiment Toxic Before → after
Timeout latency 3 s pay p95, browse p95
Retry reset_peer 30% confirmed %
Circuit breaker proxy disabled pay p95, pay_fast_fail
Bulkhead latency 700 ms, 70 pay/s browse p95, rejected %

A test run (45 s, payment + 3 s): browse p50 9.04 s unprotected, 6.5 ms with every pattern — reproduce the contrast, not the numbers.

k6 load/pay-under-chaos.js: pay (uses payment) and browse (does not) at the same time — the second one reveals the cascade.

5 · Game day

game masterinjects, hidden
→
alert fires
→
operatorsdashboard → trace → logs
→
mitigate first
→
scribedetect / mitigate times
  • Before: steady state, hypotheses, blast radius, abort = POST /reset; the burn-rate rules for order.
  • After: blameless review — time to detect, time to mitigate, action items. Rotate roles.

No hypothesis, no experiment. A refuted hypothesis is the valuable result.

Try it in the playground

  • boxoffice-payment: this lab — at t = 60 s payment gets +3 s; add timeout, retries, breaker and a bulkhead of 0.2.
  • cascading-failure: raise the 3rd-party latency, watch utilization hit 100%; add timeout + breaker.
  • retry-storm: retries without backoff amplify an outage; add exp+jitter.

The simulator shows the shape; k6 + Toxiproxy give you your own numbers.

What breaks next

lab 9 · containedGET /orders/{id}: 9 s median unprotected, milliseconds with the patterns
→
the limitevery game-day mitigation was a person at a terminal
→
lab 10 · Kubernetesrestart, scale before 10:00, roll out: control loops
  • At 10:00, nobody should be restarting containers by hand.

Takeaways

  1. A slow dependency is worse than a dead one: it exhausts threads (Little's Law).
  2. Timeouts everywhere, shorter at each inner hop.
  3. Retries with backoff + jitter, only on transient errors, only with idempotency keys.
  4. Circuit breakers fail fast and let dependencies recover; bulkheads cap concurrency per dependency.
  5. Never hold a database transaction across a remote call.
  6. Practice failure: game days with hypotheses and blameless reviews.

Chapter 10 of the story of the course. Outline: (1) the cascade; (2) payment, psp, Toxiproxy; (3) timeout; (4) retry + idempotency; (5) circuit breaker; (6) bulkhead; (7) the five runs in the simulator; (8) game day; (9) measure.

The story of the course (2027.1 overview, "The story of the course"): chapter 10 opens with the slow provider that lab 8's dashboard can see but not stop.