Hands-on 9 — Resilience & Chaos

Nothing is down. Everything is broken.

• 20 payments/s × 3 s = 60 threads just waiting on payment.
• order has 50 → every request queues: a cascading failure.
• Little's Law: concurrency = rate × time held.

The lab setup

• order → toxiproxy:18085 → payment → toxiproxy:18095 → psp.
• New endpoint: POST /orders/{id}/pay → CONFIRMED + OrderConfirmed in the outbox.

Toxics

curl -X POST localhost:8474/proxies/payment/toxics \
  -d '{"name":"slow","type":"latency","attributes":{"latency":3000}}'

toxicity 0–1: the probability per connection. POST /reset is the abort button.

1 · Timeout

factory.setConnectTimeout(Duration.ofMillis(500));
factory.setReadTimeout(Duration.ofSeconds(1)); // 0 = forever (the default!)

• A thread is held at most 1 s: 20/s × 1 s = 20 threads → the pool survives.
• Slow failure → fast failure; fast failures do not cascade.

2 · Retry, backoff, jitter

max-attempts: 3
wait-duration: 200ms
enable-exponential-backoff: true      # 200 ms, 400 ms
enable-randomized-wait: true
randomized-wait-factor: 0.5           # ± 50% jitter
retry-exceptions: [ResourceAccessException, HttpServerErrorException]

• Transient error with p = 0.3 → three attempts fail with p³ = 2.7%.
• Costs: up to 3× load, more latency. Only on idempotent calls.

Idempotency end to end

• order sends Idempotency-Key: order-<id> on every attempt.
• payment stores it UNIQUE; a known key returns the first result (200).
• payment forwards it to the provider.

3 · Circuit breaker

• Closed: calls flow; failures and slow calls are counted (window of 20, min 10).
• Open (≥ 50% failed or slow): fail immediately for 10 s — no network, no retry.
• Half-open: 3 trial calls decide → closed or open again.
• Protects the caller's threads and gives the dependency room to recover.

Watch it: /actuator/circuitbreakers, /actuator/circuitbreakerevents.

4 · Bulkhead

• 70/s × 0.7 s ≈ 49 of 50 threads wait on payment.
• Bulkhead max-concurrent-calls: 10, max-wait-duration: 0 → the 11th call gets 503 now.
• The other 40 threads keep serving everything else.

How the annotations stack

@Retry(name = "payment", fallbackMethod = "unavailable")
@CircuitBreaker(name = "payment")
@Bulkhead(name = "payment")
public PaymentResponse charge(UUID orderId, BigDecimal amount) { ... }

• Default order: Retry ( CircuitBreaker ( Bulkhead ( call ) ) ).
• Fallback on the outermost layer → runs after retries, when open, when full.
• Retry ignores CallNotPermittedException and BulkheadFullException: fail fast.
• The breaker ignores BulkheadFullException: our refusal is not the dependency's fault.

Never wait inside a transaction

• pay() calls payment with no transaction open.
• Then a short transaction: UPDATE … WHERE status = 'PENDING_PAYMENT' + outbox event.
• Waiting 3 s inside a transaction moves the exhaustion to the connection pool (≈ 10).

5 · Game day

• Roles: game master (injects, hidden), operators (detect, mitigate), scribe (timeline).
• Before: steady state, hypotheses, blast radius, abort = POST /reset.
• During: alert → dashboard → trace → logs; mitigate first.
• After: blameless review — time to detect, time to mitigate, action items.

Try it in the playground

• cascading-failure: raise the 3rd-party latency, watch utilization hit 100%; add timeout + breaker.
• retry-storm: retries without backoff amplify an outage; add exp+jitter.

Takeaways

1. A slow dependency is worse than a dead one: it exhausts threads (Little's Law).
2. Timeouts everywhere, shorter at each inner hop.
3. Retries with backoff + jitter, only on transient errors, only with idempotency keys.
4. Circuit breakers fail fast and let dependencies recover; bulkheads cap concurrency per dependency.
5. Never hold a DB transaction across a remote call.
6. Practice failure: game days with hypotheses and blameless reviews.