Resilience Patterns
When a dependency degrades, the naive default β keep sending requests and wait for the answer β reliably produces the cascade of the overview: the caller's threads block, its queue fills, and the failure climbs to users who had nothing to do with the broken dependency. Resilience patterns are design decisions that interrupt this propagation. They do not prevent failures; they limit how far a failure travels1.
Throughout this page we follow one call of BoxOffice: order calls payment, the adapter in front of the external payment provider, to charge a fan's card during the purchase saga (see Messaging β patterns). It is the most dangerous call in the system: it is on the revenue path, it crosses the internet to a third party, and the third party's bad day becomes ours.
| Pattern | Question it answers | Protects against |
|---|---|---|
| Timeout | How long am I willing to wait? | Hanging calls holding resources |
| Retry with backoff and jitter | Is it worth trying again β when? | Transient failures |
| Circuit breaker | Should I even try? | Hammering a dependency known to be down |
| Fallback | What do I answer instead? | Losing a whole page for one feature |
| Bulkhead | How much of my capacity may one dependency take? | Exhaustion spreading between features |
| Load shedding | Which work do I refuse when overloaded? | Collapse under overload |
| Rate limiting | How much does each client get? | Abusive clients, bots, spikes |
| Health checks | Should traffic reach this instance? | Routing to broken instances |
Timeout
A timeout is a deadline on an outbound call: if the answer has not arrived in time, the caller abandons it and handles the error. Without a timeout, a hanging dependency holds a thread (and a connection, and memory) indefinitely; with one, the damage a single slow call can do is bounded.
sequenceDiagram
participant O as order
participant P as payment adapter
participant X as payment provider
O->>+P: POST /charges (deadline 2 s)
P->>+X: charge card
Note over X: provider is slow today
Note over O: 2 s elapsed: give up, free the thread
O-->>O: handle the error (retry? fallback?)
X-->>-P: (answer arrives at 4 s)
P-->>-O: (too late: discarded) How to choose the value:
- From the dependency's latency distribution. Set it a little above the p99.9 of normal operation: anything slower is already abnormal. For
payment, which normally answers in about 200 ms with a p99.9 well under a second, lab 9 reads with a 1 s timeout. - Two timeouts per HTTP call: the connect timeout (opening the TCP/TLS connection β short, a few hundred milliseconds) and the read timeout (waiting for the response).
- Per dependency. A cache lookup, a database query and a third-party API have latencies that differ by orders of magnitude. One global timeout is always wrong for most of them.
- Inside the caller's budget. If the fan's request has 5 s at the gateway, everything below must fit in it, including retries.
Deadline propagation
Timeouts must shrink as a request goes deeper. A deadline set at the edge and propagated with the request (gRPC does this natively with deadlines; over HTTP, a header such as the remaining budget) lets each service know how much time is left, and stop working on a request whose caller has already given up.
| Layer | Budget | Spent on |
|---|---|---|
| Edge (Nginx) | 10 s | everything below |
| Gateway | 5 s | everything below + its own work |
order β payment | 1 s per attempt Γ 3 attempts + backoff (200 ms, 400 ms Β± 50%) β 3.6 s | the charge |
payment β provider | 2 s (PSP_READ_TIMEOUT) | the provider call |
sequenceDiagram
participant G as gateway (5 s)
participant O as order
participant P as payment adapter
participant X as provider
G->>O: POST /orders/{id}/pay, 5 s left
O->>P: charge, attempt 1 (read timeout 1 s)
P->>X: charge card (read timeout 2 s)
Note over O,P: 1 s with no answer: order gives up
O->>O: backoff of about 200 ms
O->>P: charge, attempt 2 (same Idempotency-Key)
P->>X: charge card
X-->>P: approved after 400 ms
P-->>O: AUTHORIZED
O-->>G: CONFIRMED after about 1.7 s, inside 5 s Each layer spends part of the budget and hands the rest down: even three attempts (about 3.6 s) fit in the gateway's 5 s, and a fourth (about 5.4 s) would not. One value in the lab breaks the rule: payment waits up to 2 s for the provider, longer than order's 1 s attempt, so when the provider is slow payment keeps working on a charge order already abandoned β harmless only because of the idempotency key. In production, give the adapter less time than its caller.
Timeouts are only the first step
A timeout converts a hang into a fast error. You still need to decide what to do with the error: retry it (if transient and safe), stop calling for a while (circuit breaker), or answer something else (fallback). And a timeout on a non-idempotent call leaves an unknown outcome β the charge may have succeeded β which is why payment calls carry an idempotency key (see Integrations).
In the BoxOffice code: the timeouts of order β payment (lab 9)
Hands-on 9 β Resilience patterns gives the client of payment its two timeouts: 500 ms to connect and 1 s to read (PAYMENT_READ_TIMEOUT; 0 waits forever, the "before" experiment). The idempotency key order-<id> is the same on every attempt, which is what makes the retry of the next section safe. The lab's arithmetic: 20 payments/s against a provider 3 s slow hold 20 Γ 3 = 60 threads, more than the pool of 50; with the 1 s timeout, 20 Γ 1 = 20.
public PaymentClient(RestClient.Builder builder,
@Value("${boxoffice.payment.url}") String url,
@Value("${boxoffice.payment.connect-timeout}") Duration connectTimeout,
@Value("${boxoffice.payment.read-timeout}") Duration readTimeout) {
SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
factory.setConnectTimeout(connectTimeout);
factory.setReadTimeout(readTimeout); // 0 = wait forever (the "before" experiment)
this.http = builder.baseUrl(url).requestFactory(factory).build();
}
public PaymentResponse charge(UUID orderId, java.math.BigDecimal amount) {
return http.post()
.uri("/payments")
.header("Idempotency-Key", "order-" + orderId) // same key on every retry
.body(new PaymentRequest(orderId, amount))
.retrieve()
.body(PaymentResponse.class);
}
Retry with backoff and jitter
A retry repeats a failed call, betting that the failure was transient: a dropped connection, a pod restarting, a leader election. Retries turn many small blips into invisible ones. Done carelessly, they are also the fastest way to turn a small problem into an outage.
What to retry
| Failure | Retry? | Why |
|---|---|---|
| Connection refused / reset, connect timeout | β Yes | Typical transient failure; the request probably never ran |
| Read timeout | β οΈ Only if the operation is idempotent (or has an idempotency key) | The request may have run |
503 Service Unavailable, 429 Too Many Requests | β
Yes, with backoff; respect Retry-After | The server is asking you to come back later |
500 Internal Server Error | β οΈ At most once | Often a bug that fails again |
400, 401, 403, 404, 409, 422; "card declined" | β Never | The request itself is wrong or the answer is final |
Exponential backoff and jitter
Retrying immediately hits a struggling dependency again before it had any chance to recover. Exponential backoff waits longer after each failure:
The waits add up: what matters to the caller is the time since the first failure. With base 1 s and a cap of 16 s:
| Before retry | Wait | Time since the first failure |
|---|---|---|
| 1st (\(n\) = 0) | 1 s | 1 s |
| 2nd (\(n\) = 1) | 2 s | 3 s |
| 3rd (\(n\) = 2) | 4 s | 7 s |
| 4th (\(n\) = 3) | 8 s | 15 s |
| 5th and later | 16 s (the cap) | +16 s each |
Lab 9 uses base 200 ms and up to 3 attempts β waits of 200 ms then 400 ms, the last retry 600 ms after the first failure (plus each attempt's own time). But if many clients failed at the same moment (the provider blipped for everyone), they all retry at the same moments too β synchronized waves. Jitter randomizes the waits; Marc Brooker's analysis at AWS recommends full jitter2:
Retries amplify load
The simulation below has two experiments with illustrative numbers. On the left, steady purchase traffic (1,000 requests/s) meets a payment provider that degrades for 5 seconds; an overloaded server wastes part of its capacity on requests that time out anyway. On the right, 1,000 clients fail at the same instant (say, a connection reset during a failover) and retry against a server that can take 150 requests per 100 ms.
- Left β retries can keep a system down. Without retries (grey), successes drop during the degradation and come back the moment it ends. With 3 retries per request (red), each failure becomes up to 4 calls; the extra load keeps the provider overloaded after the trigger is gone, and the system settles in a degraded state β a metastable failure (see the overview). Backoff and jitter do not fix this: they spread retries in time but do not reduce their number. A retry budget (blue) β retries may add at most 10% to the normal traffic β caps the amplification and the system recovers immediately.
- Right β jitter prevents synchronized waves. Without jitter (orange), all clients that failed together retry together: 1,000 calls in one 100 ms slot, only 150 succeed, the other 850 retry together againβ¦ after 15 seconds 250 clients still have not succeeded. With full jitter (blue), the same retries are spread below the capacity line and everyone succeeds in about 1.5 s.
Three rules follow:
- Retry at one layer only. If the gateway,
orderand thepaymentadapter each make up to 3 attempts, one fan click can become \(3 \times 3 \times 3 = 27\) calls to the provider. Retry close to the failing dependency and let upper layers fail fast. - Cap attempts and total time: few attempts (1β2 retries), all inside the caller's deadline.
- Use a retry budget (Envoy, gRPC retry throttling, Finagle): retries are allowed only while they are a small fraction of requests. When the dependency is really down, retries stop by themselves.
Try it in the playground
The retry-storm scenario has clients retrying failed calls with no backoff against a service that suffers a short outage. Press Run and watch the request rate to the service jump to several times the client rate during the outage, and the service stay overloaded after it should have recovered. Then switch the retry policy to exponential backoff with jitter and add a retry budget, and compare the recovery time.
Never retry a non-idempotent call without an idempotency key
Retrying POST /charges after a read timeout can charge the fan twice: the first attempt may have succeeded. The order service sends an Idempotency-Key (the order id) with every charge; the adapter and the provider return the stored result for a key they have already seen. See Messaging β delivery semantics and Integrations.
Circuit breaker
A circuit breaker watches the outcomes of calls to a dependency. When the failure rate crosses a threshold, it opens and rejects calls immediately, without touching the network, for a while; then it lets a few probe calls through to test recovery3.
stateDiagram-v2
[*] --> Closed
Closed --> Open : failure or slow-call rate β₯ 50% (last 20 calls)
Open --> HalfOpen : after 10 s
HalfOpen --> Closed : the 3 probes' rates below the thresholds
HalfOpen --> Open : failure or slow-call rate of the probes β₯ 50% Closed (normal) : Calls pass through; the breaker records successes, failures and slow calls in a sliding window.
Open (failing fast) : Calls fail immediately (CallNotPermittedException in Resilience4j). The caller's threads are free, and the dependency gets breathing room instead of a retry storm.
Half-open (testing) : After the wait duration, a few probe calls are allowed and every other call is still rejected. When the probes are done, the breaker computes their failure and slow-call rates: below the thresholds, it closes; otherwise it opens for another wait.
| Parameter (Resilience4j property) | Resilience4j default | Lab 9 value for payment | Effect |
|---|---|---|---|
Sliding window (sliding-window-size) | 100 calls | last 20 calls (count-based) | How much history decides |
Minimum calls (minimum-number-of-calls) | 100 | 10 | No decision before this many calls β avoids opening on 1 failure out of 2 |
Failure-rate threshold (failure-rate-threshold) | 50% | 50% | Opens when the share of failed calls is equal to or above it |
Slow calls (slow-call-duration-threshold, slow-call-rate-threshold) | 60 s, 100% | 800 ms, 50% | A slow provider also opens the breaker β essential against gray failures |
Wait in open state (wait-duration-in-open-state) | 60 s | 10 s | Time the dependency gets to recover β and how long fans see fail-fast |
Calls in half-open (permitted-number-of-calls-in-half-open-state) | 10 | 3 | How carefully recovery is tested |
The defaults are conservative: with a window of 100 calls, a breaker that sees 1 call per second needs well over a minute to decide. Slow calls have their own rate threshold: with the default 100%, they open the breaker only when every call in the window is slow, and with the default 60 s duration a call behind a 1 s read timeout never counts as slow. Lab 9 sets the slow-call threshold to 800 ms, below its 1 s read timeout: a call slower than the timeout never completes β it becomes a failure β so a threshold above the timeout could never see a slow call, and the breaker would wait for timeouts instead of reacting to the slowness first.
Which errors count matters: a declined card is a correct answer from a healthy provider, not a failure. Configure it as ignored, or a wave of declined cards on a stolen-card attack would open the breaker for every honest fan. Lab 9 has no card declines; its retry ignores every 4xx (HttpClientErrorException), because the same request would fail again, and its breaker ignores BulkheadFullException, a refusal of our own that says nothing about payment.
The wait is a trade-off you can compute. With 10 s, a provider that recovers one second after the breaker opened still costs fans about 9 s of "payment temporarily unavailable", until the half-open probes see it healthy; a provider that stays down keeps the breaker open in 10 s steps, each costing only 3 probe calls. A longer wait protects a struggling provider better and fails more fans after it recovers.
Fallback and graceful degradation
Failing fast is better than hanging, but an error is still an error. A fallback is what the caller does instead. The goal is graceful degradation: lose a feature, not the page.
| Dependency down | BoxOffice fallback |
|---|---|
| Recommendations ("fans also bought") | Hide the widget, or show a static "popular this week" list from the cache |
| Search | Serve the last cached results; show "results may be a few minutes old" |
| Seat-map rendering service | Show the best-available selector ("2 seats, sector B") instead of the interactive map |
notification | Nothing to do synchronously: the event is already in Kafka and will be delivered later |
payment | No fake success. Keep the seats held, tell the fan "payment temporarily unavailable β your seats are held for 10 minutes, try again", and let the saga retry or release |
A fallback must be safe and simple
A fallback that calls another remote service can fail too and needs its own protection. Some operations β charging a card, confirming a seat β must never be faked: failing clearly is the graceful behaviour. And test your fallbacks; a fallback first exercised during an outage usually fails.
Bulkhead
A bulkhead β named after the watertight compartments of a ship β gives each dependency its own limited pool of threads, connections or concurrent calls, so one slow dependency cannot consume the capacity of the others.
flowchart LR
req["order: 50 Tomcat threads"] --> bp["payment bulkhead<br/>max 10 concurrent, no queue"]
req --> rest["the other 40 threads<br/>POST /orders, GET /orders/{id}"]
bp --> pay["payment<br/>(slow: 3 s)"]
bp -.->|"11th concurrent call: 503 at once"| fb["fallback"]
rest --> db[("orders database<br/>(healthy)")] Sizing comes from Little's Law again. At lab 9's 20 payments/s and 200 ms per charge, about 4 calls are in flight. A bulkhead of 10 leaves room for latency to grow 2.5Γ (to 0.5 s) before calls are rejected; when the provider goes to 3 s, at most 10 threads wait on it (instead of 60, more than the pool), and the other 40 keep creating and reading orders. With max-wait-duration: 0 the 11th concurrent call does not queue: it fails in microseconds and goes to the fallback.
The cost of a bulkhead is capacity that cannot be shared: with the payment bulkhead full, an 11th payment is refused at once even while other threads sit idle, whereas one shared pool would have served it. That refusal is the point β the idle threads are kept for the routes that work.
Resilience4j offers a semaphore bulkhead (limits concurrent calls on the caller's threads; cheap, the default) and a thread-pool bulkhead (runs calls on a dedicated pool). The idea is general: separate connection pools per workload in a database, separate Kafka consumer groups, separate deployments for critical and non-critical traffic, separate queues for confirmations and marketing e-mails.
Load shedding
When demand exceeds capacity, some requests will fail. Load shedding decides which and makes them fail early and cheaply, so the rest succeed. Without it, the server accepts everything, every request gets slow, most time out after consuming resources, and goodput collapses β everyone fails4.
- Shed at the door. Reject with
503+Retry-Afterbefore doing any work: a rejection costs microseconds, a timed-out request costs its full processing. - Shed by priority. During the stadium on-sale, BoxOffice classifies traffic: purchase and seat-hold requests are critical; catalog browsing is important; recommendations, autocomplete and "related events" are sheddable. Under overload, the sheddable go first.
- Shed stale work. A request that already waited longer than its caller's deadline is useless; drop it from the queue instead of processing it (bounded queues with deadlines, last in, first out (LIFO) under overload).
- Adapt the limit. Instead of a fixed concurrency limit, adaptive limiters (inspired by TCP congestion control, e.g. Netflix's concurrency-limits library) lower the number of concurrent requests when latency rises and raise it when latency is healthy.
BoxOffice: the waiting room is load shedding
The flash-sale waiting room is load shedding done kindly: instead of rejecting fans, it admits them at the rate the system can serve and tells the others their position. Behind it, the gateway still sheds sheddable routes when order latency rises, as a second line of defence.
Rate limiting
Rate limiting caps how many requests a client (an IP, a user, an API key, a partner) may send in a time window. Load shedding protects the server from total overload; rate limiting protects it from individual clients β bots refreshing the seat map hundreds of times per second, a partner's buggy integration, a scraper.
The common algorithm is the token bucket: tokens refill at a steady rate (the sustained limit), the bucket's size allows short bursts, and each request takes a token; an empty bucket means 429 Too Many Requests. Limits per user or API key live in the API gateway, backed by a shared store such as Redis, where the authenticated identity is known. Algorithms (token bucket, leaky bucket, sliding window) and their implementation are covered in Edge β API gateway; the rate-limiter playground scenario shows a gateway protecting a backend.
Health checks
Health checks let orchestrators and load balancers stop sending traffic to broken instances β automatic failover at the smallest scale.
Liveness : Is the process alive and not stuck? If it fails, Kubernetes restarts the container.
Readiness : Can this instance serve traffic now? If it fails, the instance is removed from the load balancer without being restarted (warming up, overloaded, lost its own database connection).
Startup : Has the application finished starting? Liveness and readiness are not checked until it succeeds, so a slow-starting JVM is not killed during boot.
Spring Boot Actuator serves the probes at /actuator/health/liveness and /actuator/health/readiness. In Spring Boot 4 (the version of the labs) these probe groups are on by default, in any environment; in Spring Boot 2.3 to 3.x they were automatic only on Kubernetes, and elsewhere β Docker Compose, for instance β needed management.endpoint.health.probes.enabled=true (lab 9 still sets it, which is harmless). By default they reflect only the application's own state: a dependency appears in readiness only if you add it.
stateDiagram-v2
[*] --> Starting
Starting --> Ready : startup passes, then readiness passes
Starting --> Restarted : startup fails 30 times (150 s)
Ready --> Unready : readiness fails twice
Unready --> Ready : readiness passes again
Ready --> Restarted : liveness fails 3 times
Unready --> Restarted : liveness fails 3 times
Restarted --> Starting
note right of Unready : out of the load balancer, not restarted # order Deployment, container spec (Spring Boot Actuator endpoints)
startupProbe:
httpGet: { path: /actuator/health/liveness, port: 8080 }
periodSeconds: 5
failureThreshold: 30 # up to 150 s to start
livenessProbe:
httpGet: { path: /actuator/health/liveness, port: 8080 }
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: 8080 }
periodSeconds: 5
failureThreshold: 2
Never put dependencies in the liveness check
If order's liveness checks the payment provider and the provider blips, Kubernetes restarts every order replica at once β turning a third-party hiccup into a full BoxOffice outage while the JVMs boot. Liveness answers only "is this process stuck?". Be careful with readiness too: if every replica depends on the same database, they all become unready together and the service vanishes instead of returning clear errors.
Putting it together
The patterns are layered. For the order β payment call, BoxOffice uses all of them, configured with Resilience4j5 in lab 9. The numbers are laptop-sized β 20 payments/s against 50 Tomcat threads β and every knob reads an environment variable, so each experiment switches one pattern off without a rebuild.
In the BoxOffice code: the protection stack on order (lab 9)
Hands-on 9 β Resilience patterns configures each layer, and each value has a reason:
- Timeouts: 500 ms to connect and 1 s to read (
PAYMENT_READ_TIMEOUT;0waits forever, the "before" experiment) β about five times the provider's normal 200 ms. - Retry: 3 attempts (1 call + 2 retries), 200 ms then 400 ms of backoff, each Β± 50% (jitter), on I/O errors, 5xx and connection resets only; a 4xx is final, and an open breaker (
CallNotPermittedException) or a full bulkhead (BulkheadFullException) fails fast instead of being retried. - Circuit breaker: count-based over the last 20 calls (at least 10), opens when 50% failed or took longer than 800 ms β a slow provider trips it too β, stays open 10 s, then lets 3 probes decide; it ignores
BulkheadFullException. - Bulkhead: 10 concurrent calls of the 50 threads,
max-wait-duration: 0β full means 503 at once.
PaymentGateway stacks the annotations in Resilience4j's default order β Retry ( CircuitBreaker ( Bulkhead ( call ) ) ) β with the fallback on the outer layer. The fallback answers 503 and the order stays PENDING_PAYMENT, the honest "try again" of Fallback. With the provider 3 s slow, the lab measured GET /orders/{id} β which never calls payment β at a median of 9.04 s unprotected and 6.5 ms with the patterns.
payment: # lab 9
url: ${PAYMENT_URL:http://payment:8085}
resilience: ${PAYMENT_RESILIENCE:true} # false = call payment with no Resilience4j at all
connect-timeout: 500ms
read-timeout: ${PAYMENT_READ_TIMEOUT:1s} # 0 = wait forever
resilience4j:
retry:
instances:
payment:
max-attempts: ${PAYMENT_RETRY_ATTEMPTS:3} # 1 = no retry
wait-duration: 200ms
enable-exponential-backoff: true
exponential-backoff-multiplier: 2 # 200 ms, 400 ms ...
enable-randomized-wait: true
randomized-wait-factor: 0.5 # ... each +/- 50% (jitter)
retry-exceptions:
- org.springframework.web.client.ResourceAccessException # timeouts, connection refused/reset
- org.springframework.web.client.HttpServerErrorException # 5xx
- org.springframework.web.client.RestClientException # connection reset while reading the response
ignore-exceptions:
- org.springframework.web.client.HttpClientErrorException # 4xx: the same request fails again
- io.github.resilience4j.circuitbreaker.CallNotPermittedException # circuit open: fail fast
- io.github.resilience4j.bulkhead.BulkheadFullException # bulkhead full: fail fast
circuitbreaker:
instances:
payment:
sliding-window-type: COUNT_BASED
sliding-window-size: 20
minimum-number-of-calls: 10
failure-rate-threshold: 50 # open when >= 50% of the last 20 calls failed...
slow-call-duration-threshold: 800ms
slow-call-rate-threshold: 50 # ... or >= 50% were slower than 800 ms
wait-duration-in-open-state: 10s
permitted-number-of-calls-in-half-open-state: 3
automatic-transition-from-open-to-half-open-enabled: true
register-health-indicator: true
ignore-exceptions:
- io.github.resilience4j.bulkhead.BulkheadFullException # our own refusal is not the dependency's fault
bulkhead:
instances:
payment:
max-concurrent-calls: ${PAYMENT_BULKHEAD:10} # at most 10 of the 50 threads wait on payment
max-wait-duration: 0 # full = reject immediately
@Retry(name = "payment", fallbackMethod = "unavailable")
@CircuitBreaker(name = "payment")
@Bulkhead(name = "payment")
public PaymentResponse charge(UUID orderId, BigDecimal amount) {
return client.charge(orderId, amount);
}
Try it
It is 10:02 and the provider answers in 4 s instead of 200 ms. The switches use the values of the box above (a 1 s read timeout; 3 attempts with 200 ms Γ 2 backoff Β± 50%; a breaker over the last 20 calls that opens at 50% failed or slower than 800 ms and waits 10 s). Turn them on one at a time, restarting each run:
- Nothing on: 20 payments/s Γ 4 s needs 80 threads; all 50 are busy within seconds and, in 30 s, about 220 payments find no thread at all β nor would
GET /orders. - Timeout: only about 20 threads are busy, but 0% of the payments succeed. The timeout protects
order, not the payment. - + retries: the provider now receives about 2Γ the payments (44 calls/s) exactly while it is struggling, and the 50 threads are full again β 3 attempts of 1 s plus backoff hold each thread for about 3.6 s. A timeout plus retries without a breaker is worse than nothing: after 30 s, 0 payments succeed per 10 s (128 did with nothing on, slowly), about 200 have found no thread, and the provider takes twice the load.
- + circuit breaker: it opens 1.6 s into the run; payments fail fast, threads drop to about 1 and the provider gets no calls. Drag the latency back to 200 ms: 10 s after opening, the breaker goes half-open, 3 probes succeed and it closes. Red dots in its window are failures, orange ones slow calls.
The annotations decide the path of every call. Read it from the left: the retry wraps the breaker, which wraps the bulkhead, which wraps the HTTP call β and every refusal ends in the same fallback.
flowchart LR
call(["charge"]) --> R["Retry<br/>3 attempts, 200 ms Γ 2 Β± 50%"]
R --> CB{"circuit breaker<br/>lets it through?"}
CB -->|"open: CallNotPermitted"| FB["fallback<br/>payment unavailable, seats held"]:::problem
CB -->|"closed or a half-open probe"| BH{"bulkhead<br/>free permit?"}
BH -->|"full: BulkheadFull"| FB
BH -->|"yes"| HTTP["HTTP call<br/>connect 500 ms, read 1 s"]
HTTP -->|"timeout, 5xx or reset: counted by the breaker"| R
HTTP -->|"2xx"| OK(["charged"]):::comp
R -->|"attempts used up"| FB How the layers interact β check these before shipping:
- Order of the decorators. Resilience4j's Spring Boot default is
Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( call ) ) ) ) ): the retry is outermost, so every attempt is counted by the breaker and an open breaker makes the retry fail immediately. Make sure retries never bypass the breaker. Put the fallback on the outermost layer (the retry): a fallback on the breaker turns every failure into a normal answer, so the retry never sees an error and never retries. - Time budget. 3 attempts Γ 1 s + backoffs of about 200 ms and 400 ms β 3.6 s, inside the gateway's 5 s.
- Metrics. Resilience4j exports breaker state, bulkhead saturation and retry counts through Micrometer; graph them next to the service level objective (SLO) dashboard (Hands-on 8), because a breaker that opens is an incident in progress.
- Proof. None of this is trustworthy until you have injected the failure and watched it work β chaos engineering and Hands-on 9.
Try it in the playground
Open the cascading-failure scenario in full mode. Run it once as it is (no protection) and note the error rate of the healthy routes. Then add, one at a time, a timeout, a circuit breaker and a bulkhead on the 3rd-party call, running after each change. Which one protects the healthy routes the most? What does each cost the requests that need the 3rd party?
Try it in the playground: lab 9's numbers
The boxoffice-payment preset is Hands-on 9 in the playground: pay-under-chaos.js sends 20 payments/s and 20 browse requests/s to order, which has 50 Tomcat threads, and at t = 60 s Toxiproxy adds 3 s to every call to payment. Run it as it is: 20 payments/s Γ 3 s = 60 calls in flight > 50 threads, and browsing, which never touches the provider, fails too. Then protect the edge order β payment the way the lab does β a 1 000 ms timeout, 3 attempts with jittered backoff, the circuit breaker and a bulkhead of 0.2 of the threads (10 of 50) β and run again: payments fail fast during the incident, and browsing keeps its threads.
The layers also have an order, outermost first, and each position has a reason:
| Layer, outermost first | Handles | Why in this position | BoxOffice setting |
|---|---|---|---|
| Rate limiting, load shedding (gateway) | Abusive clients, overload | Refuses excess work before it costs anything | per-account token bucket; recommendations and autocomplete shed first |
| Retry + backoff + jitter, with the fallback | Transient errors | Outside the breaker: an open breaker stops the retries at once; the fallback runs once, after everything inside gave up | 3 attempts, 200 ms Γ 2 Β± 50%, idempotency key; 503, seats held |
| Circuit breaker | A dependency known to be failing or slow | Sees every attempt, timeouts included, as a call to judge | 50% failures or calls over 800 ms in the last 20, open 10 s |
| Bulkhead | One slow dependency eating all threads | Caps the threads the attempts can hold | 10 concurrent payment calls of 50 threads, no queue |
| Timeout | Unbounded waiting | Innermost: bounds each attempt, not the sum | 500 ms connect, 1 s read |
This is Resilience4j's default aspect order in Spring Boot, Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( call ) ) ) ) ) β in lab 9 the read timeout of the HTTP client plays the time limiter's role, inside the call. Microsoft's standard resilience handler for .NET's HttpClient stacks the same way: rate limiter β total timeout β retry β circuit breaker β attempt timeout6. Whatever the library, check that retries do not bypass the breaker and that the timeout bounds each attempt.
In the interview
When a design has a call to a third party or a critical dependency, volunteer the protection stack in one breath: "timeout at roughly the p99.9, one or two retries with jittered backoff and an idempotency key, a circuit breaker that also counts slow calls, a bulkhead sized with Little's Law, and a fallback that tells the user the truth". Then justify one number (the bulkhead or the timeout) with arithmetic. If asked about overload, distinguish rate limiting (per client) from load shedding (whole system, by priority), and mention retry amplification across layers.
Best practices
- Derive each timeout from the dependency's measured p99.9, and keep it below the caller's remaining budget β the budget shrinks at every hop (edge 10 s > gateway 5 s > one attempt 1 s). In BoxOffice: a 1 s read timeout on
payment(box above). - Size each bulkhead with Little's Law and recompute it when latency or traffic changes β 20 payments/s Γ 0.2 s = 4 calls in flight, so a bulkhead of 10 leaves room for 2.5Γ the latency. In BoxOffice: 10 of the 50 threads of
order(Hands-on 9). - Make every resilience parameter configuration, not code β during an incident you raise a timeout or shrink a bulkhead with a redeploy of config, not a build. In BoxOffice:
PAYMENT_READ_TIMEOUT,PAYMENT_RETRY_ATTEMPTS,PAYMENT_BULKHEADinapplication.yaml. - Cap retries with a budget, not only with a count β retries adding at most 10% to the normal traffic let the provider recover in the simulation above, where 3 retries per call kept it down.
- Publish breaker state, bulkhead permits and retry counts as metrics and alert on an open breaker β an open breaker is an incident in progress. In BoxOffice:
/actuator/circuitbreakersand the breaker in/actuator/health(lab 9). - Return the fallback as an explicit state the client can act on β "payment unavailable, seats held" lets the fan retry; a generic 500 does not. In BoxOffice:
503with the order kept inPENDING_PAYMENT.
Common pitfalls
- Default timeouts of minutes (or none) in HTTP and database clients.
- Retries at every layer β 3 Γ 3 Γ 3 = 27 calls per click to a struggling service.
- Retrying non-idempotent calls without an idempotency key: double charges.
- Counting business errors as failures: declined cards opening the breaker for everyone.
- Breakers that only count errors, not slowness: a provider at 4 s per call never trips them.
- Dependencies in liveness probes: a third-party blip restarts the whole fleet.
- Untested fallbacks that fail the first time they are needed.
How it connects
flowchart LR
SLOW["slow dependency<br/>(payment provider)"] --> TO["timeout β p99.9<br/>+ deadline propagation"]
TO --> RT["retry: backoff, jitter,<br/>retry budget, one layer"]
RT --> IK["idempotency key<br/>β Compute β integrations"]
TO --> CB["circuit breaker<br/>(also counts slow calls)"]
CB --> FB["fallback: seats held,<br/>tell the truth"]
SLOW --> BH["bulkhead sized by<br/>Little's Law β Estimation"]
OVER["overload"] --> LS["load shedding<br/>β waiting room (Messaging)"]
OVER --> RL["rate limiting<br/>β Edge β API gateway"]
HC["liveness, readiness, startup<br/>β Cloud-native β Kubernetes"] --> LB["out of the load balancer<br/>β Edge β load balancing"]
CB --> CH["prove it under fault<br/>β Chaos engineering"] Palette components: thirdparty Β· payment (the call every pattern on this page protects) Β· app (the order replicas whose threads the bulkhead guards) Β· apigw (rate limiting and load shedding at the door) Β· queue (the waiting room) Β· healthcheck (liveness, readiness, startup) Β· lb (drops unready instances). Scenarios: cascading-failure, retry-storm, rate-limiter, payment-system, flash-sale. Continues in: Compute β integrations, Messaging β delivery semantics, Messaging β the waiting room, Edge β API gateway, Edge β load balancing, Cloud-native β Kubernetes, Hands-on 9 β payment and Toxiproxy.
-
NYGARD, M. Release It! Design and Deploy Production-Ready Software, 2nd ed. Pragmatic Bookshelf, 2018. Chapter "Stability Patterns".Β β©
-
BROOKER, M. Exponential Backoff And Jitter. AWS Architecture Blog, 2015; BROOKER, M. Timeouts, retries, and backoff with jitter. Amazon Builders' Library.Β β©
-
FOWLER, M. CircuitBreaker. martinfowler.com, 2014.Β β©
-
BEYER, B. et al. Site Reliability Engineering. O'Reilly, 2016. Chapter 21, "Handling Overload"; YANACEK, D. Using load shedding to avoid overload. Amazon Builders' Library.Β β©
-
MICROSOFT. Build resilient HTTP apps: key development patterns. Microsoft Learn (the standard resilience handler).Β β©