Resilience Patterns

When a dependency degrades, the naive default — keep sending requests and wait for the answer — reliably produces the cascade of the overview: the caller's threads block, its queue fills, and the failure climbs to users who had nothing to do with the broken dependency. Resilience patterns are design decisions that interrupt this propagation. They do not prevent failures; they limit how far a failure travels¹.

Throughout this page we follow one call of BoxOffice: order calls payment, the adapter in front of the external payment provider, to charge a fan's card during the purchase saga (see Messaging — patterns). It is the most dangerous call in the system: it is on the revenue path, it crosses the internet to a third party, and the third party's bad day becomes ours.

Timeout

A timeout is a deadline on an outbound call: if the answer has not arrived in time, the caller abandons it and handles the error. Without a timeout, a hanging dependency holds a thread (and a connection, and memory) indefinitely; with one, the damage a single slow call can do is bounded.

sequenceDiagram
    participant O as order
    participant P as payment adapter
    participant X as payment provider
    O->>+P: POST /charges (deadline 2 s)
    P->>+X: charge card
    Note over X: provider is slow today
    Note over O: 2 s elapsed: give up, free the thread
    O-->>O: handle the error (retry? fallback?)
    X-->>-P: (answer arrives at 4 s)
    P-->>-O: (too late: discarded)

How to choose the value:

• From the dependency's latency distribution. Set it a little above the p99.9 of normal operation: anything slower is already abnormal. For the provider, with p99.9 ≈ 1.2 s, a 2 s timeout.
• Two timeouts per HTTP call: the connect timeout (opening the TCP/TLS connection — short, a few hundred milliseconds) and the read timeout (waiting for the response).
• Per dependency. A cache lookup, a database query and a third-party API have latencies that differ by orders of magnitude. One global timeout is always wrong for most of them.
• Inside the caller's budget. If the fan's request has 6 s at the gateway, everything below must fit in it, including retries.

Deadline propagation

Timeouts must shrink as a request goes deeper. A deadline set at the edge and propagated with the request (gRPC does this natively with deadlines; over HTTP, a header such as the remaining budget) lets each service know how much time is left, and stop working on a request whose caller has already given up.

public PaymentClient(RestClient.Builder builder,
                     @Value("${boxoffice.payment.url}") String url,
                     @Value("${boxoffice.payment.connect-timeout}") Duration connectTimeout,
                     @Value("${boxoffice.payment.read-timeout}") Duration readTimeout) {
    SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
    factory.setConnectTimeout(connectTimeout);
    factory.setReadTimeout(readTimeout); // 0 = wait forever (the "before" experiment)
    this.http = builder.baseUrl(url).requestFactory(factory).build();
}

public PaymentResponse charge(UUID orderId, java.math.BigDecimal amount) {
    return http.post()
            .uri("/payments")
            .header("Idempotency-Key", "order-" + orderId) // same key on every retry
            .body(new PaymentRequest(orderId, amount))
            .retrieve()
            .body(PaymentResponse.class);
}

Retry with backoff and jitter

A retry repeats a failed call, betting that the failure was transient: a dropped connection, a pod restarting, a leader election. Retries turn many small blips into invisible ones. Done carelessly, they are also the fastest way to turn a small problem into an outage.

What to retry

Exponential backoff and jitter

Retrying immediately hits a struggling dependency again before it had any chance to recover. Exponential backoff waits longer after each failure:

With base 200 ms: 200 ms, 400 ms, 800 ms, … up to the cap. But if many clients failed at the same moment (the provider blipped for everyone), they all retry at the same moments too — synchronized waves. Jitter randomizes the waits; Marc Brooker's analysis at AWS recommends full jitter²:

Retries amplify load

The simulation below has two experiments with illustrative numbers. On the left, steady purchase traffic (1,000 requests/s) meets a payment provider that degrades for 5 seconds; an overloaded server wastes part of its capacity on requests that time out anyway. On the right, 1,000 clients fail at the same instant (say, a connection reset during a failover) and retry against a server that can take 150 requests per 100 ms.

• Left — retries can keep a system down. Without retries (grey), successes drop during the degradation and come back the moment it ends. With 3 retries per request (red), each failure becomes up to 4 calls; the extra load keeps the provider overloaded after the trigger is gone, and the system settles in a degraded state — a metastable failure (see the overview). Backoff and jitter do not fix this: they spread retries in time but do not reduce their number. A retry budget (blue) — retries may add at most 10% to the normal traffic — caps the amplification and the system recovers immediately.
• Right — jitter prevents synchronized waves. Without jitter (orange), all clients that failed together retry together: 1,000 calls in one 100 ms slot, only 150 succeed, the other 850 retry together again… after 15 seconds 250 clients still have not succeeded. With full jitter (blue), the same retries are spread below the capacity line and everyone succeeds in about 1.5 s.

Three rules follow:

1. Retry at one layer only. If the gateway, order and the payment adapter each retry 3 times, one fan click can become \(3 \times 3 \times 3 = 27\) calls to the provider. Retry close to the failing dependency and let upper layers fail fast.
2. Cap attempts and total time: few attempts (1–2 retries), all inside the caller's deadline.
3. Use a retry budget (Envoy, gRPC retry throttling, Finagle): retries are allowed only while they are a small fraction of requests. When the dependency is really down, retries stop by themselves.

Circuit breaker

A circuit breaker watches the outcomes of calls to a dependency. When the failure rate crosses a threshold, it opens and rejects calls immediately, without touching the network, for a while; then it lets a few probe calls through to test recovery³.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open : failure or slow-call rate ≥ 50% (last 20 calls)
    Open --> HalfOpen : after 30 s
    HalfOpen --> Closed : 3 probe calls succeed
    HalfOpen --> Open : a probe fails

Closed (normal) : Calls pass through; the breaker records successes, failures and slow calls in a sliding window.

Open (failing fast) : Calls fail immediately (CallNotPermittedException in Resilience4j). The caller's threads are free, and the dependency gets breathing room instead of a retry storm.

Half-open (testing) : After the wait duration, a few calls are allowed. If they succeed, the breaker closes; if not, it opens again.

Which errors count matters: a declined card is a correct answer from a healthy provider, not a failure. Configure it as ignored, or a wave of declined cards on a stolen-card attack would open the breaker for every honest fan.

Fallback and graceful degradation

Failing fast is better than hanging, but an error is still an error. A fallback is what the caller does instead. The goal is graceful degradation: lose a feature, not the page.

Bulkhead

A bulkhead — named after the watertight compartments of a ship — gives each dependency its own limited pool of threads, connections or concurrent calls, so one slow dependency cannot consume the capacity of the others.

flowchart LR
    req["order: 200 request threads"] --> bp["payment bulkhead<br/>max 50 concurrent"]
    req --> bi["inventory bulkhead<br/>max 40 concurrent"]
    req --> bc["catalog bulkhead<br/>max 20 concurrent"]
    bp --> pay["payment<br/>(slow: 4 s)"]
    bi --> inv["inventory<br/>(healthy)"]
    bc --> cat["catalog<br/>(healthy)"]

Sizing comes from Little's Law again. At 100 purchases/s and 200 ms per charge, about 20 calls are in flight. A bulkhead of 50 leaves room for latency to grow 2.5× before calls are rejected; when the provider goes to 4 s, at most 50 threads wait on it (instead of 400), and the other 150 keep serving holds, order pages and everything else. Rejected calls fail in microseconds and go to the fallback.

Resilience4j offers a semaphore bulkhead (limits concurrent calls on the caller's threads; cheap, the default) and a thread-pool bulkhead (runs calls on a dedicated pool). The idea is general: separate connection pools per workload in a database, separate Kafka consumer groups, separate deployments for critical and non-critical traffic, separate queues for confirmations and marketing e-mails.

Load shedding

When demand exceeds capacity, some requests will fail. Load shedding decides which and makes them fail early and cheaply, so the rest succeed. Without it, the server accepts everything, every request gets slow, most time out after consuming resources, and goodput collapses — everyone fails⁴.

• Shed at the door. Reject with 503 + Retry-After before doing any work: a rejection costs microseconds, a timed-out request costs its full processing.
• Shed by priority. During the stadium on-sale, BoxOffice classifies traffic: purchase and seat-hold requests are critical; catalog browsing is important; recommendations, autocomplete and "related events" are sheddable. Under overload, the sheddable go first.
• Shed stale work. A request that already waited longer than its caller's deadline is useless; drop it from the queue instead of processing it (bounded queues with deadlines, LIFO under overload).
• Adapt the limit. Instead of a fixed concurrency limit, adaptive limiters (inspired by TCP congestion control, e.g. Netflix's concurrency-limits library) lower the number of concurrent requests when latency rises and raise it when latency is healthy.

Rate limiting

Rate limiting caps how many requests a client (an IP, a user, an API key, a partner) may send in a time window. Load shedding protects the server from total overload; rate limiting protects it from individual clients — bots refreshing the seat map hundreds of times per second, a partner's buggy integration, a scraper.

The common algorithm is the token bucket: tokens refill at a steady rate (the sustained limit), the bucket's size allows short bursts, and each request takes a token; an empty bucket means 429 Too Many Requests. Limits per user or API key live in the API gateway, backed by a shared store such as Redis, where the authenticated identity is known. Algorithms (token bucket, leaky bucket, sliding window) and their implementation are covered in Edge — API gateway; the rate-limiter playground scenario shows a gateway protecting a backend.

Health checks

Health checks let orchestrators and load balancers stop sending traffic to broken instances — automatic failover at the smallest scale.

Liveness : Is the process alive and not stuck? If it fails, Kubernetes restarts the container.

Readiness : Can this instance serve traffic now? If it fails, the instance is removed from the load balancer without being restarted (warming up, overloaded, lost its own database connection).

Startup : Has the application finished starting? Liveness and readiness are not checked until it succeeds, so a slow-starting JVM is not killed during boot.

# order Deployment, container spec (Spring Boot Actuator endpoints)
startupProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30        # up to 150 s to start
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2

Putting it together

The patterns are layered. For the order → payment call, BoxOffice uses all of them, configured with Resilience4j⁵. The values below are the design for the stadium on-sale (100 purchases/s, 200 order threads); lab 9 runs the same stack with smaller numbers, so that a laptop can show each effect (box below).

resilience4j:
  bulkhead:
    instances:
      payment:
        max-concurrent-calls: 50          # Little's Law: 20 in flight normally
        max-wait-duration: 10ms           # do not queue: reject and fall back
  circuitbreaker:
    instances:
      payment:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50
        slow-call-duration-threshold: 1500ms
        slow-call-rate-threshold: 50      # a slow provider also opens the breaker
        wait-duration-in-open-state: 30s
        permitted-number-of-calls-in-half-open-state: 3
        ignore-exceptions:
          - boxoffice.payment.CardDeclinedException   # a correct answer, not a failure
  retry:
    instances:
      payment:
        max-attempts: 2                   # 1 call + 1 retry, at this layer only
        wait-duration: 200ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2
        enable-randomized-wait: true      # jitter
        randomized-wait-factor: 0.5
        retry-exceptions:
          - org.springframework.web.client.ResourceAccessException   # I/O errors, timeouts
          - org.springframework.web.client.HttpServerErrorException$ServiceUnavailable
        ignore-exceptions:
          - boxoffice.payment.CardDeclinedException

@Service
@RequiredArgsConstructor
public class PaymentClient {

    private final RestClient payment;

    @Bulkhead(name = "payment")
    @CircuitBreaker(name = "payment")
    @Retry(name = "payment", fallbackMethod = "unavailable")   // fallback on the outermost layer
    public PaymentResult charge(ChargeRequest req) {
        return payment.post().uri("/charges")
            .header("Idempotency-Key", req.orderId())   // a retry never charges twice
            .body(req)
            .retrieve()
            .body(PaymentResult.class);
    }

    // Called after the last retry fails, or when the breaker is open or the bulkhead is full.
    private PaymentResult unavailable(ChargeRequest req, Throwable t) {
        return PaymentResult.temporarilyUnavailable(req.orderId());  // seats stay held
    }
}

@Bean
RestClient paymentRestClient(RestClient.Builder builder) {
    HttpClient http = HttpClient.newBuilder()
        .connectTimeout(Duration.ofMillis(300))          // connect timeout
        .build();
    JdkClientHttpRequestFactory factory = new JdkClientHttpRequestFactory(http);
    factory.setReadTimeout(Duration.ofSeconds(2));       // read timeout per attempt
    return builder.baseUrl("http://payment:8080").requestFactory(factory).build();
}

resilience4j:
  retry:
    instances:
      payment:
        max-attempts: ${PAYMENT_RETRY_ATTEMPTS:3}    # 1 = no retry
        wait-duration: 200ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2            # 200 ms, 400 ms ...
        enable-randomized-wait: true
        randomized-wait-factor: 0.5                  # ... each +/- 50% (jitter)
        retry-exceptions:
          - org.springframework.web.client.ResourceAccessException   # timeouts, connection refused/reset
          - org.springframework.web.client.HttpServerErrorException  # 5xx
          - org.springframework.web.client.RestClientException       # connection reset while reading the response
        ignore-exceptions:
          - org.springframework.web.client.HttpClientErrorException  # 4xx: the same request fails again
          - io.github.resilience4j.circuitbreaker.CallNotPermittedException  # circuit open: fail fast
          - io.github.resilience4j.bulkhead.BulkheadFullException           # bulkhead full: fail fast
  circuitbreaker:
    instances:
      payment:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50                   # open when >= 50% of the last 20 calls failed...
        slow-call-duration-threshold: 800ms
        slow-call-rate-threshold: 50                 # ... or >= 50% were slower than 800 ms
        wait-duration-in-open-state: 10s
        permitted-number-of-calls-in-half-open-state: 3
        automatic-transition-from-open-to-half-open-enabled: true
        register-health-indicator: true
        ignore-exceptions:
          - io.github.resilience4j.bulkhead.BulkheadFullException  # our own refusal is not the dependency's fault
  bulkhead:
    instances:
      payment:
        max-concurrent-calls: ${PAYMENT_BULKHEAD:10} # at most 10 of the 50 threads wait on payment
        max-wait-duration: 0                         # full = reject immediately

@Retry(name = "payment", fallbackMethod = "unavailable")
@CircuitBreaker(name = "payment")
@Bulkhead(name = "payment")
public PaymentResponse charge(UUID orderId, BigDecimal amount) {
    return client.charge(orderId, amount);
}

How the layers interact — check these before shipping:

• Order of the decorators. Resilience4j's Spring Boot default is Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( call ) ) ) ) ): the retry is outermost, so every attempt is counted by the breaker and an open breaker makes the retry fail immediately. Make sure retries never bypass the breaker. Put the fallback on the outermost layer (the retry): a fallback on the breaker turns every failure into a normal answer, so the retry never sees an error and never retries.
• Time budget. 2 attempts × 2 s + one backoff of ~200–300 ms ≈ 4.3 s, inside order's 5 s budget and the gateway's 6 s.
• Metrics. Resilience4j exports breaker state, bulkhead saturation and retry counts through Micrometer; graph them next to the SLO dashboard (Hands-on 8), because a breaker that opens is an incident in progress.
• Proof. None of this is trustworthy until you have injected the failure and watched it work — chaos engineering and Hands-on 9.

How it connects

flowchart LR
    SLOW["slow dependency<br/>(payment provider)"] --> TO["timeout ≈ p99.9<br/>+ deadline propagation"]
    TO --> RT["retry: backoff, jitter,<br/>retry budget, one layer"]
    RT --> IK["idempotency key<br/>→ Compute — integrations"]
    TO --> CB["circuit breaker<br/>(also counts slow calls)"]
    CB --> FB["fallback: seats held,<br/>tell the truth"]
    SLOW --> BH["bulkhead sized by<br/>Little's Law → Estimation"]
    OVER["overload"] --> LS["load shedding<br/>→ waiting room (Messaging)"]
    OVER --> RL["rate limiting<br/>→ Edge — API gateway"]
    HC["liveness, readiness, startup<br/>→ Cloud-native — Kubernetes"] --> LB["out of the load balancer<br/>→ Edge — load balancing"]
    CB --> CH["prove it under fault<br/>→ Chaos engineering"]

Palette components: thirdparty · payment (the call every pattern on this page protects) · app (the order replicas whose threads the bulkhead guards) · apigw (rate limiting and load shedding at the door) · queue (the waiting room) · healthcheck (liveness, readiness, startup) · lb (drops unready instances). Scenarios: cascading-failure, retry-storm, rate-limiter, payment-system, flash-sale. Continues in: Compute — integrations, Messaging — delivery semantics, Messaging — the waiting room, Edge — API gateway, Edge — load balancing, Cloud-native — Kubernetes, Hands-on 9 — payment and Toxiproxy.