Resilience Patterns

When a downstream service degrades or fails, the naive default — keep sending requests and wait for a response — reliably produces cascading failures. The calling service's threads block, its queue fills, and the failure propagates upstream to clients who had nothing to do with the broken dependency.

Resilience patterns are design decisions that interrupt this propagation. They do not prevent failures; they limit how far a failure travels through the system¹.

flowchart LR
    client --> gateway
    gateway -->|"Timeout\nCircuit Breaker\nBulkhead"| downstream:::slow
    gateway -->|"fast failure\nor fallback"| client:::ok

    classDef slow fill:#f90,color:#000
    classDef ok fill:#6c6,color:#fff

The patterns below are complementary — production systems use them in combination, not in isolation. Throughout the page we use one running example from the store: the order service calls the product service (through an OpenFeign client, as in Hands-on 1) to check prices before saving an order.

Timeout

A timeout is a deadline applied to an outbound call. If the downstream service does not respond within the deadline, the call is abandoned and an error is returned to the caller immediately.

Without a timeout, a single hanging dependency can hold threads indefinitely. With a timeout, the maximum damage any single slow call can do is bounded.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant P as Payment Service

    C->>+G: POST /checkout
    G->>+P: POST /charge
    Note over P: processing...
    Note over G: deadline exceeded (500 ms)
    G-->>-C: 503 Service Unavailable
    P-->>-G: (response arrives late — discarded)

Choosing timeout values

• Percentile-based: set the timeout at the P99.9 latency of the downstream service under normal conditions. Anything slower than that is already abnormal.
• Budget-aware: if the upstream call has its own timeout of 1 s, all downstream calls within that request must complete well under 1 s combined.
• Distinct per dependency: a database query, a third-party API, and an internal microservice have very different latency profiles. A single global timeout value is almost always wrong.

There are usually two timeouts on an HTTP call: the connect timeout (how long to wait to open the TCP connection) and the read timeout (how long to wait for the response once connected). Configure both.

# order-service application.yaml (Spring Cloud OpenFeign)
spring:
  cloud:
    openfeign:
      client:
        config:
          product:              # the @FeignClient name
            connect-timeout: 500    # ms
            read-timeout: 1000      # ms

Retry with Exponential Backoff

A retry re-attempts a failed request after a waiting period. Retries are appropriate for transient failures — brief network blips, momentary service restarts, or transient resource exhaustion — where the same request is likely to succeed on a subsequent attempt.

sequenceDiagram
    participant C as Client
    participant S as Service

    C->>+S: Request (attempt 1)
    S-->>-C: 503 Service Unavailable
    Note over C: wait 1 s
    C->>+S: Request (attempt 2)
    S-->>-C: 503 Service Unavailable
    Note over C: wait 2 s
    C->>+S: Request (attempt 3)
    S-->>-C: 200 OK ✓

Exponential backoff

Retrying immediately on failure is dangerous: if many clients all retry simultaneously after a service restart, the resulting spike can overwhelm the recovering service — a thundering herd. Exponential backoff spaces retries out:

Jitter

Even with exponential backoff, clients that all failed at the same moment will retry at the same moments — a synchronised wave of retries. Adding a random jitter breaks this synchronisation:

Marc Brooker's analysis at AWS² compared several variants and recommends full jitter, which randomises the entire wait instead of adding a small random amount:

With 1,000 clients failing at the same instant, full jitter spreads their retries uniformly over the interval instead of sending 1,000 requests at exactly t = 1 s, 2 s, 4 s.

What to retry — and what not to

An operation is idempotent when doing it twice has the same effect as doing it once. The common way to make a POST safe to retry is an idempotency key: the client generates a unique ID (e.g. a UUID) per logical operation and sends it in a header; the server stores the keys it has already processed and, on a repeated key, returns the stored result instead of creating a second order.

sequenceDiagram
    participant C as Client
    participant O as Order service
    participant DB as Postgres
    C->>O: POST /orders (Idempotency-Key: k-42)
    O->>DB: key k-42 seen? no → insert order + key
    O--xC: 201 Created (response lost)
    C->>O: retry POST /orders (Idempotency-Key: k-42)
    O->>DB: key k-42 seen? yes
    O-->>C: 201 Created (same order, no duplicate)

Circuit Breaker

The circuit breaker³ is named by analogy with the electrical component: when the current load exceeds safe limits, the breaker trips and interrupts the circuit. In software, when a downstream service failure rate exceeds a threshold, the circuit breaker opens and immediately rejects calls to that service — without attempting the network call at all.

This prevents the calling service from wasting threads and connections on a dependency that is known to be unavailable.

stateDiagram-v2
    [*] --> Closed

    Closed --> Open : failure rate exceeds threshold\n(e.g. > 50% errors in last 10 calls)
    Open --> HalfOpen : wait timeout expires\n(e.g. 30 seconds)
    HalfOpen --> Closed : probe request succeeds
    HalfOpen --> Open : probe request fails

Closed (normal operation) : Requests pass through. The breaker counts successes and failures. If the failure rate crosses the configured threshold, the breaker trips to Open.

Open (failing fast) : All requests are rejected immediately with an error — no network call is made. This protects the caller's resources and gives the downstream service time to recover. After a configured wait timeout, the breaker moves to Half-Open.

Half-Open (testing recovery) : A limited number of probe requests are allowed through. If they succeed, the downstream service has recovered and the breaker closes. If they fail, the breaker returns to Open.

Configuration knobs

Spring Boot: the Resilience4j library⁴ implements all of these patterns with @CircuitBreaker, @Retry, @Bulkhead, @RateLimiter and @TimeLimiter annotations (dependency io.github.resilience4j:resilience4j-spring-boot3 plus spring-boot-starter-aop).

@CircuitBreaker(name = "product", fallbackMethod = "priceFallback")
@Retry(name = "product")
public ProductOut findProduct(String id) {
    return productController.findById(id).getBody();   // Feign call
}

// Same signature + the exception. Called when the call fails
// or when the breaker is open (CallNotPermittedException).
private ProductOut priceFallback(String id, Throwable t) {
    throw new ResponseStatusException(
        HttpStatus.SERVICE_UNAVAILABLE, "catalogue unavailable, try again later");
}

resilience4j:
  circuitbreaker:
    instances:
      product:
        sliding-window-size: 10                  # last 10 calls
        failure-rate-threshold: 50               # % → Open
        slow-call-duration-threshold: 2s         # slow = failure
        wait-duration-in-open-state: 30s         # Open → Half-Open
        permitted-number-of-calls-in-half-open-state: 3
  retry:
    instances:
      product:
        max-attempts: 3                          # 1 call + 2 retries
        wait-duration: 500ms
        enable-exponential-backoff: true         # 500 ms, 1 s, ...
        exponential-backoff-multiplier: 2

The simulation below shows the circuit breaker cycling through all three states as the failure rate changes:

Fallback and graceful degradation

Failing fast is better than hanging, but an error is still an error. A fallback is what the caller does instead when a call fails or the breaker is open. The goal is graceful degradation: lose a feature, not the whole page.

Bulkhead

The bulkhead pattern — named after watertight compartments in a ship hull — isolates resources by dependency. Rather than sharing a single thread pool across all downstream calls, each dependency gets its own pool.

flowchart TD
    gateway --> poolA["Thread Pool A\n(max 10 threads)"]
    gateway --> poolB["Thread Pool B\n(max 10 threads)"]
    poolA --> payment["Payment Service\n(degraded ⚠️)"]:::slow
    poolB --> inventory["Inventory Service\n(healthy ✓)"]:::ok

    classDef slow fill:#f90,color:#000
    classDef ok fill:#6c6,color:#fff

If Payment Service degrades and fills all 10 threads in Pool A, Pool B is completely unaffected. Inventory Service continues to handle requests at full speed. Without bulkheads, a single slow dependency saturates the shared pool and degrades all other calls equally.

The cost of bulkheads is resource overhead: ten separate pools of 10 threads each use more memory than one pool of 100 threads. The trade-off is almost always worth it for critical-path dependencies.

Resilience4j offers two kinds of bulkhead: a semaphore bulkhead (max-concurrent-calls, limits how many threads may be inside the call at once — cheap, the default) and a thread-pool bulkhead (the call runs in a dedicated pool). The same idea applies outside the JVM: separate database connection pools per workload, or separate Kubernetes deployments for critical and non-critical traffic.

Rate Limiting

Rate limiting caps the number of requests a client (or the system) can make within a time window, protecting services from traffic spikes and abusive clients.

Token bucket algorithm

The token bucket is the most common rate limiting algorithm. It models a bucket that fills at a constant rate with tokens; each request consumes one token; requests that arrive when the bucket is empty are rejected or queued.

flowchart LR
    refill["Token source\n(refill rate: 100 req/s)"] -->|adds tokens| bucket["Token Bucket\n(capacity: 200)"]
    request["Incoming\nRequest"] -->|consumes 1 token| bucket
    bucket -->|tokens available| allowed["✓ Allowed"]
    bucket -->|bucket empty| rejected["✗ 429 Too Many Requests"]

A bucket capacity larger than the refill rate allows short bursts while still enforcing a long-term average. Spring Cloud Gateway's RequestRateLimiter filter (backed by Redis) and AWS API Gateway use a token bucket. Nginx limit_req uses the closely related leaky bucket: requests drain at a fixed rate and the burst parameter sets how many may wait in the queue.

# 10 requests/s per client IP, bursts of up to 20 queued requests
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    location / {
        limit_req zone=api burst=20 nodelay;
        limit_req_status 429;
        proxy_pass http://gateway;
    }
}

Health Checks

Health check endpoints allow orchestrators (Kubernetes, Docker Compose, load balancers) to detect failed instances and stop routing traffic to them before clients notice.

Two distinct types serve different purposes:

Liveness : Is the process alive? Returns 200 OK if the application is running and not deadlocked. If a liveness check fails, the orchestrator restarts the container.

Readiness : Is the service ready to serve traffic? Returns 200 OK only if the instance has finished starting and can do useful work (e.g. its own database is reachable). If a readiness check fails, the orchestrator removes the instance from the load balancer pool without restarting it.

Startup (Kubernetes) : Has the application finished starting? Liveness and readiness checks are not run until it succeeds, so a slow-starting JVM is not killed during boot.

flowchart LR
    lb["Load Balancer"] -->|"GET /actuator/health/readiness"| svc["Service Instance"]
    svc -->|200 OK| lb
    lb -->|routes traffic| svc

    svc2["Service Instance\n(db connection lost)"]
    lb -->|"GET /actuator/health/readiness"| svc2
    svc2 -->|503 DOWN| lb
    lb -->|removed from pool| svc2:::failed

    classDef failed fill:#f66,color:#fff

Spring Boot Actuator exposes both at /actuator/health/liveness and /actuator/health/readiness. They are enabled automatically when the application detects it runs on Kubernetes; elsewhere (e.g. Docker Compose) set management.endpoint.health.probes.enabled=true. By default they reflect only the application's internal state; add a dependency to readiness explicitly, e.g. management.endpoint.health.group.readiness.include=readinessState,db.

# inside the container spec of the Deployment
startupProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  failureThreshold: 30        # up to 30 × 5 s = 150 s to start
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }
  periodSeconds: 5

Defence in depth

These patterns are layered, not chosen in isolation. A production-grade call chain combines all of them:

flowchart LR
    client --> rl["Rate Limiter\n(protect service)"]
    rl --> cb["Circuit Breaker\n(fast-fail open deps)"]
    cb --> bh["Bulkhead\n(isolate thread pool)"]
    bh --> to["Timeout\n(bound wait time)"]
    to --> retry["Retry + Backoff\n(handle transients)"]
    retry --> downstream[Downstream Service]

Each layer handles a different failure scenario:

Connections

• Hands-on 1 — the Feign clients that need timeouts and breakers.
• Hands-on 7 — Nginx in front of the gateway, where edge rate limiting lives.
• Hands-on 9 — retries and dead letter queues for asynchronous consumers.
• SLI / SLO / SLA — the budget these patterns protect.

Key takeaways

• Every remote call needs a timeout (connect + read), chosen per dependency and inside the caller's budget.
• Retry only transient failures and idempotent operations, with exponential backoff and full jitter, at one layer only; use idempotency keys to make POST safe.
• A circuit breaker (Closed → Open → Half-Open) fails fast on a dependency known to be down; pair it with a safe fallback.
• Bulkheads keep one slow dependency from consuming all threads or connections; rate limiting (token / leaky bucket) protects the service from excess traffic.
• Liveness restarts, readiness removes from the load balancer; never check external dependencies in liveness.
• The patterns are layered; their order and combined timeouts must be designed, then verified with chaos experiments.