2. Resilience Patterns
order now calls payment synchronously: POST /orders/{id}/pay charges the fan and, on success, confirms the order. This step first shows what one slow dependency does to an unprotected service, then adds the four patterns of the reliability class one at a time — timeout, retry, circuit breaker, bulkhead — measuring after each.
The failure we are defending against
sequenceDiagram
participant F as fans (20/s)
participant O as order (50 threads)
participant P as payment (+3 s latency)
F->>O: POST /orders/1/pay
O->>P: charge (waits 3 s)
F->>O: POST /orders/2/pay
O->>P: charge (waits 3 s)
Note over O: 20 req/s x 3 s = 60 calls in flight > 50 threads
F->>O: GET /orders/7 (unrelated!)
Note over O: no free thread: queued, then times out
O-->>F: 503 / timeout for EVERY endpoint Little's Law says it all: threads busy = arrival rate × time each call holds a thread. With 20 payments/s and a dependency that takes 3 s, order needs 60 threads just to wait. When the pool (50 here, 200 by default) runs out, every endpoint of order stops — including GET /orders/{id}, which never touches payment. One slow dependency took the whole service down: a cascading failure.
Why order gets only 50 threads in this lab
Tomcat's default pool is 200 threads. A laptop running fourteen containers cannot generate enough load to exhaust 200 threads against a 3-second dependency without k6 itself becoming the bottleneck, so the lab shrinks the pool (server.tomcat.threads.max: 50) — the same trick lab 1 used when it limited catalog to one CPU. The mechanism is identical at any size.
Code
Add Resilience4j to order (its Spring Boot 4 module, resilience4j-spring-boot4); the annotations need AspectJ (spring-boot-starter-aspectj). The complete pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>4.1.1</version>
<relativePath/>
</parent>
<groupId>boxoffice</groupId>
<artifactId>order</artifactId>
<version>1.0.0</version>
<properties>
<java.version>21</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webmvc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-restclient</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-flyway</artifactId>
</dependency>
<dependency>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-database-postgresql</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<!-- lab 8: only the API (no SDK); the Java agent provides the implementation at runtime
and, without the agent, every call is a no-op. Version managed by Spring Boot. -->
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
</dependency>
<!-- lab 9: resilience patterns (the annotations need AspectJ) -->
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-spring-boot4</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aspectj</artifactId>
</dependency>
</dependencies>
<build>
<finalName>app</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
The new package boxoffice.order.payment holds the client and the protected gateway:
package boxoffice.order.payment;
import java.time.Duration;
import java.util.UUID;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;
@Component
public class PaymentClient {
private final RestClient http;
public PaymentClient(RestClient.Builder builder,
@Value("${boxoffice.payment.url}") String url,
@Value("${boxoffice.payment.connect-timeout}") Duration connectTimeout,
@Value("${boxoffice.payment.read-timeout}") Duration readTimeout) {
SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
factory.setConnectTimeout(connectTimeout);
factory.setReadTimeout(readTimeout); // 0 = wait forever (the "before" experiment)
this.http = builder.baseUrl(url).requestFactory(factory).build();
}
public PaymentResponse charge(UUID orderId, java.math.BigDecimal amount) {
return http.post()
.uri("/payments")
.header("Idempotency-Key", "order-" + orderId) // same key on every retry
.body(new PaymentRequest(orderId, amount))
.retrieve()
.body(PaymentResponse.class);
}
}
package boxoffice.order.payment;
import io.github.resilience4j.bulkhead.annotation.Bulkhead;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.retry.annotation.Retry;
import java.math.BigDecimal;
import java.util.UUID;
import org.springframework.stereotype.Component;
/**
* Default aspect order: Retry ( CircuitBreaker ( Bulkhead ( call ) ) ).
* The fallback sits on the OUTERMOST layer, so it runs only after the retries are exhausted,
* when the circuit is open or when the bulkhead is full.
*/
@Component
public class PaymentGateway {
private final PaymentClient client;
public PaymentGateway(PaymentClient client) {
this.client = client;
}
@Retry(name = "payment", fallbackMethod = "unavailable")
@CircuitBreaker(name = "payment")
@Bulkhead(name = "payment")
public PaymentResponse charge(UUID orderId, BigDecimal amount) {
return client.charge(orderId, amount);
}
/** The "before" path of the experiments: no Resilience4j at all. */
public PaymentResponse chargeUnprotected(UUID orderId, BigDecimal amount) {
return client.charge(orderId, amount);
}
/** Same parameters + the exception: Resilience4j finds it by name and signature. */
private PaymentResponse unavailable(UUID orderId, BigDecimal amount, Throwable cause) {
throw new PaymentUnavailableException(cause);
}
}
package boxoffice.order.payment;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ResponseStatus;
/** Fast, explicit failure: the fan sees "try again in a moment", not a 60-second spinner. */
@ResponseStatus(HttpStatus.SERVICE_UNAVAILABLE)
public class PaymentUnavailableException extends RuntimeException {
public PaymentUnavailableException(Throwable cause) {
super("payment temporarily unavailable: " + cause.getClass().getSimpleName(), cause);
}
}
OrderService gains pay. Note where the transaction is: the remote call happens with no transaction open, and only the short status update runs in one — holding a database connection while waiting 3 s for a dependency would move the cascade into the connection pool.
package boxoffice.order;
import boxoffice.order.OrderProperties.NotifyMode;
import boxoffice.order.outbox.Outbox;
import boxoffice.order.payment.PaymentGateway;
import boxoffice.order.payment.PaymentResponse;
import java.time.Instant;
import java.util.Optional;
import java.util.UUID;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus;
import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionTemplate;
import org.springframework.web.server.ResponseStatusException;
@Service
public class OrderService {
public record PlaceResult(OrderView order, boolean created) {
}
private static final String SELECT = """
SELECT id, show_id, hold_id, customer_email, amount, status, created_at
FROM orders
""";
private final JdbcClient jdbc;
private final Outbox outbox;
private final NotificationClient notifications;
private final OrderProperties props;
private final PaymentGateway payments;
private final TransactionTemplate tx;
private final boolean resilience;
public OrderService(JdbcClient jdbc, Outbox outbox, NotificationClient notifications,
OrderProperties props, PaymentGateway payments, TransactionTemplate tx,
@Value("${boxoffice.payment.resilience:true}") boolean resilience) {
this.jdbc = jdbc;
this.outbox = outbox;
this.notifications = notifications;
this.props = props;
this.payments = payments;
this.tx = tx;
this.resilience = resilience;
}
@Transactional
public PlaceResult place(OrderRequest req) {
int inserted = jdbc.sql("""
INSERT INTO orders (id, show_id, hold_id, customer_email, amount, status)
VALUES (?, ?, ?, ?, ?, 'PENDING_PAYMENT')
ON CONFLICT (hold_id) DO NOTHING
""")
.params(UUID.randomUUID(), req.showId(), req.holdId(), req.customerEmail(), req.amount())
.update();
OrderView order = jdbc.sql(SELECT + " WHERE hold_id = ?")
.param(req.holdId())
.query(OrderView.class)
.single();
if (inserted == 0) {
return new PlaceResult(order, false);
}
OrderEvent event = OrderEvent.placed(order);
if (props.notifyMode() == NotifyMode.SYNC) {
notifications.send(event);
} else {
outbox.append(event.eventId(), "order", order.id(), event.eventType(), event);
}
return new PlaceResult(order, true);
}
/** No @Transactional here: never hold a DB connection during a remote call. */
public OrderView pay(UUID id) {
OrderView order = find(id).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND));
if (!"PENDING_PAYMENT".equals(order.status())) {
return order; // already paid or cancelled: paying twice is a no-op
}
PaymentResponse payment = resilience
? payments.charge(order.id(), order.amount())
: payments.chargeUnprotected(order.id(), order.amount());
String next = "AUTHORIZED".equals(payment.status()) ? "CONFIRMED" : "CANCELLED";
tx.executeWithoutResult(status -> {
int changed = jdbc.sql("UPDATE orders SET status = ? WHERE id = ? AND status = 'PENDING_PAYMENT'")
.params(next, id)
.update();
if (changed == 1) {
String type = "CONFIRMED".equals(next) ? "OrderConfirmed" : "OrderCancelled";
OrderEvent event = new OrderEvent(UUID.randomUUID(), type, order.id(), order.showId(),
order.holdId(), order.customerEmail(), order.amount(), Instant.now());
outbox.append(event.eventId(), "order", order.id(), type, event);
}
});
return find(id).orElseThrow();
}
public Optional<OrderView> find(UUID id) {
return jdbc.sql(SELECT + " WHERE id = ?").param(id).query(OrderView.class).optional();
}
}
package boxoffice.order;
import boxoffice.order.waitingroom.WaitingRoom;
import jakarta.validation.Valid;
import java.net.URI;
import java.util.UUID;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
@RestController
@RequestMapping("/orders")
public class OrderController {
private final OrderService orders;
private final WaitingRoom waitingRoom;
public OrderController(OrderService orders, WaitingRoom waitingRoom) {
this.orders = orders;
this.waitingRoom = waitingRoom;
}
@PostMapping
public ResponseEntity<OrderView> place(
@Valid @RequestBody OrderRequest req,
@RequestHeader(name = "X-Admission-Ticket", required = false) String ticket) {
if (!waitingRoom.mayEnter(req.showId(), ticket)) {
throw new ResponseStatusException(HttpStatus.FORBIDDEN,
"on-sale in progress: join /orders/waiting-room/" + req.showId() + "/join first");
}
OrderService.PlaceResult result = orders.place(req);
if (!result.created()) {
return ResponseEntity.ok(result.order());
}
return ResponseEntity.created(URI.create("/orders/" + result.order().id())).body(result.order());
}
@GetMapping("/{id}")
public OrderView get(@PathVariable UUID id) {
return orders.find(id).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND));
}
@PostMapping("/{id}/pay")
public OrderView pay(@PathVariable UUID id) {
return orders.pay(id);
}
}
Rebuild order and pay an order end to end (from the repository root):
docker compose up -d --build --wait order
ID=$(curl -s -X POST localhost:8084/orders -H 'Content-Type: application/json' \
-d '{"showId":"show-9","holdId":"hold-pay-1","customerEmail":"bia@example.com","amount":350.00}' | jq -r .id)
curl -s -X POST localhost:8084/orders/$ID/pay
You should see the order with "status":"CONFIRMED". The first payment after a restart can answer 503 instead: the cold JVMs of order, payment and WireMock may need more than the 1 s read timeout for the very first call — the fallback at work. Call pay again: payment already stored the charge under the same idempotency key, so the second call answers at once with CONFIRMED (no second charge).
The UPDATE … WHERE status = 'PENDING_PAYMENT' makes the confirmation idempotent too: two concurrent pay calls for the same order may both reach payment (which deduplicates by key), but only one changes the status and writes the OrderConfirmed event.
Configuration
All the knobs live in application.yaml (the complete file; the lab 9 parts are server.tomcat, boxoffice.payment, resilience4j and the new actuator endpoints); the environment variables let you switch them off one by one in the experiments without rebuilding.
server:
port: 8084
tomcat:
threads:
max: 50 # lab 9: small on purpose, to make thread exhaustion visible
spring:
application:
name: order
mvc:
problemdetails:
enabled: true
datasource:
url: jdbc:postgresql://${DB_HOST:db}:${DB_PORT:5432}/${DB_NAME:boxoffice}?currentSchema=orders
username: ${DB_USER:boxoffice}
password: ${DB_PASSWORD:boxoffice}
flyway:
schemas: orders
data:
redis:
host: ${REDIS_HOST:redis}
port: 6379
kafka:
bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:kafka:9092}
producer:
acks: all
properties:
enable.idempotence: true
linger.ms: 5
max.block.ms: 5000 # Kafka down: fail the relay run in 5 s (default 60 s) and retry
boxoffice:
order:
notify-mode: ${NOTIFY_MODE:outbox}
notification-url: ${NOTIFICATION_URL:http://notification:8086}
topic: order-events
relay-interval-ms: 200
waiting-room:
admit-per-second: ${ADMIT_PER_SECOND:20}
admission-ttl: 5m
payment: # lab 9
url: ${PAYMENT_URL:http://payment:8085}
resilience: ${PAYMENT_RESILIENCE:true} # false = call payment with no Resilience4j at all
connect-timeout: 500ms
read-timeout: ${PAYMENT_READ_TIMEOUT:1s} # 0 = wait forever
resilience4j:
retry:
instances:
payment:
max-attempts: ${PAYMENT_RETRY_ATTEMPTS:3} # 1 = no retry
wait-duration: 200ms
enable-exponential-backoff: true
exponential-backoff-multiplier: 2 # 200 ms, 400 ms ...
enable-randomized-wait: true
randomized-wait-factor: 0.5 # ... each +/- 50% (jitter)
retry-exceptions:
- org.springframework.web.client.ResourceAccessException # timeouts, connection refused/reset
- org.springframework.web.client.HttpServerErrorException # 5xx
- org.springframework.web.client.RestClientException # connection reset while reading the response
ignore-exceptions:
- org.springframework.web.client.HttpClientErrorException # 4xx: the same request fails again
- io.github.resilience4j.circuitbreaker.CallNotPermittedException # circuit open: fail fast
- io.github.resilience4j.bulkhead.BulkheadFullException # bulkhead full: fail fast
circuitbreaker:
instances:
payment:
sliding-window-type: COUNT_BASED
sliding-window-size: 20
minimum-number-of-calls: 10
failure-rate-threshold: 50 # open when >= 50% of the last 20 calls failed...
slow-call-duration-threshold: 800ms
slow-call-rate-threshold: 50 # ... or >= 50% were slower than 800 ms
wait-duration-in-open-state: 10s
permitted-number-of-calls-in-half-open-state: 3
automatic-transition-from-open-to-half-open-enabled: true
register-health-indicator: true
ignore-exceptions:
- io.github.resilience4j.bulkhead.BulkheadFullException # our own refusal is not the dependency's fault
bulkhead:
instances:
payment:
max-concurrent-calls: ${PAYMENT_BULKHEAD:10} # at most 10 of the 50 threads wait on payment
max-wait-duration: 0 # full = reject immediately
management:
endpoints:
web:
exposure:
include: health,info,metrics,circuitbreakers,circuitbreakerevents,retries,bulkheads
endpoint:
health:
probes:
enabled: true # /actuator/health/liveness and /readiness
health:
circuitbreakers:
enabled: true # lab 9: breaker state in /actuator/health
Pattern by pattern
Each experiment uses the k6 script of step 4 (load/pay-under-chaos.js): 20 payments per second plus 20 GET /orders/{id} per second, for 2 minutes. Rebuild order once (docker compose up -d --build --wait order), then change only environment variables.
0. Unprotected
PAYMENT_RESILIENCE=false PAYMENT_READ_TIMEOUT=0 docker compose up -d --wait order
curl -s -X POST localhost:8474/proxies/payment/toxics \
-d '{"name":"slow","type":"latency","stream":"downstream","attributes":{"latency":3000}}'
k6 run load/pay-under-chaos.js
Watch the browse scenario: GET /orders/{id} — which does not call payment — degrades with pay. That is the cascade.
1. Timeout
PAYMENT_RESILIENCE=false PAYMENT_READ_TIMEOUT=1s docker compose up -d --wait order
k6 run load/pay-under-chaos.js
Every payment now fails after 1 s instead of hanging for 3 s: a thread is held 1 s, not 3 s, so 20/s × 1 s = 20 threads — inside the pool. GET /orders/{id} recovers. A timeout converts a slow failure into a fast failure, and fast failures do not cascade.
Timeouts must be shorter than the caller's
The edge or the gateway in front of order has its own timeout. If order waits longer for payment than the gateway waits for order, the gateway gives up first and order keeps working for nobody. Budgets shrink at each hop: edge 10 s > gateway 5 s > order→payment 1 s × retries.
2. Retry with backoff and jitter
Replace the latency with a flaky network — 30% of the connections reset:
curl -s -X DELETE localhost:8474/proxies/payment/toxics/slow
curl -s -X POST localhost:8474/proxies/payment/toxics \
-d '{"name":"flaky","type":"reset_peer","toxicity":0.3,"attributes":{"timeout":0}}'
PAYMENT_RETRY_ATTEMPTS=1 docker compose up -d --wait order # resilience on, retry off
k6 run load/pay-under-chaos.js
PAYMENT_RETRY_ATTEMPTS=3 docker compose up -d --wait order
k6 run load/pay-under-chaos.js
With one attempt, a share of the payments fail (not exactly 30%: the toxicity applies per TCP connection, and healthy connections are reused). With three attempts, a payment fails only if all three fail: if one attempt fails with probability p, three fail with p³ — 30% becomes 2.7%. In a test run of this lab (second runs, since the first run after a recreate also shows the cold JVM's bulkhead rejections), about 0.5% of the payments failed with one attempt and 0.1% with three: the toxic only hits new connections. A reset can arrive while order reads the response, which RestClient reports as a plain RestClientException — that is why application.yaml lists it among the retry-exceptions, next to ResourceAccessException, while the 4xx HttpClientErrorException is ignored. Retries fix transient errors — at the price of extra load on the dependency (up to 3×) and extra latency (the backoff). They are safe only because the request carries an idempotency key.
Why jitter?
Without it, every client that failed at the same instant retries at the same instant (200 ms later, then 400 ms later): synchronized waves that hit a recovering dependency like a hammer. A random ±50% spreads the retries over time. Try the retry-storm scenario in the playground to see the amplification.
3. Circuit breaker
Now a full outage: disable the proxy, so every connection to payment is refused.
curl -s -X DELETE localhost:8474/proxies/payment/toxics/flaky
curl -s -X POST localhost:8474/proxies/payment -d '{"enabled": false}'
k6 run load/pay-under-chaos.js # terminal 1
watch -n 1 "curl -s localhost:8084/actuator/circuitbreakers | jq '.circuitBreakers.payment.state'" # terminal 2
After 10 failed calls (the minimum-number-of-calls) the breaker opens: for the next 10 s, charge fails immediately with CallNotPermittedException — no network call, no retry, no waiting — and the fallback answers 503. Then it goes half-open, lets 3 trial calls through and closes again if they succeed. Re-enable the proxy in the middle of the run (curl -s -X POST localhost:8474/proxies/payment -d '{"enabled": true}') and watch it close. /actuator/circuitbreakerevents lists every transition.
The breaker protects both sides: order stops wasting threads and time on calls that will fail, and payment gets breathing room to recover instead of a flood of retries.
4. Bulkhead
Back to latency, but this time calls that succeed slowly — 700 ms: under the read timeout (1 s) and, by themselves, under the circuit breaker's slow-call threshold (800 ms) — and more of them: 70 payments per second.
curl -s -X POST localhost:8474/proxies/payment -d '{"enabled": true}'
curl -s -X POST localhost:8474/proxies/payment/toxics \
-d '{"name":"slow","type":"latency","stream":"downstream","attributes":{"latency":700}}'
PAYMENT_BULKHEAD=1000 docker compose up -d --wait order # effectively no bulkhead
k6 run -e PAY_RATE=70 load/pay-under-chaos.js
PAYMENT_BULKHEAD=10 docker compose up -d --wait order
k6 run -e PAY_RATE=70 load/pay-under-chaos.js
curl -s -X POST localhost:8474/reset # remove every toxic
Without a bulkhead, 70/s × 0.7 s ≈ 49 threads wait on payment: the pool of 50 is practically full and GET /orders/{id} queues behind the payments. With max-concurrent-calls: 10, at most 10 threads wait on payment; the extra concurrent payments are rejected at once (503), and the other 40 threads keep serving GET /orders/{id} at normal latency.
In a test run of this lab (second runs), the calls that got through without the bulkhead took 750–1000 ms — 700 ms plus payment's own time under this load — so the breaker's slow-call rule opened it for part of the run and about half of the payments failed fast; even so, browse p95 was about 0.75 s. With the bulkhead, about 85% of the payments were rejected (10 calls / 0.7 s ≈ 14 of the 70 per second fit) and browse p95 was about 25 ms. A bulkhead is a budget of concurrency per dependency — named after the watertight compartments of a ship.
| Pattern | Protects against | Cost |
|---|---|---|
| Timeout | waiting forever on a slow dependency | some slow-but-successful calls become errors |
| Retry + backoff + jitter | transient errors | more load on the dependency, more latency; needs idempotency |
| Circuit breaker | hammering a dependency that is down | rejects calls for wait-duration even if it recovered sooner |
| Bulkhead | one dependency consuming all threads | rejects calls above the limit even if threads are free elsewhere |
| Fallback | showing a raw error | must be honest: "try again", not a fake success |
In the interview
"Every remote call gets a timeout shorter than its caller's, retries only on transient errors with exponential backoff and jitter, on idempotent operations, behind a circuit breaker; each dependency gets its own bulkhead so one slow dependency cannot exhaust the service."