System Architecture · Insper

Reliability and Resilience

Fail gracefully, recover quickly, limit the blast radius

10:02 on the stadium on-sale

The payment provider goes from 200 ms to 4 s. Nothing in BoxOffice is broken.
Why do fans who only want to browse the catalog get errors a minute later?

  • order waits on payment with no deadline → its threads fill up.
  • The gateway waits on order → its threads fill up too.
  • Every route shares those threads → the slowness spreads to healthy features.

Availability: two levers

Term Meaning
MTTF mean time to failure — how long it runs
MTTR mean time to repair — detect + diagnose + fix

Availability = MTTF / (MTTF + MTTR)

servingon average for MTTF
failure →
failed
alert →
detected
traces →
diagnosed
restart →
serving
  • inventory: MTTF 720 h, MTTR 2 h → 99.72%. Halve failures → 99.86%. Auto-restart in 1 min → 99.998%.

Recovering faster is usually the cheaper lever — and each arrow of the MTTR is a separate number to cut.

The nines

Availability Per month Per 28 days Operational posture
99.9% 43.8 min 40.3 min a human can roll back
99.95% 21.9 min 20.2 min fast alerts, rehearsed runbooks
99.99% 4.38 min 4.03 min automated detection and failover
99.999% 26.3 s 24.2 s automated in seconds; deploys are the risk

Each nine is 10× less downtime — and always state the window (and time vs requests).

Composite availability

fan
→
load balancer99.99%
→
in parallel: any one
gateway × 31 − 0.005³
→
in parallel
order × 3
inventory × 3
→
SQL99.9%: one copy
→
provider99.9%: one contract
  • In series every component is needed: A = A₁ × A₂ × … × Aₙ — every synchronous dependency lowers the ceiling.
  • In parallel any one is enough: A = 1 − (1 − a)ⁿ — two replicas at 99% → 99.99%, if failures are independent.

Purchase path (illustrative): 1 instance each ≈ 98.3% (745 min/month) → 3 replicas ≈ 99.79% → + database failover ≈ 99.84% → + second payment provider ≈ 99.94% (26 min/month). Each step exposes the next weakest link.

Simulator: where do the minutes go?

1 instance each98.30% · 745 min/month
2 replicas →
99.78%95 min · weakest: SQL
failover + 2nd provider →
99.94%26 min
all in one zone →
99.84%70 min · weakest: the zone
  • The overview page's availability-path simulator: replicas per service, SQL failover, a second provider, one availability zone (AZ) for everything.
  • A third replica adds almost nothing (99.79%): replicate the weakest link, not the easiest one.

Redundancy counts only if its failures are independent — one zone (one config, one host) fails them all.

Redundancy and failover

Pattern Failover time BoxOffice example
Active-active seconds (health checks) stateless gateway, order, catalog replicas
Active-passive (hot standby) seconds to minutes PostgreSQL primary + standby in another AZ
Cold standby minutes to hours analytics workers rebuilt from images
  • N + 1: after losing one replica (or one AZ of three), the rest carry the peak.
  • Stateless fails over trivially; state needs replication, fencing, no split brain.
  • AZs are ~1 ms apart (cheap protection); regions are tens of ms apart (disaster recovery).

How distributed systems fail

how long?
transient · intermittent · permanent
×
how does it behave?
crash-stop · slow · Byzantine
Class Handling
Crash (fail-stop) health checks, restart, failover
Transient retry with backoff and jitter
Slow / gray failure timeouts, breakers that count slow calls, latency service level indicators (SLIs)
Byzantine / incorrect checksums, validation, reconciliation

The most dangerous component is not the dead one — it is the slow one. The caller cannot tell slow from dead, so without a deadline it waits.

Cascading failure, in numbers

fans100 purchases/s
→
gatewaythreads waiting on order
→
order200 threads, 400 needed
→
payment provider4 s instead of 200 ms

calls in flight = arrival rate × time per call (Little's Law)

Payment latency In flight at 100/s order pool of 200
200 ms 20 10% busy
1 s 100 50% busy
4 s 400 exhausted → every route waits

Timeout caps the time per call · circuit breaker stops calling · bulkhead caps the threads. Playground: cascading-failure.

Failures that feed themselves

  • Retry storms: retries multiply load when the dependency is weakest.
  • Thundering herd on recovery: everyone reconnects at the same instant.
  • Health-check death spiral: overloaded instances are removed, the rest get more load.
  • Metastable failure: the trigger is gone, but the work it created (retries, timeouts, misses) keeps the system down.

The way out is to drop load: load shedding, retry budgets, circuit breakers.

Timeouts and deadline budgets

edge10 s
>
gateway5 s
>
order → payment1 s × 3 attempts + backoff ≈ 3.6 s
but
payment → provider2 s: longer than its caller
  • Near the dependency's normal p99.9: lab 9 uses 500 ms to connect and 1 s to read (PAYMENT_READ_TIMEOUT).
  • Per dependency, shrinking with depth; propagate the deadline (gRPC deadlines, a header over HTTP).
  • A timeout on a charge leaves an unknown outcome → idempotency key order-<id>.

The lab's adapter waits 2 s for the provider: `order` gives up first and `payment` works for nobody. In production, give the adapter less than its caller.

Retries: amplification, jitter, budgets

  • Retry transient failures only (connect errors, 503, 429); never 4xx or "card declined".
  • One layer: 3 layers × up to 3 attempts = 27 calls per click.
  • Waits add up: 1, 2, 4 s → retries at 1, 3 and 7 s after the first failure.
  • Full jitter: wait = random(0, base · 2ⁿ) — 1,000 clients failing together recover in ~1.5 s instead of waves for 15 s.
  • Retry budget (retries ≤ 10% of traffic): without it, retries can keep a system metastably down.

Lab 9: 3 attempts, 200 ms × 2 ± 50%, on I/O errors, 5xx and resets. Playground: retry-storm.

Circuit breaker and fallback

closedcalls flow · last 20 counted (min 10)
≥ 50% failed or > 800 ms →
openfail in µs for 10 s
10 s →
half-open3 probes decide
  • Opens at a rate ≥ the threshold; slow calls have their own rate. Half-open judges the probes' rates.
  • Slow-call threshold below the timeout (800 ms < 1 s): a call past the timeout is a failure, never "slow".
  • Resilience4j defaults: window 100, minimum 100, wait 60 s, 10 probes, slow calls at 60 s / 100% — far too slow to react.
  • 10 s open = up to ~10 s of "payment temporarily unavailable" after the provider recovers. Never fake success.

Bulkheads and load shedding

Bulkhead

order50 threads
→
payment10 permits, no queue
everything else40 threads
  • 20 payments/s × 0.2 s = 4 in flight → 10 = 2.5× slack.
  • Provider at 3 s → 10 threads wait, not 60.
  • Cost: capacity that cannot be shared.

Load shedding

  • Reject early and cheap (503 + Retry-After).
  • By priority: purchase > browsing > recommendations.
  • Drop stale work; adaptive concurrency limits.

Rate limiting is per client (token bucket at the gateway); load shedding protects the whole system. The waiting room is load shedding done kindly.

The stack on order → payment (lab 9)

charge
→
Retry3 attempts
→
CircuitBreakeropen? 503
→
Bulkheadfull? 503
→
HTTP500 ms / 1 s
resilience4j:
  retry.instances.payment:
    max-attempts: ${PAYMENT_RETRY_ATTEMPTS:3}   # 200 ms, 400 ms, ± 50%
  circuitbreaker.instances.payment:
    sliding-window-size: 20                      # min 10 calls
    failure-rate-threshold: 50
    slow-call-duration-threshold: 800ms
    wait-duration-in-open-state: 10s
  bulkhead.instances.payment:
    max-concurrent-calls: ${PAYMENT_BULKHEAD:10}
    max-wait-duration: 0

Outermost first, each for a reason: the retry outside the breaker (an open breaker stops it), the fallback on the retry (runs once), the timeout innermost (bounds each attempt). Same order in Microsoft's .NET standard resilience handler. Measured: GET /orders/{id} median 9.04 s unprotected, 6.5 ms protected.

Simulator: what stops a slow provider?

nothing50 threads busy · ~220 payments with no thread
→
timeout~20 threads · 0% succeed
→
+ retries2× load · pool full again
→
+ breakeropens at 1.6 s · ~1 thread
  • The patterns page's resilience simulator: provider at 4 s, lab 9's switches, one at a time.
  • Drag the latency back to 200 ms: 10 s after opening, half-open, 3 probes, closed.

Timeout + retries without a breaker is worse than nothing: 0 payments succeed per 10 s (128 with nothing on), the pool is full again and the provider takes 2× the load.

Health checks

Probe Question On failure
Startup finished booting? keep waiting (up to a limit)
Liveness process stuck? restart the container
Readiness can serve traffic now? remove from the load balancer

Never put dependencies in liveness: if order's liveness checks the payment provider, one provider blip restarts every order replica at once.

Spring Boot 4: /actuator/health/liveness and /readiness are on by default (before, only on Kubernetes).

Disaster recovery (DR): how much data, how much time?

  • Recovery point objective (RPO): how much data (in time) you accept to lose.
  • Recovery time objective (RTO): how long until the service is back.
  • Set per data class: payments ≈ 0 / 15 min · catalog 1 h / 1 h · analytics 24 h / 2 days.

Replication is not backup: the standby replicates DELETE FROM orders in milliseconds. Logical failures need backups with point-in-time recovery (PITR) — and a measured restore time (2 TB at 200 MB/s ≈ 2.8 h).

3-2-1 backups + one immutable copy; separate account; test restores monthly (remember GitLab, 2017).

The four disaster-recovery strategies

backup and restorebackups + infrastructure as code
RPO hours · RTO hours–day · $
→
pilot lightdata replicated, services off
RPO minutes · RTO tens of min · $$
→
warm standbyfull copy, scaled down, running
RPO s–min · RTO minutes · $$$
→
active-activefull copies, all serving
RPO ≈ 0 · RTO ≈ 0 · $$$$
  • Cross-region replication is usually asynchronous: RPO = replication lag.
  • BoxOffice tier 0 (order, payment, inventory): warm standby — restore is too slow, active-active double-sells seats.

Data residency (Brazil's LGPD, Europe's GDPR) constrains which regions can hold the copy.

Regional failover

RTO ≈ detect (2 min) + decide (5) + promote the database (1) + scale up (5) + Domain Name System (DNS) time to live (TTL) (1) ≈ 14 min

  • Detect from outside the region; decide with a rehearsed runbook.
  • Fence the old primary — two primaries = double-sold seats.
  • Static stability: images, secrets, quotas already in region B.
  • Seats cannot be multi-leader: home region per event + warm standby, or a globally consistent database.

Playground: multi-region — fail the primary region, then change the DNS TTL.

Chaos engineering

steady stateSLIs
→
hypothesis
→
injectone real fault
→
observe
→
analyse
→
improvethen widen
  • BoxOffice: Toxiproxy adds 3 s to the provider under lab 9's k6 load → expect the breaker open < 10 s and browse flat — purchases are not in the steady state, the hypothesis expects them to fail fast.
  • Small blast radius: Compose → staging → one canary pod → wider; abort on burn rate > 10×.
  • Observability first: dashboards show the drop in seconds, alerts in minutes — then break things.

Netflix wrote the principles down (principlesofchaos.org, IEEE Software 2016). Tools: Toxiproxy, Chaos Mesh, AWS Fault Injection Service (FIS, renamed from Fault Injection Simulator in Nov 2023).

Game day: the flash sale

Time Injection Expected
10:10 provider latency 3 s breaker opens < 10 s; seats held
10:25 one order pod killed per minute N+1 absorbs it
10:40 Kafka broker stopped outbox grows and drains; no lost e-mails
10:55 database failover ~30 s of write errors
11:10 Redis flushed stampede protection holds

Roles: game master, on-call, observers, scribe. Output: a blameless list of improvements.

The reliability concept map

Component Question it answers Key concepts
lb, app does losing a replica or a zone stop us? nines, series/parallel, N+1, multi-AZ
thirdparty, payment what if a dependency gets slow? timeout, retry + jitter, breaker, bulkhead
apigw what do we refuse under overload? rate limiting, load shedding, waiting room
healthcheck should traffic reach this instance? liveness, readiness, startup
metrics, alerting are we reliable enough? SLI, service level objective (SLO), error budget, burn rate (class 9)
sql, objstore, dns how much data and time can we lose? RPO/RTO, PITR, DR strategies, failover

Redundancy keeps instances serving, patterns contain slow dependencies, DR covers what redundancy cannot, SLOs say how much is enough — and chaos proves all of it.

Takeaways

  1. Availability = MTTF / (MTTF + MTTR); series multiply, parallel 1 − (1 − a)ⁿ — only with independent failures.
  2. Lowering the MTTR is usually cheaper than raising the MTTF.
  3. Slow dependencies cascade through exhausted pools — Little's Law tells you how fast.
  4. Timeout · retry with jitter · breaker counting slow calls · bulkhead · honest fallback — sized with numbers.
  5. Redundancy is bought: the error budget decides how much; DR starts from RPO/RTO, and replication ≠ backup.
  6. Chaos experiments and game days prove the defences — observability first.

Outline: (1) availability, nines, composite availability, redundancy; (2) failure modes and cascades; (3) resilience patterns, with lab 9's values; (4) disaster recovery; (5) chaos engineering and game days. The SLOs that judge all of it come from class 9.

Let the class reason. The rest of the class: measure how much failure is acceptable, build defences, recover from disasters, and prove the defences work.