Overview

It is 10:02, two minutes into the stadium on-sale. BoxOffice is holding up: the waiting room admits fans at a steady pace, inventory is at 60% of its capacity. Then the external payment provider starts answering in 4 seconds instead of 200 ms. Within a minute, order has no free threads; the gateway, waiting on order, has none either; and fans who only wanted to browse the catalog get errors. Nothing in BoxOffice is broken — no bug, no crash, no full disk. The system simply had no defence against a slow dependency, and the slowness spread until it took everything down.

This class is about building those defences, and about deciding with numbers how much failure is acceptable. Reliability is the probability that a system performs its required function without failure, under stated conditions, for a stated period of time¹. In distributed systems that definition covers much more than hardware: software bugs, overloaded dependencies, network partitions, bad deploys, lost data centres and operator mistakes.

A reliable system is not one that never fails. It is one that fails gracefully, recovers quickly and limits the blast radius of any single failure — the blast radius being the set of users and components affected by one failure.

Roadmap

1. This overview — availability, MTTF/MTTR, the "nines", composite availability, redundancy and failover, failure modes and cascading failures.
2. Resilience patterns — timeout, retry with backoff and jitter, circuit breaker, bulkhead, load shedding, rate limiting, health checks, with Resilience4j.
3. Disaster recovery — RPO and RTO, backups, multi-AZ and multi-region, and the four DR strategies.
4. Chaos engineering — injecting failures on purpose to prove that the defences work; game days.
5. Quiz — check your understanding.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Availability

The most visible face of reliability is availability: the fraction of time (or of requests) in which the system works. Reliability and availability are related but not the same. A service that crashes for one second every minute fails often (poor reliability) but is still about 98% available; a service that fails once a year but takes a week to repair is very reliable and poorly available. Availability comes from two measurable quantities:

Reliability work pulls one of two levers. Raising MTTF (fewer failures): testing, code review, redundancy, fault-tolerant design, careful deploys. Lowering MTTR (faster recovery): good alerts, dashboards and traces, runbooks, automatic restarts and failover, quick rollbacks. In modern systems MTTR is usually the cheaper lever, and MTTR is itself a sum — time to detect + time to diagnose + time to fix — each part attacked by a different practice (see Observability).

The "nines"

Availability targets are quoted as a number of "nines". The table shows the downtime each target allows (year = 365.25 days, month = 1/12 of a year ≈ 30.44 days):

Each additional nine is 10× less downtime, and it is not an incremental improvement: it changes how you operate. At three nines, a human can be paged, open a laptop and roll back. At four nines, the whole monthly budget is shorter than it takes to wake up; detection and recovery must be automatic. At five nines, even automatic failover must complete in seconds, and a single bad deploy can spend a year's budget.

Composite availability

A request crosses many components, and the availability the user sees depends on how they are combined². Two rules cover almost every case (see also Availability math).

Apply both rules to the BoxOffice purchase path. The numbers are illustrative assumptions, but the shape of the result is general:

Reading the chart from top to bottom:

• With one instance of everything, the three services at 99.5% dominate: about 98.3%, over 12 hours a month without purchases.
• Three replicas of the stateless services make them practically perfect, and the bottleneck moves to the components that are not replicated: the database and the external payment provider.
• Automatic database failover (a standby in another zone) and a second payment provider attack those. Each step removes the current weakest link — and the next weakest becomes visible.

flowchart LR
    fan(["fan"]) --> lb["load balancer<br/>99.99%"]
    lb --> g1["gateway #1"]
    lb --> g2["gateway #2"]
    lb --> g3["gateway #3"]
    g1 --> o["order × 3"]
    g2 --> o
    g3 --> o
    o --> i["inventory × 3"]
    i --> db[("SQL primary<br/>+ standby in AZ-b")]
    o --> p1["payment provider A<br/>99.9%"]
    o -.->|"fallback"| p2["payment provider B<br/>99.9%"]

Redundancy and failover

Redundancy only helps if traffic actually moves away from the failed component — that is failover — and if the redundant copy is ready to take it.

Two rules make redundancy work:

• N + 1 (or N + 2) capacity. If the flash sale needs 6 order replicas, run 7 or 8: after losing one (or during a rolling deploy), the rest must still carry the peak. With replicas spread over 3 availability zones, losing a whole zone removes a third of capacity — plan for it.
• Stateless where possible. Stateless services fail over trivially. State (databases, caches, queues) needs replication, and replication brings the hard questions: synchronous or asynchronous, how much data can be lost, and how to avoid two primaries at once (split brain; see Consensus). The disaster recovery page treats failover across regions.

Failure modes

Distributed systems fail in ways a single machine does not. Partial failures — some components work and others do not — are far more common than total outages, and harder to handle.

The most dangerous of these is not the crash but the slow component. A crashed dependency answers immediately — with a connection error — and the caller moves on. A slow dependency holds the caller's resources while it thinks. The caller cannot tell slow from dead (see Distributed systems), so without a deadline it waits.

Cascading failures

A slow dependency exhausts a resource in its caller — threads, connections, memory — and the exhausted caller becomes slow for its callers. The exhaustion climbs the call graph until unrelated features fail. This is the most common way microservice systems go down³.

sequenceDiagram
    participant F as Fans
    participant G as gateway
    participant O as order (200 threads)
    participant P as payment provider
    F->>G: POST /orders (100 per second)
    G->>O: create order
    O->>P: charge (normally 200 ms)
    Note over P: degrades: 4 s per call
    Note over O: 100/s × 4 s = 400 calls in flight > 200 threads
    F->>G: GET /events/ev-88 (catalog)
    G->>O: (gateway threads also waiting on order)
    G-->>F: 503 — no free thread
    Note over F,G: browsing fails too, although catalog is healthy

The arithmetic is Little's Law (see Queueing): calls in flight = arrival rate × time each call takes.

Nothing about the purchase rate changed. A 20× increase in a dependency's latency became a 20× increase in occupied threads. When the pool is full, every route of order waits — including the ones that never call payment — and the gateway, waiting on order, fills its own pool.

Three things stop the cascade, each covered in Resilience patterns:

• A timeout caps how long one call can hold a thread (1 s timeout → at most 100 threads, not 400).
• A circuit breaker stops calling a dependency that is failing, so calls fail in microseconds instead of seconds.
• A bulkhead gives payment calls their own limited pool, so they cannot take the threads of other routes.

Other amplifiers

Cascades are one example of a broader family: failures that feed themselves.

• Retry storms. Every caller retries failed calls, multiplying the load on a dependency exactly when it is weakest (patterns).
• Thundering herd on recovery. A service comes back and every client, cache and queue consumer hits it at the same instant; it falls over again.
• Death spiral of autoscaling or health checks. Overloaded instances fail their health checks and are removed, so the remaining ones get more load and fail too.
• Metastable failures. The system stays broken after the trigger is gone, because the work generated by the failure (retries, timeouts, cache misses) keeps it overloaded⁴. The fix is to drop load until it recovers — load shedding, retry budgets, circuit breakers.

The reliability concept map

Reliability is not one component: it is a property you buy, piece by piece, across the whole palette. The map links each palette component (green) to the reliability concepts it carries (blue); the arrows show how the class builds up — redundancy and probes keep instances serving, patterns contain slow dependencies, DR covers what redundancy cannot, SLOs say how much is enough, and chaos proves it all works:

flowchart LR
    lbapp["lb · app"]:::comp -.- red["MTTF/MTTR, nines, series vs parallel,<br/>redundancy, N+1, multi-AZ"]:::tech
    tp["thirdparty · payment"]:::comp -.- casc["cascading failures<br/>(Little's Law)"]:::tech
    casc --> pat["timeout, retry + jitter,<br/>circuit breaker, bulkhead, fallback"]:::tech
    apigw["apigw"]:::comp -.- shed["rate limiting,<br/>load shedding"]:::tech
    hc["healthcheck"]:::comp -.- probes["liveness, readiness,<br/>startup"]:::tech
    data["sql · objstore"]:::comp -.- dr["RPO / RTO, PITR backups,<br/>DR strategies"]:::tech
    dns["dns"]:::comp -.- dr
    obs["metrics · alerting"]:::comp -.- slo["SLI, SLO, error budget,<br/>burn rate"]:::tech
    probes --> red
    pat --> shed
    pat --> chaos["chaos experiments,<br/>game days"]:::tech
    dr --> chaos
    slo --> chaos
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed

Related

• Hands-on (the week after this class and class 9): Hands-on 8 — Observability first — OpenTelemetry, a RED dashboard and SLO burn-rate alerts, so that the experiments have something to show; then Hands-on 9 — Resilience & chaos: Resilience4j between order and payment, Toxiproxy latency and faults, a game day. Hands-on 10 — Kubernetes: probes and self-healing.
• Project: Chaos game day.
• Case studies: Payment system, Ticketing (BoxOffice flash sale).
• Classes: Estimation — availability math, Observability (previous class, the same week: the SLOs, dashboards and alerts these defences are judged by), Messaging (asynchrony removes dependencies from the critical path), Cloud-Native (next class: probes, self-healing, autoscaling).

What breaks next

The patterns contain a slow provider, redundancy and the DR plan cover lost machines and regions, and chaos experiments prove both. But every defence on this page assumes that something restarts a dead replica, adds replicas before 10:00 and rolls a bad release back — and on one laptop with Compose, that something is you. The inventory example above improved from 99.72% to 99.998% only because the restart took 1 minute instead of 2 hours; that minute exists only if a machine does the restart. Healing, scaling at 09:45 and rolling back without a human is class 11 — Cloud-Native.