Overview

Picture the store on a Black Friday evening. The product service is healthy, the database is healthy, but the payment provider starts answering in 3 seconds instead of 100 ms. Within a minute the gateway stops answering everything — login, catalogue, cart — even though nothing on those routes is broken. Nobody wrote a bug; the system simply had no defence against a slow dependency. This class is about building the defences, and about deciding with numbers how much failure is acceptable.

Reliability is the probability that a system performs its required function without failure, under stated conditions, for a specified period of time¹. In distributed systems, this definition expands beyond hardware to encompass software failures, network partitions, overloaded dependencies, and incorrect deployments.

A reliable system is not one that never fails. It is one that fails gracefully, recovers quickly, and limits the blast radius of any single failure on its users. (The blast radius is the set of users and components affected by one failure.)

Roadmap

1. This overview — availability, the "nines", composite availability, failure modes and cascading failures.
2. SLI / SLO / SLA — how to measure reliability, set targets, and turn the gap into an error budget.
3. Resilience Patterns — timeout, retry, circuit breaker, bulkhead, rate limiting and health checks, and how to layer them.
4. Chaos Engineering — injecting failures on purpose to prove that the patterns work.
5. Quiz — check your understanding.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Availability

The most visible expression of reliability is availability — the fraction of time a system is in a functioning state. Reliability and availability are related but not identical: a service that crashes for one second every minute has poor reliability (it fails often) but still about 98% availability; a service that fails once a year but takes a week to repair is very reliable and poorly available. Availability is derived from two measurable quantities:

Key insight: reliability improves by either preventing failures (↑ MTTF) or recovering faster (↓ MTTR). Most engineering investments target one of these two levers. Incident response, automation, and runbooks primarily attack MTTR. Fault-tolerant design, testing, and redundancy primarily attack MTTF.

The "nines" of availability

Availability targets are usually quoted as a number of "nines". The table below shows how much downtime each target allows, per window (year = 365.25 days, month = 1/12 of a year ≈ 30.44 days):

Each additional "nine" is a 10× reduction in allowable downtime. Moving from three nines to four nines is not an incremental improvement — it requires a fundamentally different operational posture. At five nines, human reaction time is too slow; detection and recovery must be fully automated.

Composite availability

Real requests cross several components, so the availability the user sees depends on how the components are combined³.

flowchart LR
    user([User]) --> lb["Nginx"]
    lb --> g1["gateway #1<br/>99%"]
    lb --> g2["gateway #2<br/>99%"]
    g1 --> order["order<br/>99.9%"]
    g2 --> order
    order --> product["product<br/>99.9%"]

This is why redundancy (multiple instances) and isolation patterns (circuit breakers) exist: to break the failure-propagation chain.

Failure modes

Distributed systems fail in ways that single-machine systems do not. Partial failures — where some components work and others do not — are far more common than total outages, and substantially harder to handle correctly.

flowchart TD
    F[Failure] --> T["Transient<br/>Temporary, self-healing<br/>e.g. brief network blip"]
    F --> P["Permanent<br/>Requires intervention<br/>e.g. disk crash, OOM kill"]
    F --> I["Intermittent<br/>Unpredictable recurrence<br/>e.g. memory leak, GC pause"]
    P --> CS["Crash-stop<br/>Node halts and stays halted"]
    P --> BZ["Byzantine<br/>Node behaves arbitrarily or incorrectly"]

The most common failure in practice is not a crash but a slow response: the caller cannot tell a slow node from a dead one (see Distributed Systems), so it keeps waiting.

Cascading failures

A single slow dependency can exhaust resources in its caller — thread pool slots, connection pool entries, memory — and propagate that exhaustion upstream through the entire service graph. This is the most dangerous failure pattern in microservice architectures².

sequenceDiagram
    participant U as Users
    participant O as Order service (200 threads)
    participant P as Payment service
    U->>O: POST /orders (many per second)
    O->>P: charge (normally 100 ms)
    Note over P: degrades: 3 s per call
    Note over O: threads wait 30× longer → all 200 busy
    U->>O: GET /orders/42 (no payment needed)
    O-->>U: 503 — no free thread
    Note over U,O: unrelated routes fail too

The arithmetic is simple (Little's law: requests in flight = arrival rate × time each one takes). At 60 orders per second and 100 ms per payment call, about 6 threads are busy waiting on payment. At 3 s per call, 180 threads are — almost the whole pool — and every other route of the order service starves. If the gateway in turn waits on the order service, the exhaustion climbs one more level. Timeouts, circuit breakers, and bulkheads (Resilience Patterns) are the engineering response to this failure mode.

Connections

• Distributed Systems — partial failure and the impossibility of telling slow from dead.
• Observability — SLIs are computed from metrics; you cannot run SLOs or chaos experiments without it.
• Event-Driven Architecture — asynchronous messaging removes the caller's wait from the chain; DLQs handle poison messages.
• Hands-on 5 (Kubernetes self-healing), Hands-on 7 (replicas behind a load balancer) and Hands-on 6 (Prometheus and Grafana for SLIs).

Key takeaways

• Availability = MTTF / (MTTF + MTTR); faster recovery is often the cheapest improvement.
• Each extra nine is 10× less downtime — and always state the window a number refers to.
• In series availabilities multiply (worse than any part); in parallel, redundancy gives \(1-(1-a)^n\).
• Partial and slow failures are the norm; a slow dependency exhausts its caller's resources and cascades upstream.
• The rest of the class measures reliability (SLOs, error budgets), contains failures (resilience patterns), and proves the containment works (chaos engineering).