Overview

Reliability is the probability that a system performs its required function without failure, under stated conditions, for a specified period of time¹. In distributed systems, this definition expands beyond hardware to encompass software failures, network partitions, overloaded dependencies, and incorrect deployments.

A reliable system is not one that never fails. It is one that fails gracefully, recovers quickly, and limits the blast radius of any single failure on its users.

Availability

The most visible expression of reliability is availability — the fraction of time a system is in a functioning state. It is derived from two measurable rates:

Key insight: reliability improves by either preventing failures (↑ MTTF) or recovering faster (↓ MTTR). Most engineering investments target one of these two levers. Incident response, automation, and runbooks primarily attack MTTR. Fault-tolerant design, testing, and redundancy primarily attack MTTF.

The "nines" of availability

Each additional "nine" is a 10× reduction in allowable downtime. Moving from three nines to four nines is not an incremental improvement — it requires a fundamentally different operational posture. At five nines, human reaction time is too slow; detection and recovery must be fully automated.

Failure modes

Distributed systems fail in ways that single-machine systems do not. Partial failures — where some components work and others do not — are far more common than total outages, and substantially harder to handle correctly.

flowchart TD
    F[Failure] --> T[Transient\nTemporary, self-healing\ne.g. brief network blip]
    F --> P[Permanent\nRequires intervention\ne.g. disk crash, OOM kill]
    F --> I[Intermittent\nUnpredictable recurrence\ne.g. memory leak, GC pause]
    P --> CS[Crash-stop\nNode halts and stays halted]
    P --> BZ[Byzantine\nNode behaves arbitrarily or incorrectly]

Cascading failures

A single slow dependency can exhaust resources in its caller — thread pool slots, connection pool entries, memory — and propagate that exhaustion upstream through the entire service graph. This is the most dangerous failure pattern in microservice architectures².

flowchart LR
    client -->|many concurrent requests| gateway:::saturated
    gateway -->|requests queue up\nwaiting for reply| payment:::slow
    client:::failed

    classDef slow fill:#f90,color:#000
    classDef saturated fill:#f66,color:#fff
    classDef failed fill:#900,color:#fff

A payment service degrading from 100 ms to 3 s response time causes the gateway's thread pool to fill with blocked threads, starving all other traffic — even routes unrelated to payments. Timeouts, circuit breakers, and bulkheads are the engineering response to this failure mode.

Topics covered