Distributed Systems Fundamentals

10:00:00 — the on-sale opens

• On one machine the function returned or threw — the question would not exist.
• BoxOffice runs on dozens of machines connected only by a network.
• A distributed system: nodes that cooperate only by messages, with no shared memory and no shared clock.

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." — Leslie Lamport

Why distribute — and what it costs

At scale failure is the norm: 1 − (1 − p)^N. With p = 0.1% per machine per day, 3,000 machines see a failure on ~95% of days.

Failure models

The 8 fallacies of distributed computing

1. The network is reliable
2. Latency is zero — same DC ≈ 0.5 ms RTT, across continents ≈ 150 ms
3. Bandwidth is infinite
4. The network is secure
5. Topology doesn't change
6. There is one administrator
7. Transport cost is zero
8. The network is homogeneous

Deutsch et al., Sun Microsystems (1994; Gosling added the 8th in 1997). Each one returns in a later class.

Slow or dead?

inventory → payment: authorise R$ 450 … silence … timeout after 2 s.

• (a) request lost · (b) payment crashed · (c) reply lost · (d) payment just slow
• All four look identical to the caller.
• A timeout lets you assume failure — not know it.
• In (c) and (d) the charge happened: a blind retry charges twice.

Delivery semantics

Effectively once = at-least-once delivery + idempotent processing.

Making retries safe

• Idempotent: applying twice = applying once. GET, PUT, DELETE yes; POST, x = x + 1 no.
• Conditional write (compare-and-set): the duplicate changes nothing.

UPDATE seats SET status = 'HELD', fan_id = 42, hold_key = :key
 WHERE id = 'A-12' AND (status = 'FREE' OR hold_key = :key);

• Idempotency key: the client sends a UUID per intended action; the server stores key + result in the same transaction and replays the stored result on a duplicate.
• Store keys in shared storage — the retry lands on another replica.

Replication models

When to say OK: async (fast, may lose writes on failover) · sync (slowest follower decides) · semi-sync / quorum (the usual compromise).

Replication lag in the wild

• Quiet day, mean lag 20 ms → almost never stale.
• On-sale, mean lag 800 ms → stale for ~69% of fans at 300 ms (exponential model, illustrative).
• Under load, the bug is not rare — it is expected.

The consistency spectrum

• Linearizable for seats, money, uniqueness, leaders.
• Session guarantees via leader reads after a write, LSN tokens, sticky replicas.
• Eventual conflicts: LWW (loses writes), version vectors, CRDTs.

CAP: during a partition, choose

PACELC: the trade-off you pay every day

Partition → A or C; Else → Latency or Consistency.

Hence: synchronous across zones (EC), asynchronous or home-region partitioning across regions (EL). Illustrative orders of magnitude.

BoxOffice: one choice per data domain

Multi-region: each event has a home region (the venue's) — strong consistency stays local, a partition only affects cross-region holds.

Consensus: agreeing on one log

• Replicated state machine: same commands, same order → same state on every node.
• Safety always (never two different decisions); liveness when the network behaves.
• FLP: no deterministic algorithm always terminates in an asynchronous system → timeouts.
• Majorities overlap → no split brain.

Use 3 or 5 nodes, spread so that no zone holds a majority. Two nodes at 99% each = 98.01%.

Raft in one slide

• Follower → Candidate → Leader; time divided into terms, at most one leader per term.
• No heartbeat for an election timeout (randomised) → become candidate, term + 1, request votes.
• One vote per term; vote only for a log at least as up-to-date (election restriction).
• Leader appends, replicates with AppendEntries, commits after a majority wrote to disk.
• A stale leader sees a higher term → steps down.

etcd defaults: heartbeat 100 ms, election timeout 1,000 ms → a leader crash costs ~1–2 s of no writes. Used by etcd, Consul, CockroachDB, Kafka KRaft.

Leaderless quorums: R + W > N

Leases need fencing tokens

1. Worker A gets the lease, token 33 — then a 15 s GC pause; the lease expires.
2. Worker B gets the lease, token 34, writes with 34.
3. A wakes up, still "holds the lock", writes with 33 → storage rejects 33 < 34.

UPDATE scheduled_jobs SET last_run_at = now(), fencing_token = :token
 WHERE job_id = 'onsale-reminder-42' AND fencing_token <= :token;

Atomic across services? Two-phase commit

1. Prepare: every participant writes its changes + a prepared record, keeps the locks, votes yes/no.
2. Commit: all yes → coordinator logs COMMIT and tells everyone; any no → ABORT.

2PC lives on inside distributed databases with a replicated coordinator (Spanner, CockroachDB, Kafka transactions) — not across microservices.

Sagas: compensate instead of lock

• Orchestrated (one brain, e.g. order or Temporal) or choreographed (events).
• No isolation → semantic locks with expiry; every step idempotent.
• Each step writes DB + event atomically with a transactional outbox.

No global clock

• Quartz drifts: 20 ppm ≈ 1.7 s/day; NTP keeps servers within ~ms — if it keeps running.
• Wall clock (currentTimeMillis) can jump → only for human timestamps.
• Monotonic clock (nanoTime) → durations, timeouts.

Ordering without trusting clocks

The concept map

Every component in the playground inherits one of these problems — ask which one before adding it to a design.

Takeaways

1. Distribute only for scale, fault tolerance, latency, isolation, residency; failure becomes the norm.
2. Slow ≈ dead to the caller: timeouts + idempotent retries (conditional writes, idempotency keys).
3. Exactly-once delivery is impossible → at-least-once + idempotent processing.
4. Pick a consistency model per data domain: linearizable for seats and money, read-your-writes for a user's data, eventual for the catalogue.
5. CAP during partitions, PACELC every day; partition by home region; sagas + outbox, not 2PC, across services.
6. Majorities (3 or 5 nodes), Raft, R + W > N, fencing tokens — use etcd, don't build it.
7. Never order events by timestamps from different machines.