System Architecture · Insper

Distributed Systems Fundamentals

No shared memory, no shared clock, and a network that lies

10:00:00 — the on-sale opens

A fan on a train taps Hold seat. The spinner turns for 5 s, then "Something went wrong".
She taps again. Did the first tap hold a seat? Will she hold two? Be charged twice?

  • On one machine the function returned or threw — the question would not exist.
  • BoxOffice runs on dozens of machines connected only by a network.
  • A distributed system: nodes that cooperate only by messages, with no shared memory and no shared clock.

"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." — Leslie Lamport

Why distribute — and what it costs

Reasons

  • Scale beyond one machine
  • Fault tolerance: no single point of failure
  • Latency: be close to users
  • Isolation: teams, blast radius
  • Data residency (Brazil's LGPD, Lei Geral de Proteção de Dados; Europe's GDPR, General Data Protection Regulation)

Costs

  • Partial failure
  • Unreliable network
  • No global clock
  • Copies diverge (consistency)
  • Coordination and operations

At scale failure is the norm: 1 − (1 − p)N. With p = 0.1% per machine per day, 3,000 machines see a failure on ~95% of days.

BoxOffice is a distributed system

Fan's phone
POST /holds →
gatewayvalidates the token with auth
→
inventory
UPDATE →
PostgreSQL primary
replicates →
PostgreSQL replicathrough the write-ahead log (WAL)
inventory also →
Rediscaches the seat map
and
Kafka, 3 brokersSeatHeld event
consume →
notification
  • Every arrow is a message over the network; every box can fail, pause or slow down on its own.
  • Redis slow → inventory waits · replica lags → stale read · Kafka unreachable → hold committed, SeatHeld never published.

Nothing shares memory or a clock: each of these is a partial failure.

Failure models

Model The faulty node… BoxOffice example
Crash-stop halts forever virtual machine (VM) terminated
Crash-recovery halts, restarts from disk inventory pod killed out of memory (OOM)
Omission drops messages the SeatHeld event never sent
Timing answers too late 3 s garbage-collection (GC) pause in payment
Byzantine lies, corrupts forged ticket, bit flip
Gray looks healthy, fails users /health 200, every query times out

The failures that hurt are not clean crashes: they are slow, partial and gray.

The 8 fallacies of distributed computing

  1. The network is reliable
  2. Latency is zero — same data centre ≈ 0.5 ms round-trip time (RTT), across continents ≈ 150 ms
  3. Bandwidth is infinite
  4. The network is secure
  5. Topology doesn't change
  6. There is one administrator
  7. Transport cost is zero
  8. The network is homogeneous

Deutsch et al., Sun Microsystems (1994; Gosling added the 8th in 1997). Each one returns in a later class.

Slow or dead?

inventoryauthorise R$ 450
silence →
(a) request lostnot charged
(b) payment crashednot charged
(c) reply lostcharged
(d) just slowcharged
→ timeout 2 s
  • All four look identical to the caller: a timeout lets you assume failure — not know it.
  • In (c) and (d) the charge happened: a blind retry charges twice.

Timeouts are a trade-off (too long: threads pile up; too short: false alarms and extra load). Retries must be safe → idempotency.

Slow or dead? The simulator

give upat most once · 9 holds with no order · 13 told "failed" for an order that exists
retry, timeout 2 sat least once · 13 holds with two orders
retry, timeout 300 ms26 holds with two orders
retry + dedupe by hold0 duplicates at any timeout
  • On the overview page: the gateway turns 200 seat holds into orders; 5% of requests and of replies are lost; at most 3 attempts.
  • A longer timeout only removes case (d): at 5 s, lost replies still make 6 duplicates.
  • Dedupe is lab 7's ON CONFLICT (hold_id) DO NOTHING; the payment version (idempotency key, lab 9) is on the integrations page.

Delivery semantics

Semantics How Risk
At most once send, never retry lost messages
At least once retry until acknowledged duplicates
Exactly once impossible over an unreliable network alone —

Effectively once = at-least-once delivery + idempotent processing.

"Exactly-once delivery" without qualification is a red flag in an interview.

Making retries safe

  • Idempotent: applying twice = applying once. GET, PUT, DELETE yes; POST, x = x + 1 no.
  • Conditional write (compare-and-set): the duplicate changes nothing.
UPDATE seats SET status = 'HELD', fan_id = 42, hold_key = :key
 WHERE id = 'A-12' AND (status = 'FREE' OR hold_key = :key);
  • Idempotency key: the client sends a universally unique identifier (UUID) per intended action; the server stores key + result in the same transaction and replays the stored result on a duplicate.
  • Store keys in shared storage — the retry lands on another replica.

The concept map

Partial failure
some nodes fail, nobody sees which
replication · consistency models
CAP / PACELC · multi-region
consensus · quorums · fencing
sql · nosql · cache · cdn · dns
Unreliable network
messages lost, delayed, duplicated
timeouts · retries · idempotency
sagas · outbox · 2PC
payment · queue · stream · kafka
No global clock
clocks drift and disagree
logical / hybrid clocks
single authority · ordered ids
kafka · sql · scheduler

Red: the root problem · blue: the techniques of this class · green: palette components where you meet it in the playground. Every component inherits one of these problems — ask which one before adding it to a design.

Replication models

Single-leader Multi-leader Leaderless
Who writes one leader one per region any replica
Write conflicts ✅ none ⚠️ yes ⚠️ yes
Strong consistency ✅ possible ❌ not across leaders ⚠️ only approximate
Examples PostgreSQL, Kafka partitions multi-region setups, CouchDB Cassandra, Dynamo

When to say OK: async (fast, may lose writes on failover) · sync (slowest follower decides) · semi-sync / quorum (the usual compromise).

Replication lag in the wild

The fan pays at 10:00:41, opens My tickets 300 ms later — served by an async replica. The list is empty.

  • Quiet day, mean lag 20 ms → almost never stale.
  • On-sale, mean lag 800 ms → stale for ~69% of fans at 300 ms (exponential model, illustrative).
  • Under load, the bug is not rare — it is expected.

Fix it precisely: read-your-writes, not global strong consistency.

The consistency spectrum

Model Promise Available in a partition?
Linearizable behaves like a single copy, real-time recency ❌ no
Causal cause before effect, for everyone ✅ yes
Read-your-writes you see your own writes yes (per session)
Monotonic reads your view never goes back in time yes (per session)
Eventual replicas converge if writes stop yes
  • Linearizable for seats, money, uniqueness, leaders.
  • Session guarantees via leader reads after a write, log sequence number (LSN) tokens, sticky replicas.
  • Eventual conflicts: last-write-wins (LWW) loses writes; version vectors; conflict-free replicated data types (CRDTs).

CAP: during a partition, choose

Consistency, Availability, Partition tolerance (CAP): a replicated store cannot guarantee all three.

Fan Aholds A-12
→
node A · majorityA-12 = HELD
✗ partition ✗
node B · minorityA-12 = FREE
←
Fan Bis A-12 free?
CP — node B refuses: 503, no double-sold seatetcd, ZooKeeper, Spanner, PostgreSQL sync
AP — node B answers "FREE": reconcile afterwardsCassandra (defaults), DNS, CDN, caches

P is not optional — "CA" is not a choice for replicated data. And CAP's C is linearizability, not the C of ACID (atomicity, consistency, isolation, durability).

PACELC: the trade-off you pay every day

Partition → A or C; Else → Latency or Consistency.

Confirming replicas are… Extra RTT per consistent write
same rack < 0.5 ms
another zone, same region ~1–2 ms
another region, same continent ~10–40 ms
another continent ~150–200 ms

Hence: synchronous across zones (EC), asynchronous or home-region partitioning across regions (EL). Illustrative orders of magnitude.

BoxOffice: one choice per data domain

Data Choice How
Seats, holds, payments CP / linearizable one primary per event, conditional UPDATE
My tickets read-your-writes user's own data from the primary
Seats-left counter monotonic reads sticky replica / short time-to-live (TTL) cache
Catalogue, images AP / eventual replicas, cache, CDN
Interest counters eventual (CRDT) per-region counters merged

Multi-region (active–active partitioned by home): each event has a home region (the venue's) — strong consistency stays local, a partition only refuses the holds that must cross it.

Consensus: agreeing on one log

  • Replicated state machine: same commands, same order → same state on every node.
  • Safety always (never two different decisions); liveness when the network behaves.
  • FLP (Fischer, Lynch, Paterson): no deterministic algorithm always terminates in an asynchronous system → timeouts.
  • Majorities overlap → no split brain.

Split brain, and why the majority prevents it

majority side · 3 of 5
node 1 · leader
node 2
node 3
✗ partition ✗
minority side · 2 of 5
node 4
node 5
elects and commits: keeps selling seats
cannot reach 3 votes: rejects writes, 503
N 1 2 3 4 5 7
Failures tolerated 0 0 1 1 2 3

Use 3 or 5 nodes, spread so that no zone holds a majority. Two nodes at 99% each = 98.01%.

Raft in one slide

  • Follower → Candidate → Leader; time divided into terms, at most one leader per term.
  • No heartbeat for an election timeout (randomised) → become candidate, term + 1, request votes.
  • One vote per term; vote only for a log at least as up-to-date (election restriction).
  • Leader appends, replicates with AppendEntries, commits after a majority wrote to disk.
  • A stale leader sees a higher term → steps down.

etcd defaults: heartbeat 100 ms, election timeout 1,000 ms → a leader crash costs ~1–2 s of no writes. Used by etcd, Consul, CockroachDB, Kafka KRaft (Kafka Raft).

Leaderless quorums: R + W > N

N = 3 W R R + W > N? Behaviour
ONE / ONE 1 1 ❌ no fastest, may be stale
QUORUM / QUORUM 2 2 ✅ yes sees latest ack'd write, tolerates 1 down
ALL / ONE 3 1 ✅ yes one node down blocks writes

Overlap is necessary, not sufficient: sloppy quorums, concurrent writes and LWW still produce anomalies. Tunable consistency ≠ consensus.

Leases need fencing tokens

  1. Worker A gets the lease, token 33 — then a 15 s GC pause; the lease expires.
  2. Worker B gets the lease, token 34, writes with 34.
  3. A wakes up, still "holds the lock", writes with 33 → storage rejects 33 < 34.
UPDATE scheduled_jobs SET last_run_at = now(), fencing_token = :token
 WHERE job_id = 'onsale-reminder-42' AND fencing_token <= :token;

BoxOffice: Patroni + 3-node etcd elects the PostgreSQL primary; the old primary demotes itself. Use consensus, don't build it.

Atomic across services? Two-phase commit (2PC)

  1. Prepare: every participant writes its changes + a prepared record, keeps the locks, votes yes/no.
  2. Commit: all yes → coordinator logs COMMIT and tells everyone; any no → ABORT.

Coordinator crashes after the votes → participants are in doubt, seat A-12 stays locked. Availabilities multiply (all up at once), two round trips + fsyncs, and the payment provider cannot join.

2PC lives on inside distributed databases with a replicated coordinator (Spanner, CockroachDB, Kafka transactions) — not across microservices.

Sagas: compensate instead of lock

Step Local transaction Compensation
1 order PENDING order CANCELLED
2 seat → RESERVED_FOR_PAYMENT release the seat
3 charge the card — the pivot once it succeeds refund
4 seat SOLD, order CONFIRMED — retry until it succeeds
  • Orchestrated (one brain, e.g. order or Temporal) or choreographed (events).
  • No isolation → semantic locks with expiry; every step idempotent.
  • Each step writes DB + event atomically with a transactional outbox.

Reservations: try, confirm, cancel

FREE
try: hold for 10 min →
HELDexpires by itself
confirm: paid →
SOLD
HELD
cancel: release or expiry →
FREE
  • The try–confirm–cancel (TCC) pattern: every participant offers a tentative operation with a timeout.
  • Card payments have the same shape: authorise → capture or void.
  • Every tentative state expires → a crash anywhere converges to a safe state instead of blocking.

No global clock

  • Quartz drifts: 20 parts per million (ppm) ≈ 1.7 s/day; the Network Time Protocol (NTP) keeps servers within ~ms — if it keeps running.
  • Wall clock (currentTimeMillis) can jump → only for human timestamps.
  • Monotonic clock (nanoTime) → durations, timeouts.

Last-write-wins with a node 50 ms fast: a release at .040 stamped .090 beats a hold at .060 → Carla paid and has no seat. The fastest clock wins every conflict.

Ordering without trusting clocks

Tool Gives you Used by
Lamport clock a → b ⇒ L(a) < L(b) Raft terms, fencing
Vector clock detects concurrency Dynamo, Riak
Hybrid logical clock (HLC) causal + close to real time, 64 bits CockroachDB, YugabyteDB, MongoDB
TrueTime bounded uncertainty, commit wait Spanner
Single authority total order from one leader DB sequence, conditional UPDATE
Kafka key → partition order per key events of one show
Snowflake / UUIDv7 unique, roughly time-sorted ids keys — not "who came first"

Takeaways

  1. Distribute only for scale, fault tolerance, latency, isolation, residency; failure becomes the norm.
  2. Slow ≈ dead to the caller: timeouts + idempotent retries (conditional writes, idempotency keys).
  3. Exactly-once delivery is impossible → at-least-once + idempotent processing.
  4. Pick a consistency model per data domain: linearizable for seats and money, read-your-writes for a user's data, eventual for the catalogue.
  5. CAP during partitions, PACELC every day; partition by home region; sagas + outbox, not 2PC, across services.
  6. Majorities (3 or 5 nodes), Raft, R + W > N, fencing tokens — use etcd, don't build it.
  7. Never order events by timestamps from different machines.

Outline: (1) why distribute and failure models; (2) the unreliable network, timeouts and idempotency; (3) replication and consistency models; (4) CAP and PACELC; (5) consensus and quorums; (6) time and clocks.

The page follows one click on "Hold seat A-12" through the same diagram, step by step.

p99 = the 99th-percentile wait: 2.8 s at a 2 s timeout, 5.8 s at 5 s. Play walks through the fans whose first attempt got no reply in time.

DNS = Domain Name System; CDN = content delivery network. The page's simulator plays both modes with a São Paulo – Lisbon link.

BoxOffice already works this way: the hold is the try, the sale the confirm, expiry or release the cancel.