Distributed Systems

Why is this hard?

• Partial failure — each node can fail on its own while the rest keeps running.
• Unreliable network — messages can be delayed, lost or duplicated.
• No shared clock — no global notion of "now".

Our store is a distributed system

One click on Place order:

Browser → Gateway → Auth → Order → Product → Redis · Order → PostgreSQL · Order → Kafka

• Every arrow is a network message; every box can fail on its own.
• Heterogeneous (Java, Python, Postgres, Redis, Kafka), loosely coupled (HTTP, events), client-server.

Slow or dead?

Order → Product: reserve 1 unit of product 42 … silence.

• Request lost? Product crashed? Reply lost? Product just slow? They all look the same.
• The only tool: a timeout — but it does not say whether the operation ran.
• A retry may apply it twice → retried operations must be idempotent (e.g. a unique request id).

No global clock: logical time

Clocks drift — never order events from different machines by currentTimeMillis().

Lamport clock — one counter per process:

1. increment before each event;
2. attach the counter to every message sent;
3. on receive: max(local, received) + 1.

• Guarantee: if a happened-before b, then L(a) < L(b) — not the converse.
• Vector clocks (one counter per process) also detect concurrent events.

Kafka orders by partition offset; Raft by term — single authorities, not wall clocks.

CAP theorem

A system that replicates data cannot guarantee all three at once (Brewer 2000; Gilbert and Lynch 2002):

• C — Consistency: every read returns the most recent write, or an error (linearisability — not ACID's C).
• A — Availability: every request to a live node gets a non-error response, maybe not the latest write.
• P — Partition tolerance: keeps working when the network splits the nodes into groups.

CP or AP?

CA only exists on a single node. CP/AP describe a configuration, not a product — choose per data domain.

PACELC: and when there is no partition?

If Partition → A or C · Else → Latency or Consistency

Consensus — and why it is hard

Consensus: nodes agree on one value (or one log order) despite lost messages and failed nodes — leader election, locks, replicated state machines.

• Safety: nobody decides differently. Liveness: everyone eventually decides.

FLP (1985): fully asynchronous + one node may crash → no deterministic algorithm always terminates.

• Paxos and Raft never break safety; they progress when the network behaves (partial synchrony).
• In practice: timeouts — a slow node looks just like a crashed one.

Failure models

• Datacenter algorithms assume crash-stop or crash-recovery.
• Byzantine fault tolerance: blockchains and safety-critical systems.

Majorities: why 3 or 5 nodes

Any two majorities overlap → two conflicting decisions can never both win (no split brain).

Raft

Designed for understandability (Ongaro and Ousterhout, 2014).

• Election: Follower → randomised election timeout → Candidate → majority vote → Leader. A higher term makes any node step down.
• Replication: client writes x=5 to the leader → AppendEntries → commit after a majority acknowledges → reply.
• Safety: a candidate's log must be at least as up-to-date as the voters'; committed entries survive any minority failure.

Raft: etcd (Kubernetes), Consul, CockroachDB, Kafka KRaft. Paxos (Lamport) is the older, harder ancestor; ZooKeeper uses ZAB.

Quorums

For N replicas, a write needs W acknowledgements and a read needs R responses.

N = 3 (Cassandra)	W + R	Effect
ONE / ONE	2	fastest, reads may be stale
QUORUM / QUORUM	4	latest write; tolerates 1 node down
ALL / ONE	4	one node down blocks every write

Eventual consistency and CRDTs

AP systems promise: with no new updates, replicas eventually converge — but not when.

• Last-Write-Wins — highest timestamp wins; loses concurrent writes, trusts drifting clocks.
• Vector clocks — detect concurrent writes for explicit resolution.
• CRDTs — a merge that is commutative and idempotent: order and repetition do not matter.

void increment() { counts[me]++; }               // G-Counter: own slot only
void merge(GCounter o) {                          // element-wise max
    for (int i = 0; i < counts.length; i++) counts[i] = Math.max(counts[i], o.counts[i]);
}

Takeaways

1. Distributed: independent nodes, messages only, no shared clock — our store is one.
2. The 8 fallacies are false; a caller cannot tell slow from dead: timeouts + idempotent retries.
3. Order events with a single authority or logical clocks, not wall clocks.
4. CAP: during a partition choose C or A; PACELC adds latency vs consistency. Choose per data domain.
5. FLP: consensus keeps safety and relies on timeouts for progress.
6. Majorities overlap — use 3 or 5 nodes; Raft elects a leader and commits on a majority.
7. Quorums need W + R > N; AP systems converge with LWW, vector clocks or CRDTs.