Replication & Consistency

A fan pays for two tickets at 10:00:41. The confirmation screen says "Order 981 confirmed". She taps My tickets — and the list is empty. In a panic she buys again. Nothing was lost and nothing was buggy in the usual sense: the payment was written to the PostgreSQL primary, while My tickets was served by a read replica that had not yet received the write. BoxOffice had silently chosen a consistency model, and it was the wrong one for that screen.

This page explains how data is copied between nodes (replication) and what a reader is allowed to see while the copies disagree (consistency models). The two are inseparable: the replication design decides which consistency models are cheap, which are expensive and which are impossible.

Why replicate?

Keeping a copy of the same data on several nodes serves three different goals, and it helps to say which one you want:

All three are easy if the data never changes. The difficulty is entirely in handling changes: each write must reach every copy, and while it travels, the copies disagree.

Replication models

There are three families, distinguished by who accepts writes¹.

Single-leader (primary–replica)

One node, the leader (primary), accepts all writes and streams a log of changes to the followers (replicas). Reads can go to the leader or to followers. This is the default in PostgreSQL, MySQL, MongoDB, Redis and Kafka (one leader per partition).

flowchart LR
    w([order: write]) -->|"INSERT order 981"| L[("leader<br/>(primary)")]
    L -->|"WAL stream"| F1[("follower 1")]
    L -->|"WAL stream"| F2[("follower 2")]
    r1([My tickets: read]) -->|"SELECT"| F1
    r2([catalogue: read]) -->|"SELECT"| F2

Multi-leader

Several nodes accept writes — typically one leader per data centre or region — and replicate to each other asynchronously. Writes are fast and local everywhere, but two leaders can accept conflicting writes to the same record, which must be resolved later. Typical uses: multi-region deployments, offline-first apps (each phone is a "leader"), collaborative editing.

Leaderless

Any replica accepts reads and writes. The client (or a coordinator) sends each write to several replicas and each read to several replicas, and quorums decide what counts as success (R + W > N, see Consensus & Quorums). This is the design of Amazon's Dynamo paper², Cassandra and ScyllaDB.

Synchronous or asynchronous?

Independently of who leads, the leader must decide when to tell the client "OK":

sequenceDiagram
    participant C as order
    participant L as leader
    participant F1 as follower 1 (sync)
    participant F2 as follower 2 (async)
    C->>L: INSERT order 981
    L->>F1: replicate
    L->>F2: replicate
    F1-->>L: written to disk
    L-->>C: OK (after 1 sync follower)
    Note over F2: still applying...<br/>a read here is stale
    F2-->>L: written (later)

PostgreSQL exposes this choice in synchronous_commit and synchronous_standby_names (for example ANY 1 (replica_a, replica_b) — semi-synchronous with one of two). In practice, "fully synchronous" is rare: one slow follower would stall every write. Semi-synchronous or quorum replication is the usual compromise.

Replication lag

With asynchronous followers, the delay between a write on the leader and its visibility on a follower is the replication lag. On a healthy system it is usually milliseconds; under heavy write load, during a long transaction or after a network hiccup, it can grow to seconds or minutes. The time between a write and a following read decides whether that read is stale:

With an (assumed) exponential lag, the same screen that is almost never stale on a quiet day is stale for about 69% of fans who open it 300 ms after paying during the on-sale, when the mean lag grows to 800 ms. The bug of the opening story is not rare under load — it is the expected behaviour.

Try it. Load the db-replicas scenario, run the load profile and watch the replica lag metric. Then raise the write rate (or switch to spike) and see the lag — and the fraction of stale reads — grow. Finally route the reads to the primary and compare the primary's CPU and p99.

Open the full playground

Consistency models: a contract

A consistency model is a contract between a data store and its programmers: if you follow these rules, the results of reads will be predictable in this way³. Stronger models are easier to program against and cost more coordination (latency, availability); weaker models are cheaper and push the complexity into the application.

flowchart TD
    LIN["Linearizable<br/>(strong consistency)"] --> SEQ["Sequential"]
    SEQ --> CAU["Causal"]
    CAU --> RYW["Read-your-writes"]
    CAU --> MR["Monotonic reads"]
    CAU --> CP["Consistent prefix"]
    RYW --> EV["Eventual"]
    MR --> EV
    CP --> EV

An arrow from A to B means "A guarantees everything B guarantees, and more". Read-your-writes and monotonic reads are session guarantees: they hold for one client's session, not for everyone; consistent prefix promises that no reader sees writes out of the order in which they happened.

Linearizability (strong consistency)

Linearizability makes a replicated system behave as if there were a single copy of the data, and every operation took effect atomically at one instant between its start and its end⁴. Once any read has returned a new value, every later read (by anyone, on any replica) must return that value or a newer one.

sequenceDiagram
    participant A as Fan A
    participant N1 as replica 1
    participant N2 as replica 2
    participant B as Fan B
    A->>N1: hold seat A-12
    N1-->>A: OK (hold committed)
    Note over A,B: the write has completed
    B->>N2: is seat A-12 free?
    N2-->>B: yes, FREE (stale)
    Note over N2,B: violates linearizability:<br/>B started after A's write finished
    B->>N2: hold seat A-12
    Note over N2: double booking if N2 accepts

Linearizability is what you need when correctness depends on one up-to-date answer:

• uniqueness and inventory — a seat, a username, the last unit of stock;
• leader election and locks — only one node may believe it is the leader;
• cross-channel timing — a fan pays, then the notification service (reading elsewhere) must see the paid order.

It is obtained by routing all reads and writes for a key through one leader that is itself elected by consensus (with care about stale leaders), or by consensus on every operation. Its price: every operation pays at least one round trip to the leader or to a majority, and nodes that cannot reach the leader/majority must refuse — the C-or-A choice of CAP.

Sequential consistency

All nodes see all operations in the same order, and each node's operations appear in its program order — but that order need not respect real time. A read may return a value that is "old" in wall-clock terms, as long as everyone agrees on the sequence. It is mainly a hardware-memory and theory concept; in system design you rarely choose it explicitly.

Causal consistency

Operations that are causally related — one could have influenced the other — are seen by everyone in that order; concurrent operations may be seen in different orders by different nodes.

The classic example is a conversation: if Ana posts "Is the 10:00 sale delayed?" and the organiser replies "No, it starts on time", nobody may see the reply without the question. In BoxOffice: a SeatReleased event caused by a HoldExpired event must not be applied before it.

Causal consistency is interesting because it is the strongest model that remains available during a network partition⁵: nodes can keep accepting reads and writes on both sides and still respect causality. It is tracked with logical clocks — see Time & Clocks.

Session guarantees

Most "my data disappeared" bugs are violations of one of these four guarantees, defined for a single client session (Terry et al., Bayou project)⁶:

sequenceDiagram
    participant F as Fan's app
    participant R1 as replica 1 (lag 10 ms)
    participant R2 as replica 2 (lag 2 s)
    F->>R1: GET /events/42/availability
    R1-->>F: 95 seats left
    F->>R2: refresh (load balancer picks R2)
    R2-->>F: 340 seats left
    Note over F: the count went back in time:<br/>monotonic reads violated

How to provide them on top of asynchronous replicas:

• Read from the leader after a write: for a short window (e.g. 1–5 s) after the fan writes, route her reads to the primary — or always read data the user owns (My tickets) from the primary.
• Version tokens: the write returns the leader's log position (in PostgreSQL, the WAL LSN — Log Sequence Number); the client sends it with the next read, and a replica only answers once it has applied at least that position (otherwise it waits or forwards to the leader). Many databases expose this as causal sessions or read-after-write tokens (MongoDB causal consistency sessions, Cosmos DB session consistency).
• Sticky replica: pin each session to one replica (hash of the session id) — gives monotonic reads, not read-your-writes.

/**
 * Two connection pools — primary and replica — behind one DataSource.
 * A transaction marked @Transactional(readOnly = true) gets a replica connection;
 * every other transaction gets a primary connection. The choice happens when the
 * transaction runs its first statement (that is what "lazy" means here).
 */
@Configuration
public class DataSourceConfig {

    @Bean
    @ConfigurationProperties("inventory.datasource.primary")
    HikariDataSource primaryDataSource() {
        return new HikariDataSource();
    }

    @Bean
    @ConfigurationProperties("inventory.datasource.replica")
    HikariDataSource replicaDataSource() {
        return new HikariDataSource();
    }

    @Bean
    @Primary
    DataSource dataSource(HikariDataSource primaryDataSource, HikariDataSource replicaDataSource) {
        var routing = new LazyConnectionDataSourceProxy(primaryDataSource);
        routing.setReadOnlyDataSource(replicaDataSource);     // Spring Framework 6.2+
        return routing;
    }

Eventual consistency

Eventual consistency only promises that if writes stop, all replicas will eventually converge to the same value. It says nothing about when, and nothing about what you read meanwhile. It is the guarantee of DNS, of most caches and CDNs, of asynchronous replicas, and of leaderless stores at low quorum settings.

It is perfectly fine for many domains — the event description, the poster image, the "312 people are looking at this event" counter. The hard part is what happens when replicas accepted concurrent writes to the same item and must converge:

A CRDT (Conflict-free Replicated Data Type)⁷ is designed so that replicas can exchange state in any order, any number of times, and still end up equal. The simplest is the grow-only counter: each replica increments only its own slot, and merge takes the element-wise maximum.

// Grow-only counter CRDT: e.g. "fans interested in event 42", counted in 3 regions
class GCounter {
    private final long[] counts;   // one slot per replica
    private final int me;          // this replica's index

    GCounter(int replicas, int me) { this.counts = new long[replicas]; this.me = me; }

    void increment() { counts[me]++; }

    long value() { return java.util.Arrays.stream(counts).sum(); }

    // merge = element-wise max: commutative, associative, idempotent
    void merge(GCounter other) {
        for (int i = 0; i < counts.length; i++)
            counts[i] = Math.max(counts[i], other.counts[i]);
    }
}

If São Paulo counted [3, 0, 0] and Lisbon [0, 2, 0], both merge to [3, 2, 0] and report 5 — no increment lost, no coordination needed. A single shared number with LWW would report 3 or 2.

The spectrum at a glance

How it connects

flowchart LR
    W["write on the leader"] --> REPL{"replication<br/>sync or async?"}
    REPL -->|"sync / quorum"| LIN["linearizable reads"]
    LIN --> COST["pays latency and availability<br/>→ CAP & PACELC"]
    REPL -->|"async: lag"| STALE["stale reads"]
    STALE --> SG["session guarantees<br/>read-your-writes, monotonic reads"]
    W2["writes on several leaders"] --> CONF["concurrent writes"]
    CONF --> RES["LWW · version vectors · CRDTs<br/>→ Time & Clocks"]
    LIN --> CONS["leader chosen by consensus<br/>→ Consensus & Quorums"]

Palette components that are replicas (and therefore have a consistency model): sql (primary + read replicas, scenario db-replicas) · nosql (leaderless, scenario sharding) · cache (scenario read-heavy-cache) · cdn (scenario cdn-static) · stream (replicated partitions). Builds on: failure models. Continues in: CAP & PACELC, Databases — replication, Caching — invalidation.

Key takeaways