Distributed Transactions

The fan has held seat A-12 and taps Pay R$ 450. Three things must now happen together: inventory turns the seat from HELD into SOLD, payment charges the card through an external provider, and order records a confirmed order. Inside one PostgreSQL database this would be a single transaction — BEGIN … COMMIT, all or nothing. But the three pieces live in three services with three databases (and one of them is someone else's company). What happens when the charge succeeds and the inventory database is down? Or the seat is marked SOLD and the charge is declined?

A distributed transaction is a unit of work that changes data on several nodes and must be atomic: either every change happens, or none does. This page shows the classic protocol that achieves it — two-phase commit (2PC) —, why it is rarely used across microservices, and what replaces it in practice: sagas, the reservation (try–confirm–cancel) pattern and the transactional outbox.

Atomicity across nodes: the problem

On one node, atomicity is provided by the database's log: a transaction's changes become durable and visible at a single point, the commit record. Across nodes there is no single log, and each participant can fail independently — so a naïve "commit everywhere, one after the other" leaves a window where some participants committed and others did not:

sequenceDiagram
    participant O as order (checkout)
    participant I as inventory DB
    participant P as payment provider
    O->>I: UPDATE seat A-12 to SOLD, then COMMIT
    I-->>O: committed
    O->>P: charge R$ 450
    P-->>O: declined (card limit)
    Note over O,P: the seat is SOLD but nobody paid:<br/>no single place can roll both back

Two families of solutions exist:

Two-phase commit (2PC)

2PC introduces a coordinator (often the service or database node that started the transaction) and splits the commit into two phases¹:

1. Prepare (voting): the coordinator asks every participant "can you commit?". A participant that answers yes must first write all the changes and a prepared record durably to disk and keep the locks: from that moment it has promised to commit if asked — it can no longer abort on its own.
2. Commit (decision): if all voted yes, the coordinator writes commit to its own log and tells everyone to commit; if anyone voted no (or timed out), it decides abort.

sequenceDiagram
    participant C as coordinator
    participant I as inventory DB
    participant R as order DB
    C->>I: PREPARE (seat A-12 → SOLD)
    C->>R: PREPARE (order 981 → CONFIRMED)
    I-->>C: YES (changes + prepared record on disk, locks held)
    R-->>C: YES
    Note over C: all YES: write COMMIT to the coordinator log
    C->>I: COMMIT
    C->>R: COMMIT
    I-->>C: done (locks released)
    R-->>C: done

PostgreSQL exposes the participant side as PREPARE TRANSACTION 'id' followed later by COMMIT PREPARED 'id' or ROLLBACK PREPARED 'id'; the X/Open XA standard defines the same protocol for databases and message brokers coordinated by a transaction manager (as in Jakarta EE).

The blocking problem

The weak point is the moment after a participant voted yes and before it hears the decision. If the coordinator crashes there, the participant is in doubt: it cannot commit (maybe someone voted no) and cannot abort (maybe the coordinator decided commit and others already committed). It must wait, holding its locks, until the coordinator recovers:

sequenceDiagram
    participant C as coordinator
    participant I as inventory DB
    participant R as order DB
    C->>I: PREPARE
    C->>R: PREPARE
    I-->>C: YES (locks on seat A-12 held)
    R-->>C: YES
    Note over C: crashes before sending the decision
    Note over I: in doubt: cannot commit, cannot abort
    Note over I,R: seat A-12 stays locked until<br/>the coordinator comes back

You can cause it yourself in the simulator under Sagas: choose 2PC and coordinator crash.

During the 10:00 on-sale, a locked seat row blocks every other fan trying to hold it, and a crashed coordinator can freeze a whole block of seats. Three-phase commit (3PC) tries to remove the blocking but assumes bounded network delays, which real networks do not guarantee; production systems instead make the coordinator itself fault-tolerant by replicating its decision with consensus — exactly what Google Spanner (2PC over Paxos groups) and CockroachDB (over Raft) do².

What 2PC costs

• Latency: two round trips to every participant, plus forced disk writes (fsync) at each prepare and at the decision. Locks are held for the whole duration — the slowest participant decides how long.
• Availability: the coordinator and every participant must be up at the same time. Availabilities multiply, like services in series:

• Coupling: every participant must speak the same protocol and expose "prepared" state. An external payment provider will never join your 2PC: its API charges or declines, with no "prepare and wait for my decision" step.

Sagas

A saga (Garcia-Molina and Salem, 1987)³ splits a long transaction into a sequence of local transactions \(T_1, T_2, …, T_n\), each committed in its own service's database. If step \(T_k\) fails, the saga runs compensating transactions \(C_{k-1}, …, C_1\) that semantically undo the steps already done. There is no global lock and no global commit point.

The BoxOffice checkout as a saga:

Two ways to coordinate the steps:

Sagas give up isolation

Between steps, other transactions see intermediate states: an order PENDING, a seat RESERVED_FOR_PAYMENT. Sagas are ACD without the I. Design countermeasures explicitly⁴:

• Semantic locks: the intermediate state is the lock — while a seat is RESERVED_FOR_PAYMENT, no other fan can hold it, and the hold expires after 10 minutes if the saga never finishes.
• Commutative and idempotent steps: every step and compensation may be retried (at-least-once), so each one needs an idempotency key (Idempotency).
• Order the steps by risk: do the cheap, invisible, easily reversed steps first, and the steps that are hard or impossible to compensate (a charge, an e-mail) last, so that a failure late in the saga has little to undo. A refund is a compensation the fan sees; releasing a seat is invisible.
• Pivot step: the step after which the saga can only go forward (here, the successful charge) — after it, the remaining steps must be retried until they succeed, never compensated.

/** No @Transactional here: never hold a DB connection during a remote call. */
public OrderView pay(UUID id) {
    OrderView order = find(id).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND));
    if (!"PENDING_PAYMENT".equals(order.status())) {
        return order; // already paid or cancelled: paying twice is a no-op
    }
    PaymentResponse payment = resilience
            ? payments.charge(order.id(), order.amount())
            : payments.chargeUnprotected(order.id(), order.amount());
    String next = "AUTHORIZED".equals(payment.status()) ? "CONFIRMED" : "CANCELLED";

    tx.executeWithoutResult(status -> {
        int changed = jdbc.sql("UPDATE orders SET status = ? WHERE id = ? AND status = 'PENDING_PAYMENT'")
                .params(next, id)
                .update();
        if (changed == 1) {
            String type = "CONFIRMED".equals(next) ? "OrderConfirmed" : "OrderCancelled";
            OrderEvent event = new OrderEvent(UUID.randomUUID(), type, order.id(), order.showId(),
                    order.holdId(), order.customerEmail(), order.amount(), Instant.now());
            outbox.append(event.eventId(), "order", order.id(), type, event);
        }
    });
    return find(id).orElseThrow();
}

Reservation: try, confirm, cancel

The TCC pattern (try–confirm–cancel) makes each participant offer a tentative operation with a timeout. BoxOffice already works this way: the hold is the try (the seat is reserved for 10 minutes), the sale is the confirm, and expiry or release is the cancel. Payment providers offer the same shape: authorise (reserve funds) → capture (take them) or void (release them). Because every tentative state expires by itself, a crash anywhere leaves the system converging to a safe state instead of blocking.

The dual-write problem and the outbox

Every saga step does two things: change its own database and tell the next step (send an event or a command). Doing them as two separate writes is the dual-write problem — if the service crashes between them, the database says "seat reserved" but no event is ever sent, and the saga stalls:

sequenceDiagram
    participant I as inventory
    participant DB as inventory DB
    participant K as Kafka
    I->>DB: UPDATE seat + INSERT INTO outbox (SeatReserved), one COMMIT
    Note over I,DB: one local transaction: both or neither
    DB-->>I: committed
    Note over DB,K: a relay (poller or CDC such as Debezium)<br/>reads the outbox table
    DB->>K: publish SeatReserved
    K-->>DB: ack: mark outbox row as sent

The transactional outbox writes the event into an outbox table in the same local transaction as the business change; a separate relay publishes outbox rows to the broker and marks them sent. Publishing is at-least-once, so consumers must be idempotent. The details (polling vs change data capture, ordering, cleanup) are in Messaging — patterns and Hands-on 7.

@Transactional
public PlaceResult place(OrderRequest req) {
    int inserted = jdbc.sql("""
            INSERT INTO orders (id, show_id, hold_id, customer_email, amount, status)
            VALUES (?, ?, ?, ?, ?, 'PENDING_PAYMENT')
            ON CONFLICT (hold_id) DO NOTHING
            """)
            .params(UUID.randomUUID(), req.showId(), req.holdId(), req.customerEmail(), req.amount())
            .update();

    OrderView order = jdbc.sql(SELECT + " WHERE hold_id = ?")
            .param(req.holdId())
            .query(OrderView.class)
            .single();

    if (inserted == 0) {
        // same hold already ordered: idempotent replay, no second event
        return new PlaceResult(order, false);
    }

    OrderEvent event = OrderEvent.placed(order);
    if (props.notifyMode() == NotifyMode.SYNC) {
        notifications.send(event); // BEFORE: the fan waits for the e-mail
    } else {
        outbox.append(event.eventId(), "order", order.id(), event.eventType(), event);
    }
    return new PlaceResult(order, true);
}

Choosing

How it connects

flowchart LR
    IDEM["idempotency keys<br/>(overview)"] --> SAGA["saga steps<br/>retried safely"]
    OUT["transactional outbox"] --> SAGA
    SAGA --> ORCH["orchestration<br/>(workflow engine)"]
    SAGA --> CHOR["choreography<br/>(events on Kafka)"]
    TCC["reservations: hold, authorise"] --> SAGA
    TPC["2PC"] --> BLOCK["blocks if the<br/>coordinator fails"]
    BLOCK --> CONS["replicate the coordinator<br/>→ Consensus"]
    CONS --> NEWSQL["inside Spanner,<br/>CockroachDB, Kafka txns"]

Palette components involved: stream / kafka (saga events, outbox relay) · queue (commands to participants) · payment (authorise / capture / void) · sql (local transactions + outbox table) · worker (orchestrator, relay). Try the payment-system and flash-sale scenarios in the playground. Continues in: Messaging — patterns, Compute — integrations, Case study: ticketing.

Key takeaways