Overview

It is 09:59:58 and a stadium show is about to go on sale on BoxOffice. At 10:00 two million fans press "Buy" for sixty thousand seats. Behind that button, the first version of BoxOffice does everything in one synchronous request: hold the seats, charge the card, send the e-mail, push a notification to the phone, update the search index and record the sale for analytics. The e-mail provider is a little slow today (800 ms instead of 80 ms), so every purchase now takes a second longer, the order service runs out of threads, and fans who were about to pay see a spinner and then an error. Nothing was broken — the system was simply wired so that every step had to finish before any user got an answer.

This class is about the other wiring: messages. Instead of calling every service and waiting, a service writes a message to a broker and moves on; whoever needs it reads it when it can. Done right, messaging absorbs spikes, decouples teams and keeps slow or failing services out of the critical path. Done carelessly, it loses orders, duplicates e-mails and hides failures in a queue nobody watches. We will learn both sides.

Roadmap

1. This overview — synchronous vs asynchronous, what a broker buys you (and what it costs), load leveling in numbers, the vocabulary, and the four messaging components of the palette.
2. Queues and pub/sub — point-to-point queues, competing consumers, acknowledgements, backpressure, the flash-sale waiting room, and pub/sub fan-out.
3. Kafka and event logs — the log abstraction, partitions, keys and ordering, consumer groups, replication, retention, compaction and partition sizing.
4. Delivery semantics — at-most/at-least/exactly-once, idempotent consumers, ordering, retries and dead letter queues, schema evolution.
5. Patterns — transactional outbox, change data capture, sagas, CQRS and event sourcing, all applied to the BoxOffice purchase.
6. Quiz — check your understanding.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Synchronous vs asynchronous

In a synchronous interaction the caller sends a request and waits for the answer before it continues — an HTTP or gRPC call. In an asynchronous interaction the caller hands a message to an intermediary (a broker) and continues immediately; the receiver processes the message later, possibly much later, possibly on another continent.

Compare the two versions of the BoxOffice purchase. The user clicks "Buy"; the order service must create the order, and five other services need to know about it.

sequenceDiagram
    autonumber
    participant U as Fan
    participant O as order
    participant N as notification
    participant S as search
    participant A as analytics
    U->>+O: POST /orders
    O->>+N: send e-mail + push
    N-->>-O: ok (800 ms today)
    O->>+S: update "seats left"
    S-->>-O: ok
    O->>+A: record sale
    A-->>-O: ok
    O-->>-U: 201 Created (after all of them)

sequenceDiagram
    autonumber
    participant U as Fan
    participant O as order
    participant K as Kafka
    participant N as notification
    participant S as search
    participant A as analytics
    U->>+O: POST /orders
    O->>K: publish OrderConfirmed
    O-->>-U: 201 Created
    K-->>N: OrderConfirmed
    K-->>S: OrderConfirmed
    K-->>A: OrderConfirmed

The difference is not only speed. It shows up in three numbers you already know how to compute (see Estimation):

What a broker buys you

A message broker is a server that receives messages from producers, stores them durably and delivers them to consumers¹. Putting a broker between two services gives four properties that a direct call cannot:

1. Temporal decoupling. Producer and consumer do not need to be up at the same time. If notification is being redeployed, the OrderConfirmed events wait in the broker.
2. Load leveling. The broker is a buffer: producers can write faster than consumers read for a while, and the backlog drains later. A spike becomes a queue instead of an outage.
3. Fan-out. One published event can be delivered to many independent consumers. Adding a new consumer (fraud detection, a loyalty program) does not touch the producer.
4. Independent scaling and failure isolation. Each consumer scales with its own load and fails on its own. A slow analytics pipeline does not slow down e-mails.

Nothing is free. The price of asynchrony:

• Eventual consistency. For a while the search page may still show "120 seats left" when there are 80. The system must be designed (and the UI worded) for that².
• Duplicates and reordering. Most brokers deliver at least once: consumers will see the same message twice and must be idempotent (see Delivery semantics).
• Harder debugging. A purchase is now a chain of events across topics; without distributed tracing and consumer-lag metrics, failures hide in a backlog (see Observability).
• One more critical system. The broker must be replicated, monitored, upgraded and capacity-planned.

Load leveling in numbers

The flash sale is the textbook case for load leveling. Fans' purchase requests arrive much faster than inventory can process seat holds, but only for a few minutes. If order called inventory directly, the excess would turn into timeouts and errors. With a queue in between, the excess turns into backlog — messages waiting — and the only cost is waiting time.

The simulation below uses illustrative numbers: 200 requests/s before 10:00, a spike of 6,000 requests/s for three minutes, and then a crowd that thins out slowly. Workers process 2,000 or 4,000 requests/s.

Three things to read from the chart:

• The backlog grows at the difference of the rates. During the spike it grows by \(6000 - 2000 = 4000\) messages per second with the smaller worker pool: in three minutes, \(4000 \times 180 = 720{,}000\) messages.
• The wait is backlog divided by throughput — Little's Law again (see Queueing): \(720{,}000 / 2000 = 360\) s, six minutes for the last fan in line. Doubling the workers halves the peak and drains it much faster.
• The backlog only drains when arrivals fall below capacity. If the average arrival rate stays above the capacity, no queue saves you: the backlog grows forever. A queue absorbs spikes, not a permanent shortage.

The vocabulary

Messaging has many words for similar things. These are the ones we use in the course.

What goes inside an event

Martin Fowler distinguishes several things people mean by "event-driven"³. Two of them change how you design the payload:

{ "type": "OrderConfirmed", "eventId": "e-9f2c", "orderId": "o-7731", "occurredAt": "2027-03-14T10:00:07Z" }

{
  "type": "OrderConfirmed", "eventId": "e-9f2c", "orderId": "o-7731",
  "occurredAt": "2027-03-14T10:00:07Z",
  "customer": { "id": "c-12", "email": "ana@example.com", "locale": "pt-BR" },
  "event": { "id": "ev-88", "name": "Stadium Show", "startsAt": "2027-05-01T21:00:00-03:00" },
  "seats": ["B-12", "B-13"], "total": { "amount": 780.00, "currency": "BRL" }
}

The other two meanings — event sourcing and CQRS — are architectural patterns covered in Patterns.

The messaging palette

The System Design Playground has four messaging components. They are not four brands; they are four shapes of communication. The component catalog has the details of each one.

The deciding question is what happens to a message after it is read:

• In a queue, reading is taking: the message is gone once acknowledged. Great for distributing work.
• In a log, reading is looking: the message stays, and each reader keeps a bookmark. Great for many independent readers, replay and history.

flowchart TD
    q1{"Does the caller need<br/>the answer to continue?"}
    q1 -->|yes| sync["Synchronous call<br/>(REST / gRPC) + timeout"]
    q1 -->|no| q2{"How many services<br/>must react?"}
    q2 -->|"exactly one<br/>(a job to do)"| queue["queue<br/>competing consumers"]
    q2 -->|"many, each its own copy"| q3{"Need replay, ordering<br/>per key, or high throughput?"}
    q3 -->|no| pubsub["pubsub<br/>topic + subscriptions"]
    q3 -->|yes| log["stream / kafka<br/>partitioned log"]

When not to go asynchronous

The messaging concept map

Every message crosses the same path: a producer hands it to one of the four messaging components, and a consumer processes it later. The map links each palette component (green) to the concepts of this class (blue) and to the classes that continue them:

flowchart LR
    order["app (order)<br/>producer"]:::comp --> queue["queue"]:::comp
    order --> pubsub["pubsub"]:::comp
    order --> log["stream · kafka"]:::comp
    queue --> worker["worker"]:::comp
    pubsub --> cons["notifications · search ·<br/>analytics"]:::comp
    log --> cons
    order -.- c0["sync vs async, outbox,<br/>sagas, CQRS, event sourcing"]:::tech
    queue -.- c1["competing consumers, ack,<br/>backpressure, waiting room"]:::tech
    pubsub -.- c2["fan-out, subscriptions,<br/>push/pull, filters"]:::tech
    log -.- c3["partitions, keys, offsets,<br/>consumer groups, retention"]:::tech
    worker -.- c4["at-least-once, idempotent consumer,<br/>ordering, DLQ, schemas"]:::tech
    c3 -.- d1["consensus, KRaft<br/>(class 3)"]
    c4 -.- d2["retries, freshness SLO<br/>(classes 9 and 10)"]
    c4 -.- d3["consumer lag, tracing<br/>(class 9)"]
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed

Related

• Hands-on: Hands-on 7 — Async: Kafka, transactional outbox in order, idempotent consumer in notification, DLQ and the waiting-room queue.
• Exercise: Event-driven exercise — outbox + idempotent consumer.
• Case studies: Ticketing (BoxOffice flash sale), Notification system, Payment system, Chat.
• Classes: Compute — async work (workers and schedulers), Observability (consumer lag and tracing across topics), Reliability (retries, backoff, idempotency).

What breaks next

The e-mail left the request: in lab 7, an e-mail outage no longer fails a single order, and the waiting room lets in only what inventory can serve. The price is visibility. A purchase is now a request, a database row, an outbox relay, a Kafka record and a consumer, in four processes and at four different times: when a fan says "I paid and got no e-mail", nobody can say where the order stopped. And when the payment provider slows down, the averages on the dashboards stay green while fans fail at checkout. Following one order across services, and measuring what fans actually feel, is class 9 — Observability.