System Architecture · Insper · Chapter 8

Hands-on 7 — Async Messaging

Outbox, Kafka, idempotent consumers, a dead-letter topic (DLT) and a waiting room for BoxOffice

The story so far

before · lab 6the seat is safe: a lock or a version decides
→
chapter 8 · lab 7outbox, Kafka, idempotent consumer, waiting room
→
next · lab 8chapter 9: traces, dashboard, an alert

Checkout still waits for the e-mail: when the provider is down, nobody can buy. And the crowd is 30× too big: ~33,000 holds/s for the ~1,100 one database takes.

  • The fan's request should end at the commit; the rest can wait in a queue.

Should the fan wait for the e-mail?

order calls notification over HTTP before answering. A real e-mail provider takes tens to hundreds
of milliseconds and sometimes fails. At 10:00 the stadium show goes on sale. What can go wrong?

  • Every order is as slow as the e-mail provider.
  • An e-mail outage becomes an ordering outage.
  • 2 M fans hit every service at once, including the slowest one.

Architecture

Before
fan
→
gateway
→
order
POST, waits →
notificationSMTP, then 201
→
201 to the fanafter the e-mail
After
waiting room
→
orderorders + outbox, 1 tx
→
201 to the fanat the commit
outbox relay
publish →
order-eventsKafka, 6 partitions
group →
notificationdedupe, Mailpit
failures →
order-events-dlt

Step 1 · Kafka in Compose (KRaft, Kafka Raft)

Container Port Role
kafka (apache/kafka:4.3.1) 9094 (host) broker + controller, no ZooKeeper
kafka-init — creates order-events and order-events-dlt, 6 partitions each
kafka-ui 8089 topics, messages, headers, lag
mailpit 8025 / 1025 fake e-mail provider with an inbox
  • Two listeners: containers use kafka:9092, your machine localhost:9094.
  • Auto-create off: the default 1 partition would cap consumers at 1.

Two listeners: the advertised address wins

order (container)
bootstrap →
kafka:9092PLAINTEXT
metadata →
leaders at kafka:9092resolves in the network
CLI on your laptop
bootstrap →
localhost:9094EXTERNAL
metadata →
leaders at localhost:9094resolves on the laptop
one listener for both
→
the laptop is told kafka:9092connects, then times out
  • The first address only finds the broker; the client reconnects to the advertised one.
  • That is why Kafka is the exception to "same port inside and outside".

A log, not a queue

  • Consuming does not delete: each group stores an offset per partition.
  • Lag = log-end offset − committed offset → the consumer's health signal.
  • Same key → same partition → per-key order (the only order Kafka promises).
  • Partitions = ceiling of consumer parallelism in a group: 8 instances on 6 partitions → 2 idle.
  • Groups can rewind (--reset-offsets) → replays → consumers must tolerate duplicates.

Step 2 · The dual-write problem

database, then Kafka
crash between →
order without its eventno e-mail, ever
Kafka, then database
insert fails →
a ghost ordere-mail for nothing
Outbox: order + event row, 1 local tx
crash anywhere →
both rows or neithera relay publishes later
  • PostgreSQL cannot roll back Kafka; Kafka cannot roll back PostgreSQL.
  • The relay publishes unpublished rows and marks them only after acks=all.

The relay

SELECT id, aggregate_id, event_type, payload::text AS payload
FROM outbox WHERE published_at IS NULL
ORDER BY created_at LIMIT 100
FOR UPDATE SKIP LOCKED
  • Every 200 ms: send all, wait for every ack (10 s), then UPDATE ... published_at.
  • Kafka down → each run fails after max.block.ms (5 s), rolls back → no event lost.
  • Crash after the ack → batch republished: at-least-once.
  • SKIP LOCKED → several replicas relay disjoint batches.

Alternative: change data capture (CDC) — Debezium reads the write-ahead log (WAL): lower delay, more infrastructure.

Idempotent API, too

INSERT INTO orders (...) VALUES (...)
ON CONFLICT (hold_id) DO NOTHING
  • One hold → at most one order: hold_id is UNIQUE.
  • A double click or a gateway retry gets the same order (200), and no second event.
  • Kafka stopped, order placed: 201 anyway; a test run published it 71 s later, when Kafka was back.

Idempotency is a design decision at every hop: API, relay, consumer.

Step 3 · Idempotent consumer

  1. Claim: INSERT INTO processed_events (event_id) ... ON CONFLICT DO NOTHING
  2. 0 rows → duplicate → skip.
  3. Effect: send the e-mail.
  4. Commit — if the e-mail throws, the claim rolls back and the retry can try again.

E-mail sent, commit fails → a second e-mail on redelivery. External effects need the provider's
idempotency key (eventId).

Retries and the dead-letter topic

record
→
e-mail sent, or duplicatecommit, next record
record
MailSendException →
retry in place0.5 s, 1 s, 2 s
still failing →
order-events-dltsame partition, exception in headers
record
IllegalArgumentException, JacksonException →
order-events-dlt at oncenot retryable
  • Only the failing partition waits, at most 0.5 + 1 + 2 = 3.5 s; the others keep flowing.
  • Non-blocking alternative: @RetryableTopic (retry topics) — no blocking, but no ordering.

Step 3 · The experiments

  • Tour messages: the new group starts at the earliest offset → 3 × IllegalArgumentException → DLT, without blocking the real events.
  • Replay (--reset-offsets --to-earliest): duplicate event … ignored, Mailpit still has 2 e-mails.
  • NOTIFY_IDEMPOTENT=false: every replay sends every e-mail again.
  • Poison order (poison@example.com): 201 for the fan, the record in order-events-dlt and in dead_letters.

A DLT nobody reads is a silent data loss: alert on its growth, fix the cause, replay — safe because the consumer is idempotent.

Step 4 · Waiting room

2 M fans press "Buy" for 60 k seats; a strongly consistent inventory cannot serve 2 M requests per second. Who gets in first?

2 M fans
join →
Redis sorted setscore = INCR sequence
ZPOPMIN N, every 1 s →
admitted tickettime to live (TTL) 5 min
X-Admission-Ticket →
POST /orders403 without it
  • Honest estimated time of arrival (ETA), by Little's Law: wait = position / rate → 30 000 / 500 = 60 s.

Waiting room pitfalls

  • One admitter: 3 replicas × N/s = 3N/s. Single worker, leader election or a lock.
  • Gate everything expensive (holds and orders) — in production at the edge/gateway.
  • The room must scale too: 2 M fans polling every second = 2 M req/s → poll less, serve the page from a content delivery network (CDN).
  • Bots join many times: rate-limit joins, challenge before joining (web application firewall, WAF).

Step 5 · Measure it

# Experiment What to record
A NOTIFY_MODE sync vs outbox POST /orders p50 / p95 / p99
B stop notification 20 s error rate, lag peak, drain time
C 100 orders/s, concurrency 1 vs 6 lag, drain time, e-mail delay p95
D reset offsets, idempotency off vs on e-mails per order
E waiting room, 20 fans/s, admit 10/s admitted/s, wait p95, ETA error, bots refused

k6 in open model (constant-arrival-rate); lag from kafka-consumer-groups.sh; delays from SQL.

What a test run gave (A and B)

Load Mode p50 p99 throughput
50/s for 60 s sync 58 ms 352 ms 50/s
50/s for 60 s outbox 3.3 ms 1.7 s 49/s
200/s for 30 s sync 2.7 s 9.3 s ❌ 113/s
200/s for 30 s outbox 1.8 ms 95 ms ✅ 200/s
  • sync holds one of the pool's 10 connections per e-mail: at most 10 / 0.055 s ≈ 180 orders/s.
  • B in sync: at least a third of the orders fail (54 % in a test run); in outbox, none.

Predict before you run

Concurrency 1, 50 ms per e-mail, 100 orders/s for 60 s. Lag at the end? Time to drain?

  • Capacity: 1 / 50 ms = 20/s per thread.
  • Backlog grows at 100 − 20 = 80/s → ≈ 4 800 messages.
  • Drains at 20/s → ≈ 240 s. With 6 threads (120/s) → lag ≈ 0. A 7th thread: idle (6 partitions).
  • A test run: about 4 900 and 320 s — each e-mail costs a little more than 50 ms.

Without Docker: the lab's twin

2 · Outbox200 on retry · Kafka down: the row waits · relay crash: 1 event twice
3 · Consumertour → DLT · replay: 2 e-mails stay 2 · dedupe off: 4
5 · Measure (B, C)sync: 1,000 of 3,000 fail · 1 thread: lag 4,800, 240 s · 6 threads: ≤ 17
  • Steps 2, 3 and 5 embed it: same topic, partitions, threads, back-off and e-mail latency as the lab.
  • Play runs a page; the buttons (POST, stop Kafka, stop Mailpit, reset offsets) work at any time.
  • In 5, change NOTIFY_CONCURRENCY: 4 or 5 threads drain as slowly as 3 — thread 1 still reads two partitions.

Try it in the playground

  • event-fanout: one producer, several consumer services, each with its own copy; raise the requests per second until one consumer lags.
  • async-queue with the spike profile: the backlog absorbs the spike while workers stay at a constant rate.

Form a hypothesis in the simulator, then check it with k6 on your own services.

What breaks next

lab 7 · the e-mail leftoutbox p50 1.8 ms vs 2.7 s inline, in a test run
→
the price: visibilitya request, a row, a relay, a record, a consumer: where did the order stop?
→
lab 8 · chapter 9one trace per order, a dashboard, an alert
  • Failures that produce no 5xx are invisible to k6, still the only instrument.

Takeaways

  1. Take slow, fallible work out of the request path: the fan's request ends at the commit.
  2. Outbox: order + event in one local transaction; a relay publishes at-least-once.
  3. Kafka orders per partition; key by aggregate id; partitions cap consumer parallelism.
  4. Idempotent consumers: claim the eventId in the same transaction as the effect.
  5. Classify errors: transient → retry with backoff; permanent → DLT, then fix and replay.
  6. A waiting room turns a spike into a constant rate with an honest ETA (Little's Law).

Chapter 8 of the story of the course. Outline: (1) why the e-mail must leave the request path; (2) Kafka in KRaft mode; (3) the dual-write problem and the outbox; (4) idempotent consumer, retries and the DLT; (5) waiting room; (6) what we measure, and the simulator that replays the lab.

The story of the course (2027.1 overview, "The story of the course"): chapter 8 opens with the synchronous write path and the 30x crowd. Lab 6 left the seat safe (naive double-booked, 9 and 11 seats in runs for that lab; pessimistic and optimistic always 0).

The page draws both architectures as step-by-step diagrams (tabs "Before" and "After").

The tour of the page: order-1 lands twice on partition 4, order-2 on partition 3. Keep the three tour messages: they become the poison messages of step 3.