Skip to content

7. Async Messaging

Goal

Take the slow, fragile work out of BoxOffice's request path. The order service records each order and an OrderPlaced event in the same database transaction (transactional outbox); a relay publishes the event to Kafka; the notification service consumes it idempotently, sends the e-mail and parks poison messages in a dead-letter topic (DLT). Finally, a waiting room queues the flash-sale crowd so that only as many fans enter as the system can serve.

It is 10:00 and the stadium show goes on sale. Every POST /orders that the order service accepts must also produce a confirmation e-mail. If order calls notification over HTTP, the fan waits for the e-mail provider, an e-mail outage becomes an ordering outage, and the 2 million fans who arrive at the same second hit every service at once. This lab replaces that coupling with three ideas from the messaging class: events on a log, at-least-once delivery handled with idempotency, and queues that absorb a spike.

The decisions you implement are argued in the class: synchronous vs asynchronous, the transactional outbox, idempotent consumers, retries and dead letters and the flash-sale waiting room.

The story so far

Chapter 8 of the story of the course. Lab 6 made the seat safe: with a row lock or a version check the double-booking audit ends at 0, where the naive version sold seats twice (9 and 11 in runs for that lab). The write path is still one synchronous chain, and the crowd is still 30 times too big: about 33,000 holds/s arrive for the ~1,100 one database can take (class 8 β€” Messaging). This lab builds the outbox, Kafka, an idempotent notification and a waiting room, and measures POST /orders with and without the e-mail inline, the orders a notification outage fails, the lag and drain time of a burst, the duplicates after a replay and the admission rate under a spike.

Slides

Click the slides and use ← β†’ to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you start

Time about 4 hours (two meetings)
Tools Docker with Compose v2 (docker compose version), k6 1.x (k6 version), curl, jq β€” all from lab 0. The Java services are built inside Docker: no local Java Development Kit (JDK) or Maven needed for this lab
Docker resources at least 4 CPUs and 8 GB of memory (Kafka and two more Java virtual machines, JVMs, join the lab 6 stack)
Ports free on your machine: 8084, 8086, 8089, 8025, 1025, 9094 (plus the lab 6 ones)

Your repository must be in the state lab 6 left it. Every command of this lab runs from the root of your boxoffice repository.

Start state (end of lab 6)
boxoffice/
β”œβ”€β”€ compose.yaml            # db, db-replica, pgbouncer, redis, catalog-1..3, inventory, auth, gateway, edge
β”œβ”€β”€ infra/                  # nginx/, postgres/, pgbouncer/
β”œβ”€β”€ load/                   # k6 scripts of labs 0-6
β”œβ”€β”€ scripts/
└── services/
    β”œβ”€β”€ auth/
    β”œβ”€β”€ catalog/
    β”œβ”€β”€ gateway/
    └── inventory/

Catch up

Skipped or broke lab 6? Download boxoffice-after-lab-6.zip, unzip it, cd boxoffice, and continue from here.

Every file of this lab is shown in full on these pages, with its path as the title, and is also published under https://insper.github.io/platform/2027.1/hands-on/7/code/<path> β€” for example curl -fsSL -o compose.yaml https://insper.github.io/platform/2027.1/hands-on/7/code/compose.yaml.

Architecture

flowchart LR
    fan[Fan] -->|POST /orders| gw[gateway]
    gw --> order[order]
    order -->|INSERT order| db[(PostgreSQL)]
    order -->|"POST /notifications (waits)"| notification[notification]:::problem
    notification -->|SMTP| mail[mailpit]
    notification -->|201 after the e-mail| order
    order -->|201| gw
flowchart LR
    fan[Fan] -->|join| wr[waiting room]:::comp
    wr -->|admitted| gw[gateway]
    gw -->|POST /orders| order[order]
    order -->|"INSERT order + outbox (1 tx)"| db[(PostgreSQL)]
    order -->|201 right away| gw
    relay[outbox relay]:::comp -->|poll unpublished| db
    relay -->|publish OrderPlaced| kafka[(Kafka: order-events)]:::comp
    kafka -->|consumer group notification| notification[notification]
    notification -->|"processed_events (dedupe)"| ndb[(PostgreSQL)]
    notification -->|SMTP| mail[mailpit]
    notification -.->|after 3 failed retries| dlt[(order-events-dlt)]:::problem

The fan's request now ends at the database commit. Everything after the commit β€” publishing, e-mail, retries β€” happens asynchronously and can be slow, fail or be repeated without the fan noticing.

BoxOffice components touched

Palette component In this lab
Kafka single-node Kafka 4 in KRaft mode, topic order-events with 6 partitions
Message Queue the waiting room: a first-in, first-out (FIFO) queue of fans in Redis, drained at a fixed admission rate
App Server order (producer, outbox relay, waiting room)
Notifications notification (idempotent consumer, DLT)
Email Mailpit, a local Simple Mail Transfer Protocol (SMTP) server with a web inbox

What you will measure

Question Metric Tool
How much faster is POST /orders without the e-mail inline? p50 / p95 / p99, error rate k6
What happens to orders while notification is down? error rate, consumer lag k6, kafka-consumer-groups.sh
Does a burst build a backlog, and how fast does it drain? lag over time, drain time, e-mail delay p95 kafka-consumer-groups.sh, SQL
Does redelivery send duplicate e-mails? e-mails per order Mailpit API
Does the waiting room hold the admission rate under a spike? admitted/s, queue position, order p95 k6

Prerequisites

  • Labs 1 to 6 done: your boxoffice repository runs db (PostgreSQL), redis, catalog, inventory and the gateway with docker compose.
  • About 2 GB of extra memory for Docker (Kafka is a JVM).
  • Read the messaging class, especially Kafka, delivery semantics and the outbox pattern.

Steps

  1. Kafka in Compose β€” a KRaft (Kafka Raft) broker, the order-events topic, Kafka UI and Mailpit; a tour of the command-line interface (CLI).
  2. Transactional outbox β€” the order service writes the order and the event in one transaction; a relay publishes it.
  3. Idempotent consumer and DLT β€” the notification service dedupes by eventId, retries with backoff and dead-letters poison messages.
  4. Waiting room β€” a Redis FIFO that admits fans at a fixed rate during the on-sale.
  5. Measure it β€” sync vs async latency, outage behavior, lag and drain time, duplicates, admission rate.

Try it in the playground

Before writing code, see the two shapes this lab builds. Try it: in the first simulator, run the load profile and watch every consumer service receive its own copy of each event (fan-out); then raise the target requests per second (RPS) until one consumer lags behind while the producer stays fast. In the second, run spike and watch the queue backlog grow and drain while the workers stay at a constant rate.

The third one is this lab in numbers (boxoffice-outbox): order calls notification, which sends each e-mail in 50 ms (at most 60 e-mails/s with 3 consumers), and a 30 s burst of 200 orders/s fails about 35% of the orders. Replace the call by order β†’ Kafka (6 partitions) β†’ notification and run with 3 consumers, then 6 and 12: with 6 nothing fails and the lag peaks around 30 s; 12 lag exactly like 6.

The full simulator is in the playground.

Steps 2, 3 and 5 also embed a twin of this lab: the same topic, partitions, consumer threads, back-off and e-mail latency, so you can replay each page (Kafka stopped, a replay with and without dedupe, a poison order, the lag of experiments B and C) before you run it on your own stack.

Measure it

The Measure it step has the k6 scripts and the results table. Fill it in and keep it in your repository: it is part of the deliverable.

Troubleshooting

Symptom Likely cause Fix
kafka stays starting/unhealthy; kafka-init never runs not enough memory for Docker, or port 9094 in use give Docker 8 GB; lsof -i :9094; docker compose logs kafka
order/notification restart with Connection to node -1 … could not be established they started before Kafka was ready (only if you removed depends_on) docker compose up -d order notification again once kafka is healthy
order fails with Unable to obtain connection from database db not healthy yet, or wrong DB_* variables docker compose ps db; the variables are in the order service of compose.yaml
POST /orders answers 403 on-sale in progress the waiting room of that show is open (step 4) use another showId, or curl -X POST localhost:8084/orders/waiting-room/show-42/close
No e-mail in Mailpit notification is down, lagging, or the event went to the DLT docker compose logs notification; sh scripts/kafka-lag.sh; query notification.dead_letters
docker compose exec kafka … fails with service "kafka" is not running the broker was stopped in an experiment docker compose start kafka
Build fails with Could not resolve dependencies no Internet access during docker compose build retry with network; Maven downloads once and caches (--mount=type=cache)
Apple Silicon: no matching manifest for linux/arm64 an image without an arm64 variant all images of this lab are multi-arch; check you typed the tags exactly

Best practices in this lab

  • Write the event in the same transaction as the business change β€” OrderService.java calls Outbox.append (which refuses to run outside a transaction) and OutboxRelay.java publishes after the commit. Why: transactional outbox.
  • Make the consumer idempotent in its own database β€” NotificationService.java inserts the eventId into processed_events (V1__create_notification.sql) in the same transaction as the e-mail row. Why: idempotent consumers.
  • Bound the retries and keep poison messages for inspection β€” KafkaConfig.java (3 retries, then order-events-dlt; parse errors not retried) and DeadLetterListener.java (the dead_letters table). Why: retries and dead letter queues.
  • Create topics explicitly and produce durably β€” compose.yaml (kafka-init, auto-creation off) and services/order/src/main/resources/application.yaml (acks: all, enable.idempotence: true). Why: replication and durability.
  • Admit the crowd at a configured, measured rate, in an order the server decides β€” WaitingRoom.java (INCR sequence, ADMIT_PER_SECOND). Why: the flash-sale waiting room.

Clean up

When you finish (or before lab 8), stop the stack and keep the data, or remove everything:

docker compose stop                 # keep containers and volumes for the next lab
docker compose down -v              # or: remove containers, networks and volumes (all data)

Lab 8 starts from this state; nothing needs to be kept besides your repository.

End state

End state (what this lab added)
boxoffice/
β”œβ”€β”€ compose.yaml            # + kafka, kafka-init, kafka-ui, mailpit, order, notification
β”œβ”€β”€ load/
β”‚   β”œβ”€β”€ orders-load.js
β”‚   └── waiting-room-spike.js
β”œβ”€β”€ scripts/
β”‚   └── kafka-lag.sh
└── services/
    β”œβ”€β”€ notification/       # new: idempotent consumer, DLT, e-mail
    └── order/              # new: orders, outbox relay, waiting room

Commit and tag: git add -A && git commit -m "lab 7: async messaging" && git tag lab7.

Checklist

  • kafka, kafka-init, kafka-ui and mailpit run in compose.yaml; order-events and order-events-dlt have 6 partitions each
  • order writes orders and outbox in one transaction; retrying POST /orders with the same holdId returns the same order and creates no new event
  • the outbox relay publishes with key = orderId and marks rows as published; stopping Kafka loses no event
  • notification sends one e-mail per order even after an offset reset (duplicates are skipped)
  • a poison message lands in order-events-dlt after the retries, and the partition keeps flowing
  • the waiting room admits at the configured rate; POST /orders without an admission ticket is refused during the on-sale
  • results table of the Measure it step filled in results/lab7.md, with a short explanation of each number
  • self-check: docker compose down -v && docker compose up -d --build from a clean clone brings everything up healthy; then pushed to GitHub with the tag lab7

What breaks next

The e-mail left the request: in a test run for this lab, at 200 orders/s POST /orders answered in 1.8 ms at p50 with the outbox against 2.7 s with the e-mail inline, and a 20 s notification outage failed no order. The price is visibility. A purchase is now a request, a database row, an outbox relay, a Kafka record and a consumer, in four processes and at four different times: when a fan says "I paid and got no e-mail", nothing in this stack says where the order stopped, and the lag of a burst (about 4,900 messages with one consumer thread, in another test run) shows only to whoever runs kafka-lag.sh. At the 10:00 on-sale, failures that produce no 5xx are invisible to k6, still the only instrument. Lab 8 β€” Observability follows one order across the services and the Kafka hop in a single trace and turns a promise to fans into a service level objective (SLO) with an alert: class 9 β€” Observability.

Start: Kafka in Compose