7. Async Messaging

It is 10:00 and the stadium show goes on sale. Every POST /orders that the order service accepts must also produce a confirmation e-mail. If order calls notification over HTTP, the fan waits for the e-mail provider, an e-mail outage becomes an ordering outage, and the 2 million fans who arrive at the same second hit every service at once. This lab replaces that coupling with three ideas from the messaging class: events on a log, at-least-once delivery handled with idempotency, and queues that absorb a spike.

It builds chapter 8 of the story of the course. The decisions you implement are argued in the class: synchronous vs asynchronous, the transactional outbox, idempotent consumers, retries and dead letters and the flash-sale waiting room.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you start

Your repository must be in the state lab 6 left it. Every command of this lab runs from the root of your boxoffice repository.

boxoffice/
├── docker-compose.yaml     # db, db-replica, pgbouncer, redis, catalog-1..3, inventory, auth, gateway, edge
├── infra/                  # nginx/, postgres/, pgbouncer/
├── load/                   # k6 scripts of labs 0-6
├── scripts/
└── services/
    ├── auth/
    ├── catalog/
    ├── gateway/
    └── inventory/

Every file of this lab is shown in full on these pages, with its path as the title, and is also published under https://insper.github.io/platform/2027.1/hands-on/7/code/<path> — for example curl -fsSL -o docker-compose.yaml https://insper.github.io/platform/2027.1/hands-on/7/code/docker-compose.yaml.

Architecture

flowchart LR
    fan[Fan] -->|POST /orders| gw[gateway]
    gw --> order[order]
    order -->|INSERT order| db[(PostgreSQL)]
    order -->|"POST /notifications (waits)"| notification[notification]
    notification -->|SMTP| mail[mailpit]
    notification -->|201 after the e-mail| order
    order -->|201| gw

flowchart LR
    fan[Fan] -->|join| wr[waiting room]
    wr -->|admitted| gw[gateway]
    gw -->|POST /orders| order[order]
    order -->|"INSERT order + outbox (1 tx)"| db[(PostgreSQL)]
    order -->|201 right away| gw
    relay[outbox relay] -->|poll unpublished| db
    relay -->|publish OrderPlaced| kafka[(Kafka: order-events)]
    kafka -->|consumer group notification| notification[notification]
    notification -->|"processed_events (dedupe)"| ndb[(PostgreSQL)]
    notification -->|SMTP| mail[mailpit]
    notification -.->|after 3 failed retries| dlt[(order-events-dlt)]

The fan's request now ends at the database commit. Everything after the commit — publishing, e-mail, retries — happens asynchronously and can be slow, fail or be repeated without the fan noticing.

What you will measure

Prerequisites

• Labs 1 to 6 done: your boxoffice repository runs db (PostgreSQL), redis, catalog, inventory and the gateway with docker compose.
• About 2 GB of extra memory for Docker (Kafka is a JVM).
• Read the messaging class, especially Kafka, delivery semantics and the outbox pattern.

Steps

1. Kafka in Compose — a KRaft broker, the order-events topic, Kafka UI and Mailpit; a tour of the CLI.
2. Transactional outbox — the order service writes the order and the event in one transaction; a relay publishes it.
3. Idempotent consumer and DLT — the notification service dedupes by eventId, retries with backoff and dead-letters poison messages.
4. Waiting room — a Redis FIFO that admits fans at a fixed rate during the on-sale.
5. Measure it — sync vs async latency, outage behavior, lag and drain time, duplicates, admission rate.

Try it in the playground

Before writing code, see the two shapes this lab builds. Try it: in the first simulator, run the load profile and watch every consumer service receive its own copy of each event (fan-out); then raise the target RPS until one consumer lags behind while the producer stays fast. In the second, run spike and watch the queue backlog grow and drain while the workers stay at a constant rate.

The full simulator is in the playground.

Measure it

The Measure it step has the k6 scripts and the results table. Fill it in and keep it in your repository: it is part of the deliverable.

Troubleshooting

Clean up

When you finish (or before lab 8), stop the stack and keep the data, or remove everything:

docker compose stop                 # keep containers and volumes for the next lab
docker compose down -v              # or: remove containers, networks and volumes (all data)

Lab 8 starts from this state; nothing needs to be kept besides your repository.

End state

boxoffice/
├── docker-compose.yaml     # + kafka, kafka-init, kafka-ui, mailpit, order, notification
├── load/
│   ├── orders-load.js
│   └── waiting-room-spike.js
├── scripts/
│   └── kafka-lag.sh
└── services/
    ├── notification/       # new: idempotent consumer, DLT, e-mail
    └── order/              # new: orders, outbox relay, waiting room

Commit and tag: git add -A && git commit -m "lab 7: async messaging" && git tag lab7.

Checklist

• kafka, kafka-init, kafka-ui and mailpit run in docker-compose.yaml; order-events and order-events-dlt have 6 partitions each
• order writes orders and outbox in one transaction; retrying POST /orders with the same holdId returns the same order and creates no new event
• the outbox relay publishes with key = orderId and marks rows as published; stopping Kafka loses no event
• notification sends one e-mail per order even after an offset reset (duplicates are skipped)
• a poison message lands in order-events-dlt after the retries, and the partition keeps flowing
• the waiting room admits at the configured rate; POST /orders without an admission ticket is refused during the on-sale
• results table of the Measure it step filled in results/lab7.md, with a short explanation of each number
• self-check: docker compose down -v && docker compose up -d --build from a clean clone brings everything up healthy; then pushed to GitHub with the tag lab7

Related

• Theory: Messaging & event-driven · Queues and pub/sub · Kafka · Patterns: outbox, saga, CQRS · Delivery semantics
• Case study: Ticketing (BoxOffice flash sale)
• Exercise: Event-driven
• Next lab: 8. Observability traces this flow end to end, across the Kafka hop.

Start: Kafka in Compose