Hands-on 7 — Async Messaging

Should the fan wait for the e-mail?

• Every order is as slow as the e-mail provider.
• An e-mail outage becomes an ordering outage.
• 2 M fans hit every service at once, including the slowest one.

Architecture

Step 1 · Kafka in Compose (KRaft)

• Two listeners: containers use kafka:9092, your machine localhost:9094.
• Auto-create off: the default 1 partition would cap consumers at 1.

A log, not a queue

• Consuming does not delete: each group stores an offset per partition.
• Lag = log-end offset − committed offset → the consumer's health signal.
• Same key → same partition → per-key order (the only order Kafka promises).
• Partitions = ceiling of consumer parallelism in a group.
• Groups can rewind (--reset-offsets) → replays → consumers must tolerate duplicates.

Step 2 · The dual-write problem

• PostgreSQL cannot roll back Kafka; Kafka cannot roll back PostgreSQL.
• Fix: write the event as a row of outbox in the same transaction as the order.
• A relay publishes unpublished rows and marks them only after acks=all.

The relay

SELECT id, aggregate_id, event_type, payload::text AS payload
FROM outbox WHERE published_at IS NULL
ORDER BY created_at LIMIT 100
FOR UPDATE SKIP LOCKED

• Send all, wait for every ack (10 s), then UPDATE ... published_at.
• Kafka down → rollback → rows retried: no event lost.
• Crash after the ack → batch republished: at-least-once.
• SKIP LOCKED → several replicas relay disjoint batches.

Alternative: CDC (Debezium reads the WAL) — lower delay, more infrastructure.

Idempotent API, too

INSERT INTO orders (...) VALUES (...)
ON CONFLICT (hold_id) DO NOTHING

• One hold → at most one order: hold_id is UNIQUE.
• A double click or a gateway retry gets the same order (200), and no second event.

Step 3 · Idempotent consumer

1. Claim: INSERT INTO processed_events (event_id) ... ON CONFLICT DO NOTHING
2. 0 rows → duplicate → skip.
3. Effect: send the e-mail.
4. Commit — if the e-mail throws, the claim rolls back and the retry can try again.

Retries and the dead-letter topic

• DefaultErrorHandler + ExponentialBackOffWithMaxRetries(3): 0.5 s, 1 s, 2 s in place.
• Then DeadLetterPublishingRecoverer → order-events-dlt, same partition, exception in headers.
• Not retryable: JacksonException, IllegalArgumentException → DLT immediately.
• Only the failing partition waits; the others keep flowing.

Non-blocking alternative: @RetryableTopic (retry topics) — no blocking, but no ordering.

Step 4 · Waiting room

• Redis sorted set per show, score = INCR sequence → fair FIFO.
• Every second the admitter pops N fans (ZPOPMIN) and marks them admitted (TTL 5 min).
• POST /orders requires an admitted X-Admission-Ticket while the on-sale is open.
• Honest ETA by Little's Law: wait = position / rate → 45 000 / 500 = 90 s.

Waiting room pitfalls

• One admitter: 3 replicas × N/s = 3N/s. Single worker, leader election or a lock.
• Gate everything expensive (holds and orders) — in production at the edge/gateway.
• The room must scale too: 2 M fans polling every second = 2 M req/s → poll less, CDN for the page.
• Bots join many times: rate-limit joins, challenge before joining (WAF).

Predict before you run

• Capacity: 1 / 50 ms = 20/s per thread.
• Backlog grows at 100 − 20 = 80/s → ≈ 4 800 messages.
• Drains at 20/s → ≈ 240 s. With 6 threads (120/s) → lag ≈ 0. A 7th thread: idle (6 partitions).

Try it in the playground

• event-fanout: one producer, several consumer services, each with its own copy; raise RPS until one consumer lags.
• async-queue with the spike profile: the backlog absorbs the spike while workers stay at a constant rate.

Takeaways

1. Take slow, fallible work out of the request path: the fan's request ends at the commit.
2. Outbox: order + event in one local transaction; a relay publishes at-least-once.
3. Kafka orders per partition; key by aggregate id; partitions cap consumer parallelism.
4. Idempotent consumers: claim the eventId in the same transaction as the effect.
5. Classify errors: transient → retry with backoff; permanent → DLT, then fix and replay.
6. A waiting room turns a spike into a constant rate with an honest ETA (Little's Law).