System Architecture · Insper

Messaging and Event-Driven Architecture

Send a message, move on — and handle what that implies

10:00 — two million fans, sixty thousand seats

The purchase holds seats, charges the card, sends the e-mail, updates search and analytics — all in one request.
The e-mail provider slows from 80 ms to 800 ms. What happens to the "Buy" button?

  • Every purchase gets 720 ms slower; order runs out of threads.
  • Fans about to pay see a spinner, then an error.
  • Nothing was broken: every step simply had to finish before anyone got an answer.

Two wirings of the purchase

Sync
fan
→
order
waits →
notification800 ms today
→
search
→
analytics
→ 201
Async
fan
→
order201 after save + publish
publish →
KafkaOrderConfirmed
→
each at its own pace
notification
search
analytics
  • Sync: the fan waits for the sum of the calls, and any failure fails the purchase.
  • Async: the fan waits for one save and one publish; a slow consumer only grows its backlog.

Synchronous chain vs asynchronous events

Synchronous chain Asynchronous events
Latency for the fan sum of every call save + publish one message
Availability of "Buy" 0.9995 × 0.999³ ≈ 99.65% 0.9995 × 0.9995 ≈ 99.90%
Coupling order knows every consumer order knows one topic + schema
A consumer fails the purchase fails its backlog grows
Consistency done when the fan sees 201 consumers catch up (eventual)

If the caller needs the answer to continue, call. If it only needs the work done eventually, send a message.

What a broker buys — and what it costs

Buys

  • Temporal decoupling — consumer may be down
  • Load leveling — a spike becomes backlog
  • Fan-out — new consumers, producer unchanged
  • Independent scaling and failure isolation

Costs

  • Eventual consistency — "120 seats left" for a while
  • Duplicates and reordering — consumers must be idempotent
  • Harder debugging — tracing and lag metrics
  • One more critical system to run

Load leveling in numbers

Spike of 6,000 req/s for 3 min, workers drain 2,000 req/s.

  • Backlog grows at λ − μ = 4,000/s → 4,000 × 180 = 720,000 messages.
  • Last fan waits backlog / μ = 720,000 / 2,000 = 360 s (Little's Law).
  • Double the workers → half the peak and a much faster drain.

A queue absorbs spikes, not a permanent shortage: if average arrivals exceed capacity, the backlog grows forever.

Playground: async-queue with the spike profile — watch queue depth and consumer lag.

The messaging palette

Component Shape Who gets a message After reading
queue point-to-point one consumer of the pool deleted
pubsub fan-out every subscription deleted per subscription
stream partitioned log (managed) every consumer group kept until retention
kafka partitioned, replicated log every consumer group kept (or compacted)
  • Command (GenerateTicketPdf) — do something; one receiver.
  • Event (OrderConfirmed) — a fact, past tense; anyone may react.

What goes inside an event

Event notification

{ type, eventId, orderId, occurredAt }

  • Small and stable.
  • Every consumer calls back GET /orders/o-7731: one extra call per consumer per event, and order must be up.

Event-carried state transfer

{ …, customer, event, seats, total }

  • Consumers never call back: decoupled from order's availability.
  • Bigger payload, a schema harder to change, personal data (the e-mail) copied to every consumer.

Name events in the past tense (OrderConfirmed); SendConfirmationEmail is a command in disguise.

Call, queue, pub/sub or log?

Does the caller need the answer to continue?
yes →
synchronous callwith a timeout: log in, hold seats
no ↓
How many services must react?
exactly one job →
queuecompeting consumers: the PDF ticket
many ↓
Need replay, order per key, high throughput?
no →
pubsube-mail and push
yes →
stream · kafkawarehouse feed, search

In a queue reading is taking; in a log reading is looking, with a bookmark per reader. The page's table "When not to go asynchronous" sorts eight BoxOffice interactions this way.

The life of a queued message

receiveworker 1 gets o-7731
→
hidden 30 svisibility timeout
→
worker 1 crashesbefore the ack
→
visible again
→
worker 2delivery 2
ack →
deleted
  • The broker deletes a message only on the ack: no message is lost when a worker dies.
  • With competing consumers there is no order; first-in, first-out (FIFO) queues keep it per group id.

At-least-once: a crash after the upload means the job runs twice → idempotent consumer (tickets/o-7731.pdf is a deterministic name). A visibility timeout shorter than the job makes every slow job run twice.

Sizing the worker pool

busy workers = λ × S (Little's Law)

  • ~30,000 orders confirmed in 3 min → λ ≈ 167/s; PDF takes S = 0.4 s.
  • Real time: 167 × 0.4 ≈ 67 busy → ~95 workers at 70% utilization.
  • 20 workers (50/s): backlog ≈ 21,000, drains in ~7 min after the sale.

Async work is sized against a delay objective ("tickets by e-mail within 15 min"), not a request latency.

Simulator queue: 95 workers (ρ = 0.70) keep up; with 60 (ρ = 1.11) the backlog only grows; a 3× spike peaks at about 5,300 jobs.

Backpressure: what happens when the queue fills?

Strategy BoxOffice example Cost
Buffer (bounded) PDF queue absorbs the on-sale delay grows
Scale consumers KEDA (Kubernetes Event-driven Autoscaling) on queue length load moves downstream
Reject at the edge 429 / 503 + Retry-After fast, honest error
Slow the producer Kafka producer blocks (max.block.ms) latency moves upstream
Drop / expire stale "seats left" updates, push time to live (TTL) data loss by design
Shed by priority confirmations before marketing separate queues

Alert on the age of the oldest message, not only on depth.

The flash-sale waiting room

2 M fans
join →
waiting roomRedis sorted set: position
admit N/s →
admission tokenshort-lived, signed
→
gatewayrejects calls without it
→
inventory~1,100 holds/s
  • N = measured capacity × a safety factor: 1,100 × 0.5 ≈ 500 fans/s; honest wait = position / N.
  • Sold out → "sold out" from the edge, no database touched. Bots: challenge and per-account limits here.
  • Fairness: a random position for fans who arrived early, first come first served afterwards.
  • Lab 7: the score comes from INCR (the server decides the order), 20 fans/s by default, an admitted key with a 5-minute TTL, checked by POST /orders.

Playground: flash-sale starts without a waiting room — watch app and inventory-db saturate at 10:00, then build the queue.

Publish/subscribe

order
OrderConfirmed →
topicorder-events
one copy each →
sub: notification
notification #1
notification #2
sub: search
search indexer
sub: analytics
analytics loader
  • Inside a subscription consumers compete; across subscriptions everyone sees everything.
  • A loyalty-points consumer = one new subscription, order unchanged and unaware.
  • Pull: consumers ask, natural flow control (workers). Push: the broker calls an HTTPS endpoint (serverless, webhooks).

Redis Pub/Sub and core NATS are ephemeral: disconnected subscribers miss messages. Fine for "typing…", not for orders.

The log: reading does not delete

  • Append-only, ordered records; each gets an offset.
  • Each reader keeps its own position → many independent readers.
  • Replay: move the offset back (re-index after a bug); start a new reader from 0.
  • A topic = N partitions, each an independent log on a different broker.
  • Order is guaranteed only inside a partition.

Keys decide the partition

partition = murmur2(key) mod N

  • Same key → same partition → in order. Key by orderId.
  • Key by show during the stadium on-sale → one partition gets ~60% of the traffic.
  • Choose the finest key that preserves the ordering you need.

Changing N moves keys (hash mod N changes) — size partitions with headroom up front.

Consumer groups and lag

order-events · 6 partitions
P0 · P1
P2 · P3
P4 · P5
one reader per
partition, per group →
group notification · 3 consumers
notification #1P0, P1
notification #2P2, P3
notification #3P4, P5
group analytics
analytics #1all 6, its own offsets
  • Max parallelism of a group = number of partitions: 8 consumers on 6 partitions → 2 idle.
  • Lag = log-end offset − committed offset, per partition: one hot partition = hot key, stuck consumer or poison message.
  • Rebalance when members join, leave or stop polling (max.poll.interval.ms).

Simulator kafka (lab 7's 6 partitions, 3 consumers): key by show → ev-1 hashes to P5, which falls behind. Playground: event-fanout.

Replication and durability

acks Confirmed when… A broker failure can…
0 sent ❌ lose records silently
1 leader wrote it ⚠️ lose acknowledged records
all every in-sync replica (ISR) has it ✅ not lose them, with min ISR ≥ 2
  • Replication factor (RF) 3 across availability zones (AZs) + min.insync.replicas=2 + acks=all: one broker down, writes continue; nothing acknowledged lives on one machine.
  • Lab 7 runs one broker with RF 1: there acks: all waits for exactly one replica — the trap above.
  • Metadata by KRaft controllers (Raft); ZooKeeper removed in Kafka 4.0.

Retention, compaction and why Kafka is fast

0 · ev-8821:00
1 · ev-90Jazz Night
2 · ev-8820:00
3 · ev-91Theater
4 · ev-90null (tombstone)
compact →
2 · ev-8820:00
3 · ev-91Theater
  • Delete by time (retention.ms, 7 days for order-events: replay and the warehouse) or by size.
  • Compact (event-catalog): the latest record of each key survives, with its offset; a tombstone deletes a key.
  • Tiered storage: old segments go to object storage — months of retention, cheap.
  • Fast: sequential appends, the operating system's page cache, zero-copy sendfile, batches compressed with zstd. The price: a few ms of linger.ms.

Sizing partitions

N ≥ max(T / p, T / c), then add headroom

  • Peak: 30,000 orders × 10 events / 180 s ≈ 1,700 events/s — trivial for producers.
  • notification: 50 ms per e-mail → c = 20/s per partition.
  • OrderConfirmed ≈ 167/s → N ≥ 167 / 20 ≈ 9 → create 24.
  • Lab 7: one broker, 6 partitions → at most 6 × 20 = 120/s; a 7th thread idles.

The consumer is usually the bottleneck. Alternatives to more partitions: batching, per-key concurrency.

Where a message gets lost or duplicated

orderproducer
1 · send →
Kafka2 · replicate
3 · deliver →
notificationconsumer
4 · e-mail →
provider
5 · commit the offset ↩
Hop Failure Effect Defence
1 ack lost, producer retries duplicate in the log idempotent producer
1 crash before sending loss (Bruno) transactional outbox
2 leader dies with acks=1 loss of an acked record acks=all, min ISR 2
3–5 crash between e-mail and commit duplicate (Ana, three times) idempotent consumer

Delivery semantics

Guarantee Kafka consumer Crash means
At-most-once commit before processing ❌ message lost
At-least-once commit after processing ⚠️ message repeated
Exactly-once Kafka transactions — inside Kafka only ✅ neither

Effectively-once = at-least-once + idempotent processing. Kafka cannot un-send an e-mail.

Simulator delivery: charge → commit with a crash → 7 charges for 6 orders; commit → charge → the order is lost; the idempotency key fixes the repeat, not the loss.

The idempotent consumer

@KafkaListener(topics = "order-events", groupId = "notification")
@Transactional
public void on(OrderConfirmed event) {
    int inserted = jdbc.update("""
        INSERT INTO processed_events (consumer, event_id)
        VALUES ('notification', ?) ON CONFLICT DO NOTHING
        """, event.eventId());
    if (inserted == 0) return;       // duplicate: already handled
    emails.schedule(event);          // same transaction
}
  • Dedup row and business change in one transaction; offset committed after.
  • External calls carry an idempotency key; the event id is created once, by the producer.
  • Lab 7: processed_events (event_id), but the e-mail is sent inside the transaction — a crash after the send repeats it.

Ordering, retries and dead letters

  • Order breaks with: producer retries (fixed by idempotence), different topics, thread pools, retry topics.
  • Cross-service views: version per entity, apply only newer (WHERE version < :v).
  • Transient error → blocking retries in place (lab 7: 0.5 s, 1 s, 2 s — the partition waits, the order holds) or retry topics (1 s, 4 s, 16 s — the partition flows, later events overtake).
  • Poison message → the dead letter queue (DLQ, in Kafka a dead letter topic) after N attempts — or at once for parse errors — and an alert.

A DLQ nobody reads is a slower way to lose messages. Redrive is safe only because consumers are idempotent.

The simulator: retry in place or in retry topics?

Blocking
o-1005 failsfor its first 2 s
→
waits 0.5 s, 1 s, 2 sin place
→
sent at attempt 4stopped 3.5 s · lag 8 · order kept ✅
Retry topics
o-1005 fails
→
retry-0 · 1 sfails again
→
retry-1 · 4 s
→
sentnever stopped · Cancelled 4.1 s earlier ❌
  • Poison without classification: blocking reaches the DLQ after 3.5 s, retry topics after 21.2 s; forever never lets go — 53 records wait after 30 s.
  • Classify permanent errors (lab 7: IllegalArgumentException): straight to the DLQ, no wait.

Schema evolution

Compatibility Safe changes Deploy first
Backward delete fields; add fields with defaults consumers
Forward add fields; delete fields with defaults producers
Full add/delete optional fields with defaults any order
  • Never: rename, change type, change meaning (reais → cents).
  • Avro/Protobuf + schema registry; CloudEvents envelope; breaking change → order-events.v2.

Dual write → transactional outbox

orderPOST /orders
one transaction →
order DB
INSERT orders
INSERT outbox
relay polls →
SKIP LOCKED
Kafkakey = orderId, acks=all
ack →
mark published
  • Dual write: COMMIT, crash, event never published (Bruno); publish first, rollback, phantom event.
  • A crash after the ack and before the mark republishes: at least once, never lost.
  • Relay by polling (lab 7) or by change data capture (CDC) with Debezium, which reads the write-ahead log. Lab 7 at 200 orders/s: POST /orders p50 2.7 s (sync) → 1.8 ms (outbox).

CDC also keeps search, cache and warehouse in sync from the catalog DB — but raw CDC leaks your table schema; use designed events for other teams.

The purchase saga

Step Service Local transaction Compensation
1 order create order PENDING cancel
2 inventory hold seats (10 min) release
3 payment charge card (key = order id) — pivot refund
4 inventory mark seats SOLD — retried
5 order CONFIRMED, publish OrderConfirmed — retried

A saga is atomic, consistent and durable, but not isolated: show the intermediate state ("seats held — processing payment"). On a payment timeout, reconcile with the provider before compensating.

The saga as a state machine

PENDING
SeatsHeld →
SEATS_HELD
PaymentCaptured →
(the pivot)
PAID
SeatsSold →
retried until success
CONFIRMED
SEATS_HELD
PaymentFailed / timeout →
RELEASING
SeatsReleased →
CANCELLED
  • Before the pivot every step can be compensated; after it, steps are retried, never undone.
  • The order's status is the saga's state: move it with conditional updates (WHERE status = 'PENDING_PAYMENT').
  • The fan gets 202 Accepted and "seats held — processing payment" (a semantic lock), not a spinner.

Choreography or orchestration?

Choreography Orchestration
Coordination services react to events orchestrator sends commands
Flow visibility spread across services one state machine
Risk event spaghetti a "god" orchestrator
Good for 2–3 simple steps 4+ steps, timers, branches

Durable execution engines — Temporal, AWS Step Functions, Azure Durable Functions, Camunda — persist saga state, timers and retries.

Command Query Responsibility Segregation (CQRS) and event sourcing

CQRS

  • Write model for invariants; projections for reads.
  • "My tickets" = one indexed table fed by events.
  • Cost: lag → handle read-your-writes (return the new state, or a version token).

Event sourcing

  • The events are the source of truth; state = replay.
  • Audit, time travel, new read models from history.
  • Cost: events are a contract forever.

Use them where requirements ask (payment ledger, very different read shapes) — not by default.

The messaging concept map

Component Question it answers Key concepts
app (producer) call and wait, or send? sync vs async, eventual consistency
queue who does this job, at what rate? competing consumers, ack, backpressure, waiting room
pubsub how does every service get a copy? subscriptions, push/pull, filters
stream · kafka who needs history, replay, order per key? partitions, keys, groups, lag, RF 3
worker + consumers what if it arrives twice or breaks? at-least-once, idempotency, DLQ, schemas
sql + kafka change data and publish, across services? outbox, CDC, sagas, CQRS, event sourcing

The producer only knows a topic and a schema; every consumer pays for that freedom with idempotency and lag monitoring.

Takeaways

  1. Call when you need the answer; message when the work can happen later.
  2. A queue turns a spike into backlog: size workers with Little's Law and a delay objective.
  3. Every queue needs backpressure and an alert on the age of the oldest message.
  4. Kafka: key = ordering + balance; partitions = max parallelism; RF 3, acks=all, min ISR 2.
  5. Delivery is at-least-once: idempotent consumers, DLQ, compatible schemas.
  6. Outbox for publishing, sagas with compensations across services, CQRS/ES only when needed.

Outline: (1) sync vs async and what a broker buys; (2) queues, backpressure and the waiting room; (3) pub/sub; (4) Kafka: log, partitions, keys, groups, replication, sizing; (5) delivery semantics, idempotency, DLQ, schemas; (6) outbox, CDC, sagas, CQRS, event sourcing.

Let the class answer. The point: the e-mail is not needed to answer the fan, yet it sits in the critical path. That is the whole motivation for messaging.

The page draws both as step-by-step sequence diagrams (the two tabs of "Synchronous vs asynchronous").

Seat hold stays synchronous: the fan needs the answer now and it must be strongly consistent.

Queue: reading is taking. Log: reading is looking, with a bookmark per reader.

The three requests from the hook: warehouse loader (history), search fix (replay), fraud (order per order). A queue can do none of them.

Ana's three e-mails: crash between send and commit, twice. Bruno's missing e-mail: crash between DB commit and publish (producer side) — fixed by the outbox.

The simulator is on the Delivery semantics page, under "Retries and dead letter queues" (retry-topics). Same numbers as its tip.