Previous slide Next slide Toggle fullscreen Toggle overview view Open presenter view
System Architecture · Insper
Messaging and Event-Driven Architecture
Send a message, move on — and handle what that implies
10:00 — two million fans, sixty thousand seats
The purchase holds seats, charges the card, sends the e-mail , updates search and analytics — all in one request.
The e-mail provider slows from 80 ms to 800 ms. What happens to the "Buy" button?
Every purchase gets 720 ms slower ; order runs out of threads.
Fans about to pay see a spinner, then an error .
Nothing was broken: every step simply had to finish before anyone got an answer .
Two wirings of the purchase
Sync
fan
→
order
waits →
notification800 ms today
→
search
→
analytics
→ 201
Async
fan
→
order201 after save + publish
publish →
KafkaOrderConfirmed
→
each at its own pace
notification
search
analytics
Sync: the fan waits for the sum of the calls, and any failure fails the purchase.
Async: the fan waits for one save and one publish; a slow consumer only grows its backlog.
Synchronous chain vs asynchronous events
Synchronous chain
Asynchronous events
Latency for the fan
sum of every call
save + publish one message
Availability of "Buy"
0.9995 × 0.999³ ≈ 99.65%
0.9995 × 0.9995 ≈ 99.90%
Coupling
order knows every consumer
order knows one topic + schema
A consumer fails
the purchase fails
its backlog grows
Consistency
done when the fan sees 201
consumers catch up (eventual)
If the caller needs the answer to continue, call. If it only needs the work done eventually, send a message.
What a broker buys — and what it costs
Buys
Temporal decoupling — consumer may be down
Load leveling — a spike becomes backlog
Fan-out — new consumers, producer unchanged
Independent scaling and failure isolation
Costs
Eventual consistency — "120 seats left" for a while
Duplicates and reordering — consumers must be idempotent
Harder debugging — tracing and lag metrics
One more critical system to run
Load leveling in numbers
Spike of 6,000 req/s for 3 min , workers drain 2,000 req/s .
Backlog grows at λ − μ = 4,000/s → 4,000 × 180 = 720,000 messages.
Last fan waits backlog / μ = 720,000 / 2,000 = 360 s (Little's Law).
Double the workers → half the peak and a much faster drain.
A queue absorbs spikes , not a permanent shortage: if average arrivals exceed capacity, the backlog grows forever.
Playground: async-queue with the spike profile — watch queue depth and consumer lag.
The messaging palette
Component
Shape
Who gets a message
After reading
queue
point-to-point
one consumer of the pool
deleted
pubsub
fan-out
every subscription
deleted per subscription
stream
partitioned log (managed)
every consumer group
kept until retention
kafka
partitioned, replicated log
every consumer group
kept (or compacted)
Command (GenerateTicketPdf) — do something; one receiver.
Event (OrderConfirmed) — a fact, past tense; anyone may react.
What goes inside an event
Event notification
{ type, eventId, orderId, occurredAt }
Small and stable.
Every consumer calls back GET /orders/o-7731: one extra call per consumer per event, and order must be up.
Event-carried state transfer
{ …, customer, event, seats, total }
Consumers never call back : decoupled from order's availability.
Bigger payload, a schema harder to change, personal data (the e-mail) copied to every consumer.
Name events in the past tense (OrderConfirmed); SendConfirmationEmail is a command in disguise.
Call, queue, pub/sub or log?
Does the caller need the answer to continue?
yes →
synchronous callwith a timeout: log in, hold seats
no ↓
How many services must react?
exactly one job →
queuecompeting consumers: the PDF ticket
many ↓
Need replay, order per key, high throughput?
no →
pubsube-mail and push
yes →
stream · kafkawarehouse feed, search
In a queue reading is taking ; in a log reading is looking , with a bookmark per reader. The page's table "When not to go asynchronous" sorts eight BoxOffice interactions this way.
The life of a queued message
receiveworker 1 gets o-7731
→
hidden 30 svisibility timeout
→
worker 1 crashesbefore the ack
→
visible again
→
worker 2delivery 2
ack →
deleted
The broker deletes a message only on the ack : no message is lost when a worker dies.
With competing consumers there is no order ; first-in, first-out (FIFO) queues keep it per group id.
At-least-once: a crash after the upload means the job runs twice → idempotent consumer (tickets/o-7731.pdf is a deterministic name). A visibility timeout shorter than the job makes every slow job run twice.
Sizing the worker pool
busy workers = λ × S (Little's Law)
~30,000 orders confirmed in 3 min → λ ≈ 167/s; PDF takes S = 0.4 s.
Real time: 167 × 0.4 ≈ 67 busy → ~95 workers at 70% utilization.
20 workers (50/s): backlog ≈ 21,000, drains in ~7 min after the sale.
Async work is sized against a delay objective ("tickets by e-mail within 15 min"), not a request latency.
Simulator queue: 95 workers (ρ = 0.70) keep up; with 60 (ρ = 1.11) the backlog only grows; a 3× spike peaks at about 5,300 jobs.
Backpressure: what happens when the queue fills?
Strategy
BoxOffice example
Cost
Buffer (bounded)
PDF queue absorbs the on-sale
delay grows
Scale consumers
KEDA (Kubernetes Event-driven Autoscaling) on queue length
load moves downstream
Reject at the edge
429 / 503 + Retry-After
fast, honest error
Slow the producer
Kafka producer blocks (max.block.ms)
latency moves upstream
Drop / expire
stale "seats left" updates, push time to live (TTL)
data loss by design
Shed by priority
confirmations before marketing
separate queues
Alert on the age of the oldest message , not only on depth.
The flash-sale waiting room
2 M fans
join →
waiting roomRedis sorted set: position
admit N/s →
admission tokenshort-lived, signed
→
gatewayrejects calls without it
→
inventory~1,100 holds/s
N = measured capacity × a safety factor: 1,100 × 0.5 ≈ 500 fans/s; honest wait = position / N.
Sold out → "sold out " from the edge, no database touched. Bots: challenge and per-account limits here.
Fairness: a random position for fans who arrived early, first come first served afterwards.
Lab 7: the score comes from INCR (the server decides the order), 20 fans/s by default, an admitted key with a 5-minute TTL, checked by POST /orders.
Playground: flash-sale starts without a waiting room — watch app and inventory-db saturate at 10:00, then build the queue.
Publish/subscribe
order
OrderConfirmed →
topicorder-events
one copy each →
sub: notification
notification #1
notification #2
sub: search
search indexer
sub: analytics
analytics loader
Inside a subscription consumers compete ; across subscriptions everyone sees everything.
A loyalty-points consumer = one new subscription , order unchanged and unaware.
Pull : consumers ask, natural flow control (workers). Push : the broker calls an HTTPS endpoint (serverless, webhooks).
Redis Pub/Sub and core NATS are ephemeral : disconnected subscribers miss messages. Fine for "typing…", not for orders.
The log: reading does not delete
Append-only, ordered records; each gets an offset .
Each reader keeps its own position → many independent readers .
Replay : move the offset back (re-index after a bug); start a new reader from 0.
A topic = N partitions , each an independent log on a different broker.
Order is guaranteed only inside a partition .
Keys decide the partition
partition = murmur2(key) mod N
Same key → same partition → in order . Key by orderId.
Key by show during the stadium on-sale → one partition gets ~60% of the traffic.
Choose the finest key that preserves the ordering you need .
Changing N moves keys (hash mod N changes) — size partitions with headroom up front.
Consumer groups and lag
order-events · 6 partitions
P0 · P1
P2 · P3
P4 · P5
one reader per partition, per group →
group notification · 3 consumers
notification #1P0, P1
notification #2P2, P3
notification #3P4, P5
group analytics
analytics #1all 6, its own offsets
Max parallelism of a group = number of partitions : 8 consumers on 6 partitions → 2 idle.
Lag = log-end offset − committed offset, per partition: one hot partition = hot key, stuck consumer or poison message.
Rebalance when members join, leave or stop polling (max.poll.interval.ms).
Simulator kafka (lab 7's 6 partitions, 3 consumers): key by show → ev-1 hashes to P5, which falls behind. Playground: event-fanout.
Replication and durability
acks
Confirmed when…
A broker failure can…
0
sent
lose records silently
1
leader wrote it
lose acknowledged records
all
every in-sync replica (ISR) has it
not lose them, with min ISR ≥ 2
Replication factor (RF) 3 across availability zones (AZs) + min.insync.replicas=2 + acks=all : one broker down, writes continue; nothing acknowledged lives on one machine.
Lab 7 runs one broker with RF 1: there acks: all waits for exactly one replica — the trap above.
Metadata by KRaft controllers (Raft); ZooKeeper removed in Kafka 4.0.
Retention, compaction and why Kafka is fast
0 · ev-8821:00
1 · ev-90Jazz Night
2 · ev-8820:00
3 · ev-91Theater
4 · ev-90null (tombstone)
compact →
2 · ev-8820:00
3 · ev-91Theater
Delete by time (retention.ms, 7 days for order-events: replay and the warehouse) or by size.
Compact (event-catalog): the latest record of each key survives, with its offset; a tombstone deletes a key.
Tiered storage : old segments go to object storage — months of retention, cheap.
Fast: sequential appends, the operating system's page cache, zero-copy sendfile, batches compressed with zstd. The price: a few ms of linger.ms.
Sizing partitions
N ≥ max(T / p, T / c), then add headroom
Peak: 30,000 orders × 10 events / 180 s ≈ 1,700 events/s — trivial for producers.
notification: 50 ms per e-mail → c = 20/s per partition.
OrderConfirmed ≈ 167/s → N ≥ 167 / 20 ≈ 9 → create 24 .
Lab 7: one broker, 6 partitions → at most 6 × 20 = 120/s ; a 7th thread idles.
The consumer is usually the bottleneck. Alternatives to more partitions: batching, per-key concurrency.
Where a message gets lost or duplicated
orderproducer
1 · send →
Kafka2 · replicate
3 · deliver →
notificationconsumer
4 · e-mail →
provider
5 · commit the offset ↩
Hop
Failure
Effect
Defence
1
ack lost, producer retries
duplicate in the log
idempotent producer
1
crash before sending
loss (Bruno)
transactional outbox
2
leader dies with acks=1
loss of an acked record
acks=all, min ISR 2
3–5
crash between e-mail and commit
duplicate (Ana, three times)
idempotent consumer
Delivery semantics
Guarantee
Kafka consumer
Crash means
At-most-once
commit before processing
message lost
At-least-once
commit after processing
message repeated
Exactly-once
Kafka transactions — inside Kafka only
neither
Effectively-once = at-least-once + idempotent processing. Kafka cannot un-send an e-mail.
Simulator delivery: charge → commit with a crash → 7 charges for 6 orders; commit → charge → the order is lost; the idempotency key fixes the repeat, not the loss.
The idempotent consumer
@KafkaListener(topics = "order-events", groupId = "notification")
@Transactional
public void on (OrderConfirmed event) {
int inserted = jdbc.update("""
INSERT INTO processed_events (consumer, event_id)
VALUES ('notification', ?) ON CONFLICT DO NOTHING
""" , event.eventId());
if (inserted == 0 ) return ;
emails.schedule(event);
}
Dedup row and business change in one transaction ; offset committed after.
External calls carry an idempotency key ; the event id is created once , by the producer.
Lab 7: processed_events (event_id), but the e-mail is sent inside the transaction — a crash after the send repeats it.
Ordering, retries and dead letters
Order breaks with: producer retries (fixed by idempotence), different topics, thread pools, retry topics.
Cross-service views: version per entity , apply only newer (WHERE version < :v).
Transient error → blocking retries in place (lab 7: 0.5 s, 1 s, 2 s — the partition waits, the order holds) or retry topics (1 s, 4 s, 16 s — the partition flows, later events overtake).
Poison message → the dead letter queue (DLQ, in Kafka a dead letter topic) after N attempts — or at once for parse errors — and an alert.
A DLQ nobody reads is a slower way to lose messages. Redrive is safe only because consumers are idempotent.
The simulator: retry in place or in retry topics?
Blocking
o-1005 failsfor its first 2 s
→
waits 0.5 s, 1 s, 2 sin place
→
sent at attempt 4stopped 3.5 s · lag 8 · order kept ✅
Retry topics
o-1005 fails
→
retry-0 · 1 sfails again
→
retry-1 · 4 s
→
sentnever stopped · Cancelled 4.1 s earlier ❌
Poison without classification: blocking reaches the DLQ after 3.5 s, retry topics after 21.2 s; forever never lets go — 53 records wait after 30 s.
Classify permanent errors (lab 7: IllegalArgumentException): straight to the DLQ, no wait.
Schema evolution
Compatibility
Safe changes
Deploy first
Backward
delete fields; add fields with defaults
consumers
Forward
add fields; delete fields with defaults
producers
Full
add/delete optional fields with defaults
any order
Never: rename, change type, change meaning (reais → cents).
Avro/Protobuf + schema registry ; CloudEvents envelope; breaking change → order-events.v2.
Dual write → transactional outbox
orderPOST /orders
one transaction →
order DB
INSERT orders
INSERT outbox
relay polls → SKIP LOCKED
Kafkakey = orderId, acks=all
ack →
mark published
Dual write : COMMIT, crash, event never published (Bruno); publish first, rollback, phantom event .
A crash after the ack and before the mark republishes : at least once, never lost.
Relay by polling (lab 7) or by change data capture (CDC) with Debezium, which reads the write-ahead log. Lab 7 at 200 orders/s: POST /orders p50 2.7 s (sync) → 1.8 ms (outbox).
CDC also keeps search, cache and warehouse in sync from the catalog DB — but raw CDC leaks your table schema; use designed events for other teams.
The purchase saga
Step
Service
Local transaction
Compensation
1
order
create order PENDING
cancel
2
inventory
hold seats (10 min)
release
3
payment
charge card (key = order id) — pivot
refund
4
inventory
mark seats SOLD
— retried
5
order
CONFIRMED, publish OrderConfirmed
— retried
A saga is atomic, consistent and durable, but not isolated : show the intermediate state ("seats held — processing payment"). On a payment timeout, reconcile with the provider before compensating.
The saga as a state machine
PENDING
SeatsHeld →
SEATS_HELD
PaymentCaptured → (the pivot)
PAID
SeatsSold → retried until success
CONFIRMED
SEATS_HELD
PaymentFailed / timeout →
RELEASING
SeatsReleased →
CANCELLED
Before the pivot every step can be compensated ; after it, steps are retried , never undone.
The order's status is the saga's state: move it with conditional updates (WHERE status = 'PENDING_PAYMENT').
The fan gets 202 Accepted and "seats held — processing payment" (a semantic lock), not a spinner.
Choreography or orchestration?
Choreography
Orchestration
Coordination
services react to events
orchestrator sends commands
Flow visibility
spread across services
one state machine
Risk
event spaghetti
a "god" orchestrator
Good for
2–3 simple steps
4+ steps, timers, branches
Durable execution engines — Temporal, AWS Step Functions, Azure Durable Functions, Camunda — persist saga state, timers and retries.
Command Query Responsibility Segregation (CQRS) and event sourcing
CQRS
Write model for invariants; projections for reads.
"My tickets" = one indexed table fed by events.
Cost: lag → handle read-your-writes (return the new state, or a version token).
Event sourcing
The events are the source of truth; state = replay.
Audit, time travel, new read models from history.
Cost: events are a contract forever .
Use them where requirements ask (payment ledger, very different read shapes) — not by default.
The messaging concept map
Component
Question it answers
Key concepts
app (producer)
call and wait, or send?
sync vs async, eventual consistency
queue
who does this job , at what rate?
competing consumers, ack, backpressure, waiting room
pubsub
how does every service get a copy?
subscriptions, push/pull, filters
stream · kafka
who needs history, replay, order per key?
partitions, keys, groups, lag, RF 3
worker + consumers
what if it arrives twice or breaks?
at-least-once, idempotency, DLQ, schemas
sql + kafka
change data and publish, across services?
outbox, CDC, sagas, CQRS, event sourcing
The producer only knows a topic and a schema; every consumer pays for that freedom with idempotency and lag monitoring .
Takeaways
Call when you need the answer; message when the work can happen later.
A queue turns a spike into backlog : size workers with Little's Law and a delay objective.
Every queue needs backpressure and an alert on the age of the oldest message.
Kafka: key = ordering + balance; partitions = max parallelism; RF 3, acks=all, min ISR 2.
Delivery is at-least-once : idempotent consumers, DLQ, compatible schemas.
Outbox for publishing, sagas with compensations across services, CQRS/ES only when needed.
Outline: (1) sync vs async and what a broker buys; (2) queues, backpressure and the waiting room;
(3) pub/sub; (4) Kafka: log, partitions, keys, groups, replication, sizing; (5) delivery
semantics, idempotency, DLQ, schemas; (6) outbox, CDC, sagas, CQRS, event sourcing.
Let the class answer. The point: the e-mail is not needed to answer the fan, yet it sits in the
critical path. That is the whole motivation for messaging.
The page draws both as step-by-step sequence diagrams (the two tabs of "Synchronous vs asynchronous").
Seat hold stays synchronous: the fan needs the answer now and it must be strongly consistent.
Queue: reading is taking. Log: reading is looking, with a bookmark per reader.
The three requests from the hook: warehouse loader (history), search fix (replay), fraud (order per
order). A queue can do none of them.
Ana's three e-mails: crash between send and commit, twice. Bruno's missing e-mail: crash between DB
commit and publish (producer side) — fixed by the outbox.
The simulator is on the Delivery semantics page, under "Retries and dead letter queues" (retry-topics). Same numbers as its tip.