4. Event-Driven
A fan pays for two seats, the BoxOffice order service saves the order… and crashes before telling anybody. No ticket e-mail. Or the opposite: the event goes out, the database transaction rolls back, and a fan receives a ticket for an order that does not exist. Or the broker delivers the same event twice and the fan gets two e-mails — and later, two tickets. This is the dual-write problem, and every event-driven system has to solve it. In this exercise you solve it with a transactional outbox, an idempotent consumer and a dead-letter topic, and then you try hard to break it.
Learning objectives
After this exercise you will be able to:
- Explain why "save to the database, then publish to Kafka" loses or invents events, and how the outbox fixes it.
- Implement a transactional outbox with a polling relay and an idempotent consumer with retries and a DLQ.
- Reason about delivery semantics: at-least-once delivery + idempotent processing = effectively-once side effects.
- Verify correctness under failures with a reconciliation query, and measure end-to-end latency and consumer lag.
Context
Class 8 covered queues, pub/sub and the event log, Kafka's partitions and consumer groups, delivery semantics and the outbox/saga patterns1. Lab 7 set up Kafka (KRaft) with BoxOffice's order and notification services. Here you make that flow correct under failure and prove it.
flowchart LR
client([k6]) -->|POST /orders| order[order]
order -->|"1 · one transaction: order + outbox row"| odb[("order_svc<br/>orders · outbox")]
relay[outbox relay] -->|"2 · poll unpublished rows"| odb
relay -->|"3 · publish, key = orderId"| kafka{{"Kafka · order-events"}}
kafka -->|"4 · consume"| notif[notification]
notif -->|"5 · one transaction: processed_events + ticket_emails"| ndb[("notification_svc<br/>processed_events · ticket_emails")]
notif -.->|"poison message after retries"| dlt{{"order-events.DLT"}} Why each piece exists:
| Piece | Failure it handles |
|---|---|
| Outbox row in the same transaction as the order | Crash between "save" and "publish": the event is in the database, it will be published later |
| Relay marks a row as published after the broker ack | Broker down or relay crash: the row is retried (→ duplicates are possible: at-least-once) |
processed_events insert in the same transaction as the side effect | Duplicate delivery: the second insert conflicts and the side effect is skipped |
| Retries with backoff, then DLT | Transient errors are retried; a poison message does not block the partition forever |
Kafka key = orderId | All events of an order go to the same partition, in order |
Task
Implement and verify the flow order confirmed → ticket e-mail for BoxOffice (or the equivalent flow of your own services):
orderwrites the order and an outbox row (event envelope) in one transaction.- A relay publishes unpublished outbox rows to the
order-eventstopic (key =orderId) and marks them after the broker acknowledges. notificationconsumes the events idempotently (deduplication by event id in the same transaction as the side effect), retries transient errors with backoff and sends poison messages to a DLT.- Run a 10-minute load while injecting failures (relay crash, broker outage, consumer crash, duplicate event, poison message) and prove with a reconciliation query that there are zero lost and zero duplicated e-mails.
- Measure end-to-end latency and consumer lag, with and without failures.
Requirements
- (mandatory) Outbox row written in the same transaction as the business change (show the code).
- (mandatory) Idempotent consumer: deduplication by event id in the same transaction as the side effect.
- (mandatory) Reconciliation after the 10-minute run with failures: 0 lost and 0 duplicated side effects (query and its output in the report).
- Event envelope with at least
eventId,type,version,occurredAtand the business key. - Relay publishes with
acks=alland marks rows only after the ack; the outbox returns to 0 unpublished rows after the broker comes back (time measured). - Retries with exponential backoff and a DLT; the poison message is in the DLT and the consumer keeps processing the next messages.
- Failure schedule executed during the load: relay/
ordercrash, broker stopped ≥ 60 s, consumer killed (SIGKILL) mid-processing, one duplicated valid event, one poison message. - End-to-end latency (order confirmed → side effect) p50/p95/p99, and consumer lag over time.
- A discussion of ordering (what the key guarantees and what it does not), and of the real e-mail (an external side effect outside your database).
- AI usage section (see the policy).
Deliverables
In your repository (tag ex4): code of both services, migrations, compose.yaml, k6 script, failure script, the reconciliation and latency queries and their outputs; an event-driven page on your documentation site following the template.
Step by step
1. Kafka (KRaft) and the schemas
Reuse the Kafka of Lab 7. For reference, a single-node KRaft broker with the official image:
services:
kafka:
image: apache/kafka:3.8.0
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:29092
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
ports:
- "29092:29092"
Each service owns its data. To make the reconciliation a single query, this exercise keeps both services in one PostgreSQL instance with two schemas (Compose service postgres, database boxoffice: order_svc and notification_svc), each service using only its own schema. In production they would be separate databases and the reconciliation a small job.
CREATE TABLE outbox (
id UUID PRIMARY KEY,
aggregate_type VARCHAR(50) NOT NULL,
aggregate_id VARCHAR(50) NOT NULL,
event_type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
published_at TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;
CREATE TABLE processed_events (
event_id UUID PRIMARY KEY,
processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE ticket_emails (
id BIGSERIAL PRIMARY KEY,
order_id UUID NOT NULL,
event_id UUID NOT NULL,
recipient VARCHAR(255) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
No unique constraint on ticket_emails.order_id
It is tempting to add one — and in production you might. Here, leave it out: the point is to show that the consumer logic prevents duplicates. A unique constraint would hide a broken consumer.
2. Write the order and the event together
package boxoffice.order;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.time.Instant;
import java.util.Map;
import java.util.UUID;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class OrderService {
private final JdbcTemplate jdbc;
private final ObjectMapper json;
public OrderService(JdbcTemplate jdbc, ObjectMapper json) {
this.jdbc = jdbc;
this.json = json;
}
@Transactional
public UUID confirm(NewOrder cmd) throws Exception {
UUID orderId = UUID.randomUUID();
Instant now = Instant.now();
jdbc.update("""
INSERT INTO orders (id, event_id, seats, customer_email, status, confirmed_at)
VALUES (?, ?, ?, ?, 'CONFIRMED', ?)
""", orderId, cmd.eventId(), cmd.seats(), cmd.customerEmail(), java.sql.Timestamp.from(now));
UUID eventId = UUID.randomUUID();
Map<String, Object> envelope = Map.of(
"eventId", eventId.toString(),
"type", "OrderConfirmed",
"version", 1,
"occurredAt", now.toString(),
"orderId", orderId.toString(),
"customerEmail", cmd.customerEmail(),
"seats", cmd.seats());
jdbc.update("""
INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (?, 'order', ?, 'OrderConfirmed', ?::jsonb)
""", eventId, orderId.toString(), json.writeValueAsString(envelope));
return orderId; // both rows commit together, or neither does
}
}
In this exercise an order is confirmed when it is created (the payment is out of scope); if your service has a separate confirm step, write the outbox row there.
3. The relay
A polling publisher reads unpublished rows, publishes each one and waits for the broker's acknowledgement before marking it. FOR UPDATE SKIP LOCKED lets you run a second relay without both publishing the same row at the same time.
package boxoffice.order;
import java.util.List;
import java.util.Map;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
@Component
public class OutboxRelay {
private final JdbcTemplate jdbc;
private final KafkaTemplate<String, String> kafka;
public OutboxRelay(JdbcTemplate jdbc, KafkaTemplate<String, String> kafka) {
this.jdbc = jdbc;
this.kafka = kafka;
}
@Scheduled(fixedDelay = 200) // requires @EnableScheduling on a configuration class
@Transactional
public void publishBatch() {
List<Map<String, Object>> rows = jdbc.queryForList("""
SELECT id, aggregate_id, payload::text AS payload
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED
""");
for (Map<String, Object> row : rows) {
String key = (String) row.get("aggregate_id");
String payload = (String) row.get("payload");
kafka.send("order-events", key, payload).join(); // wait for the ack (acks=all)
jdbc.update("UPDATE outbox SET published_at = now() WHERE id = ?", row.get("id"));
}
}
}
spring:
kafka:
bootstrap-servers: kafka:9092
producer:
acks: all
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
properties:
enable.idempotence: true
delivery.timeout.ms: 30000
If the relay crashes after send and before the transaction commits, the row is published again on the next poll. That is fine — the consumer is idempotent. Change Data Capture with Debezium (reading the outbox from the PostgreSQL WAL) is an alternative relay; it is a good A-level extension.
4. The idempotent consumer, retries and the DLT
package boxoffice.notification;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.UUID;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class TicketEmailService {
private final JdbcTemplate jdbc;
private final ObjectMapper json;
public TicketEmailService(JdbcTemplate jdbc, ObjectMapper json) {
this.jdbc = jdbc;
this.json = json;
}
@Transactional
public void handle(String message) {
JsonNode event = parse(message);
UUID eventId = UUID.fromString(event.path("eventId").asText());
int inserted = jdbc.update(
"INSERT INTO processed_events (event_id) VALUES (?) ON CONFLICT DO NOTHING", eventId);
if (inserted == 0) {
return; // duplicate delivery: already processed, skip the side effect
}
jdbc.update("INSERT INTO ticket_emails (order_id, event_id, recipient) VALUES (?, ?, ?)",
UUID.fromString(event.path("orderId").asText()), eventId, event.path("customerEmail").asText());
// The real e-mail is sent from ticket_emails by a separate sender,
// using event_id as the idempotency key with the e-mail provider.
}
private JsonNode parse(String message) {
try {
JsonNode event = json.readTree(message);
if (!"OrderConfirmed".equals(event.path("type").asText()) || event.path("eventId").isMissingNode()) {
throw new InvalidEventException("unexpected event: " + message);
}
return event;
} catch (InvalidEventException e) {
throw e;
} catch (Exception e) {
throw new InvalidEventException("malformed event: " + e.getMessage());
}
}
}
package boxoffice.notification;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.TopicPartition;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.config.TopicBuilder;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.kafka.support.ExponentialBackOffWithMaxRetries;
import org.springframework.stereotype.Component;
@Configuration
public class KafkaConfig {
@Bean
NewTopic orderEvents() {
return TopicBuilder.name("order-events").partitions(3).replicas(1).build();
}
@Bean
NewTopic orderEventsDlt() {
return TopicBuilder.name("order-events.DLT").partitions(3).replicas(1).build();
}
// Picked up by Spring Boot's listener container factory.
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<String, String> template) {
DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template,
(record, ex) -> new TopicPartition(record.topic() + ".DLT", record.partition()));
ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(3);
backOff.setInitialInterval(500);
backOff.setMultiplier(2.0);
DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
handler.addNotRetryableExceptions(InvalidEventException.class); // poison: straight to the DLT
return handler;
}
@Component
static class OrderEventsListener {
private final TicketEmailService service;
OrderEventsListener(TicketEmailService service) {
this.service = service;
}
@KafkaListener(topics = "order-events", groupId = "notification")
void onMessage(ConsumerRecord<String, String> record) {
service.handle(record.value());
}
}
}
InvalidEventException is a plain RuntimeException. The consumer commits the offset only after handle returns (Spring Kafka's default BATCH/RECORD ack modes commit after the listener)2, so a crash in the middle means redelivery — which the processed_events table absorbs.
5. Load and failures
import http from 'k6/http';
import { check } from 'k6';
const BASE = __ENV.BASE_URL || 'http://localhost:8081';
export const options = {
scenarios: {
orders: {
executor: 'constant-arrival-rate',
rate: Number(__ENV.RATE || 20), timeUnit: '1s',
duration: __ENV.DURATION || '10m',
preAllocatedVUs: 20, maxVUs: 200,
},
},
};
export default function () {
const body = JSON.stringify({
eventId: 1 + Math.floor(Math.random() * 100),
seats: 1 + Math.floor(Math.random() * 4),
customerEmail: `fan${__VU}-${__ITER}@example.com`,
});
const res = http.post(`${BASE}/orders`, body, { headers: { 'Content-Type': 'application/json' } });
check(res, { 'order accepted': (r) => r.status === 201 });
}
#!/usr/bin/env bash
# Failure schedule, started together with the 10-minute k6 run.
set -x
sleep 90; docker compose kill order; sleep 20; docker compose start order
sleep 90; docker compose stop kafka; sleep 60; docker compose start kafka
sleep 90; docker compose kill notification; sleep 20; docker compose start notification
sleep 30
# a duplicate: publish again an event that was already published
docker compose exec -T postgres psql -U boxoffice -d boxoffice -tAc \
"select aggregate_id || '|' || payload::text from order_svc.outbox where published_at is not null limit 1" \
| docker compose exec -T kafka /opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server kafka:9092 --topic order-events \
--property parse.key=true --property key.separator='|'
# a poison message
echo 'poison|{"type":"OrderConfirmed","orderId":' \
| docker compose exec -T kafka /opt/kafka/bin/kafka-console-producer.sh \
--bootstrap-server kafka:9092 --topic order-events \
--property parse.key=true --property key.separator='|'
Requests to order fail while it is down — that is expected (the k6 errors are not the point here). What must hold is: every order that was accepted gets exactly one ticket e-mail.
6. Verify and measure
-- lost: confirmed orders without exactly one ticket e-mail
SELECT o.id, count(t.id) AS emails
FROM order_svc.orders o
LEFT JOIN notification_svc.ticket_emails t ON t.order_id = o.id
WHERE o.status = 'CONFIRMED'
GROUP BY o.id
HAVING count(t.id) <> 1;
-- duplicated side effects
SELECT order_id, count(*) FROM notification_svc.ticket_emails GROUP BY order_id HAVING count(*) > 1;
-- outbox backlog (must return to 0)
SELECT count(*) AS unpublished FROM order_svc.outbox WHERE published_at IS NULL;
-- end-to-end latency in seconds: order confirmed -> ticket e-mail recorded
SELECT percentile_cont(ARRAY[0.5, 0.95, 0.99]) WITHIN GROUP (
ORDER BY extract(epoch FROM t.created_at - o.confirmed_at)) AS p50_p95_p99
FROM notification_svc.ticket_emails t
JOIN order_svc.orders o ON o.id = t.order_id;
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server kafka:9092 --describe --group notification
docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server kafka:9092 --topic order-events.DLT --from-beginning --timeout-ms 5000
Sample the consumer lag every few seconds during the run (a loop around kafka-consumer-groups.sh, or the Kafka metrics of the observability stack in Lab 8) to plot it over time: you should see it grow while notification is down and drain afterwards.
Templates
# Event-driven — outbox and idempotent consumer
## 1. Flow and design
Diagram; event envelope (fields and an example); topic, partitions, key; consumer group.
## 2. Why it is correct
| Failure | What happens | Why nothing is lost or duplicated |
|-|-|-|
| order/relay crash after the commit | | |
| broker down | | |
| consumer crash mid-processing | | |
| duplicate event | | |
| poison message | | |
## 3. Run with failures
Rate ... orders/s, 10 min. Failure timeline: T+90 s order killed, ...
## 4. Verification
| Check | Result |
|-|-|
| Accepted orders | |
| Lost (query 1) | 0 |
| Duplicated (query 2) | 0 |
| Outbox back to 0 unpublished after broker restart | ... s |
| Poison message in the DLT | yes / no |
## 5. Latency and lag
| | p50 | p95 | p99 |
|-|-|-|-|
| Without failures | | | |
| With failures | | | |
Consumer lag over time (chart) and explanation.
## 6. Discussion
Ordering per key; exactly-once vs effectively-once; sending the real e-mail; outbox vs CDC.
## 7. AI usage
# Orientado a eventos — outbox e consumidor idempotente
## 1. Fluxo e design
Diagrama; envelope do evento (campos e um exemplo); tópico, partições, chave; grupo de consumidores.
## 2. Por que está correto
| Falha | O que acontece | Por que nada se perde nem se duplica |
|-|-|-|
| queda do order/relay depois do commit | | |
| broker fora do ar | | |
| queda do consumidor no meio do processamento | | |
| evento duplicado | | |
| mensagem envenenada | | |
## 3. Execução com falhas
Taxa ... pedidos/s, 10 min. Linha do tempo das falhas: T+90 s order derrubado, ...
## 4. Verificação
| Verificação | Resultado |
|-|-|
| Pedidos aceitos | |
| Perdidos (consulta 1) | 0 |
| Duplicados (consulta 2) | 0 |
| Outbox de volta a 0 não publicados depois do reinÃcio do broker | ... s |
| Mensagem envenenada na DLT | sim / não |
## 5. Latência e atraso
| | p50 | p95 | p99 |
|-|-|-|-|
| Sem falhas | | | |
| Com falhas | | | |
Atraso do consumidor ao longo do tempo (gráfico) e explicação.
## 6. Discussão
Ordem por chave; exactly-once vs effectively-once; envio do e-mail real; outbox vs CDC.
## 7. Uso de IA
Rubric
Levels as defined in How the exercises are graded.
| Criterion | I | D | C | B | A |
|---|---|---|---|---|---|
| Transactional outbox (mandatory) | Absent | Publish outside the transaction (dual write) | Outbox in the same transaction, relay marks after ack | Envelope versioned; relay safe with two instances | CDC (Debezium) relay compared with polling |
| Idempotent consumer (mandatory) | Absent | Dedup outside the side-effect transaction | Dedup by event id in the same transaction | Retention/cleanup of processed_events discussed | Real e-mail sender idempotent with the provider (key = event id) |
| Retries and DLT | — | No DLT, or poison blocks the partition | Backoff + DLT; consumer keeps going | Retryable vs non-retryable errors justified | DLT replay procedure documented and tested |
| Verification under failures (mandatory) | No proof | Some failures not executed | All five failures; 0 lost, 0 duplicated | Timeline of the run with outbox backlog and lag | Additional failure (e.g. network partition) tested |
| Measurements | — | Only averages | p50/p95/p99 end to end + lag | With vs without failures compared and explained | Lag predicted from rates (backlog ÷ drain rate) and checked |
| Discussion | — | Missing | Ordering and external side effect addressed | Trade-offs of exactly-once in Kafka vs idempotency | Clear statement of the guarantees of the whole flow |
FAQ
Kafka has exactly-once semantics. Why do I need all this?
Kafka's transactions give exactly-once inside Kafka (consume → produce → commit offsets). Your side effects — a row in PostgreSQL, an e-mail — live outside Kafka. The robust recipe is at-least-once delivery plus idempotent processing, which makes the effect happen once.
Can I use @TransactionalEventListener instead of an outbox?
It publishes after the commit, but if the process dies between the commit and the publish, the event is lost — the dual-write problem again, only rarer. The outbox survives crashes because the event is in the database.
Will the relay publish events out of order?
With one relay and ORDER BY created_at, events are published in commit order (approximately). Kafka keeps the order per partition, and the key orderId puts all events of an order in the same partition. There is no global order across orders — and BoxOffice does not need one. Explain this in your discussion.
How long do I keep processed_events?
At least as long as a duplicate can arrive (retention of the topic, relay retries). In practice, days; then delete old rows with a scheduled job. Say what you would choose and why.
My consumer lag never goes down after the broker comes back.
Check that the consumer reconnects (logs), that it is not stuck retrying a poison message (is the DLT configured?), and that the topic has as many partitions as you expect. kafka-consumer-groups.sh --describe shows the lag per partition.
The duplicate injection did not create a duplicate e-mail. Did it work?
That is the expected result. Prove that the duplicate was delivered: the consumer's log line for a skipped duplicate, or the count of consumed messages vs rows in processed_events.
-
Chris Richardson, "Pattern: Transactional outbox", microservices.io. ↩
-
Spring for Apache Kafka reference, "Handling exceptions" and "Publishing dead-letter records", docs.spring.io/spring-kafka. ↩