Event-Driven Architecture

Synchronous REST works well when the caller needs an immediate answer. But when an order is placed, the system might need to notify 5 downstream services — email, inventory, fraud detection, shipping, analytics. Calling them all synchronously before returning to the user creates a latency cliff and a coupling explosion. Event-driven architecture solves this with a fundamentally different contract: publish an event, and let interested parties react asynchronously.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Why Event-Driven?

Two delivery models side by side: the synchronous chain forces the producer to wait for every downstream service before it can respond; the async model returns immediately after publishing.

sequenceDiagram
    autonumber
    participant Client
    participant OrderService
    participant EmailService
    participant InventoryService
    participant FraudService

    Client->>+OrderService: POST /orders
    OrderService->>+EmailService: sendConfirmation()
    EmailService-->>-OrderService: ok
    OrderService->>+InventoryService: reserveStock()
    InventoryService-->>-OrderService: ok
    OrderService->>+FraudService: checkFraud()
    FraudService-->>-OrderService: ok
    OrderService-->>-Client: 201 Created

sequenceDiagram
    participant Client
    participant OrderService
    participant Kafka
    participant EmailService
    participant InventoryService
    participant FraudService

    Client->>+OrderService: POST /orders
    OrderService->>Kafka: publish OrderCreated
    OrderService-->>-Client: 202 Accepted
    Kafka-->>EmailService: OrderCreated (async)
    Kafka-->>InventoryService: OrderCreated (async)
    Kafka-->>FraudService: OrderCreated (async)

The simulation below illustrates what happens to queue depth when the consumer cannot keep up with the producer — and what happens when it catches up.

Core Concepts

Kafka Architecture

Apache Kafka¹ is a distributed, append-only log: producers append messages to the end, consumers read from any position they choose, and messages stay on disk after being read (until the retention policy deletes them). This is the main difference from a classic message queue such as RabbitMQ, where a message is removed once a consumer acknowledges it.

Kafka organises messages into topics, each split into ordered partitions. Multiple consumer groups can read the same topic independently — each group maintains its own offset pointer per partition.

flowchart LR
    subgraph broker ["Kafka Broker"]
        subgraph topic ["Topic: order-events"]
            P0["Partition 0\n0, 1, 2, 3, →"]
            P1["Partition 1\n0, 1, 2, →"]
            P2["Partition 2\n0, 1, →"]
        end
    end
    subgraph ng ["notification-group"]
        N0[Consumer A]
        N1[Consumer B]
        N2[Consumer C]
    end
    subgraph ag ["analytics-group"]
        A0[Consumer X]
    end
    P0 --> N0
    P1 --> N1
    P2 --> N2
    P0 --> A0
    P1 --> A0
    P2 --> A0

Each consumer group tracks its own offsets, so notification-group and analytics-group progress through the log independently. Adding a new consumer group never affects existing groups.

Retention policies control how long Kafka keeps messages:

retention.ms=604800000 (7 days) is Kafka's default; retention.bytes defaults to -1 (no size limit).

Keys, partitions and ordering

When a producer sends a message with a key, Kafka's default partitioner hashes the key to choose the partition: the same key always goes to the same partition (as long as the number of partitions does not change). Messages without a key are spread across partitions.

Since order is only guaranteed inside a partition, the key is how you choose what must stay in order. In Hands-on 9 the order service sends kafkaTemplate.send("order-events", order.getId().toString(), event) — the orderId is the key, so OrderCreated, OrderPaid and OrderShipped of the same order are always read in sequence, while different orders are processed in parallel.

flowchart LR
    p["order-service<br/>(producer)"] -->|"key=order-7"| h{"hash(key) % 3"}
    h -->|"order-7 → P1"| P1["Partition 1"]
    h -.->|"other keys"| P0["Partition 0"]
    h -.->|"other keys"| P2["Partition 2"]
    P1 --> c["consumer B<br/>reads order-7 events in order"]

Durability: replication and acks

In production each partition is replicated on several brokers (replication factor, typically 3). One replica is the leader and receives all writes; the others are followers that copy it. The replicas that are up to date form the ISR (in-sync replicas). If the leader's broker dies, an in-sync follower is elected leader.

The producer setting acks chooses how much of this the producer waits for:

Hands-on 9 uses acks: all with enable.idempotence: true; the idempotent producer attaches a sequence number to each message so that the producer's own retries do not create duplicates in the log. (Since Kafka 3.0 both are already the producer defaults.)

Messaging Patterns

Pub/Sub (Fan-out)

One published event is delivered to every interested consumer group. Each group receives a full, independent copy of the message. Because groups track their own offsets, a slow analytics pipeline does not affect the notification pipeline.

Use cases: email notifications, audit logs, analytics pipelines, cache invalidation.

Competing Consumers

Multiple consumers within the same group share partitions — each partition is owned by exactly one consumer at a time, so each message is processed by exactly one consumer. This pattern enables horizontal scaling of processing without duplicated work.

Use cases: order processing, payment handling, image resizing jobs.

The maximum parallelism of a group is the number of partitions: with 3 partitions, a fourth consumer in the same group sits idle. When a consumer joins or leaves, Kafka rebalances the group and reassigns partitions.

Event Sourcing

Instead of storing only the current state, the system stores the full history of events³. Current state is derived by replaying all events from the beginning (or from a snapshot).

sequenceDiagram
    actor User
    User->>+CommandHandler: PlaceOrder
    CommandHandler->>+EventStore: append OrderCreated
    EventStore-->>-CommandHandler: ack
    CommandHandler-->>-User: 202 Accepted
    EventStore-->>Projection: OrderCreated event
    Projection->>ReadModel: update order view

This means the write model is the event log itself. Projections (read models) are rebuilt from events at any time, enabling time travel debugging and zero-downtime schema migration.

CQRS (Command Query Responsibility Segregation)

Separate the write side (commands, events, event store) from the read side (queries, projections, read model)⁴. The write side is optimised for consistency and auditability; the read side is optimised for query performance.

flowchart LR
    client --> |"POST /orders (Command)"| cmd["Command Handler"]
    cmd --> |"OrderCreated"| bus["Event Bus (Kafka)"]
    bus --> |"project"| view["Read Model\n(optimised for queries)"]
    client --> |"GET /orders (Query)"| view
    cmd --> |"append"| store["Event Store"]

CQRS is commonly paired with Event Sourcing because the event log is the natural write model. However, CQRS can also be applied without event sourcing — using separate read/write databases kept in sync via change-data-capture (CDC: a tool such as Debezium reads the database's transaction log and publishes every change as an event).

Delivery Guarantees and Idempotent Consumers

A consumer reads a message, processes it (e.g. sends an e-mail), and commits its offset to tell Kafka "I am done up to here". A crash can happen between any two of these steps, and when you commit decides what a crash costs:

sequenceDiagram
    participant K as Kafka
    participant N as notification-service
    participant M as Mail server
    K->>N: OrderCreated (offset 42)
    N->>M: send confirmation e-mail
    Note over N: crashes before commit
    N->>K: (restart) fetch from last committed offset
    K->>N: OrderCreated (offset 42) again
    N->>N: already sent for order 7? yes → skip
    N->>K: commit offset 43

Exactly-once inside Kafka does not extend to side effects outside it: Kafka cannot "un-send" an e-mail or roll back a row in Postgres. In practice, systems use at-least-once delivery + idempotent consumers: processing the same message twice has the same effect as processing it once.

Common ways to make a consumer idempotent:

• Processed-message table: store the event ID (or orderId + event type) in the consumer's database in the same transaction as the business change; skip events already stored.
• Natural idempotency: write operations that are idempotent by nature — UPSERT, "set status to PAID" instead of "add 10 to the balance".
• Unique constraints: let the database reject the second insert of the same orderId.

Keeping Services Consistent

The dual-write problem

In Hands-on 9 the order service saves the order in Postgres and then publishes OrderCreated to Kafka. These are two different systems, and no transaction covers both:

• The database commit succeeds, then the service crashes before publishing → the order exists, but nobody is ever notified.
• The event is published, then the database transaction rolls back → consumers react to an order that does not exist.

Transactional outbox

The transactional outbox² removes the dual write. The service writes the event into an outbox table in the same local database transaction as the order. A separate relay (a polling job, or CDC with Debezium) reads the outbox and publishes to Kafka, marking rows as sent.

sequenceDiagram
    participant O as order-service
    participant DB as Postgres
    participant R as Outbox relay
    participant K as Kafka
    O->>DB: BEGIN
    O->>DB: INSERT INTO orders ...
    O->>DB: INSERT INTO outbox (OrderCreated)
    O->>DB: COMMIT (both or neither)
    R->>DB: read unsent outbox rows
    R->>K: publish OrderCreated
    R->>DB: mark row as sent

If the relay crashes after publishing but before marking the row, the event is published again — another reason consumers must be idempotent.

Sagas

Some business operations span several services: placing an order must reserve stock (product service) and charge the customer (payment service). Each service has its own database, so a single ACID transaction is impossible (see Microservices). A saga² is a sequence of local transactions, one per service, where each step is triggered by the previous one's event. If a step fails, the saga runs compensating transactions that semantically undo the steps already done (release the stock, refund the payment).

sequenceDiagram
    participant O as Order
    participant I as Inventory
    participant P as Payment
    O->>I: OrderCreated
    I->>P: StockReserved
    P->>I: PaymentFailed
    I->>O: StockReleased (compensation)
    O->>O: mark order CANCELLED

Dead Letter Queue

When a consumer fails to process a message repeatedly, it must not block the rest of the stream. A Dead Letter Queue (DLQ) is a separate topic where messages are moved after exhausting retry attempts, allowing the main stream to continue while the failing messages are investigated and reprocessed manually.

stateDiagram-v2
    [*] --> Pending
    Pending --> Processing : consumer picks up
    Processing --> Committed : success
    Processing --> Retry : transient failure (attempt < max)
    Retry --> Processing : after backoff
    Processing --> DLQ : max retries exceeded
    DLQ --> Processing : manual reprocessing

// default: 3 attempts in total; Spring Kafka 4 (Spring Boot 4) annotation
@RetryableTopic(backOff = @BackOff(delay = 1000, multiplier = 2))
@KafkaListener(topics = "order-events", groupId = "invoice-group")
public void onOrderCreated(OrderCreatedEvent event) {
    invoiceService.create(event);   // throws → retried after 1 s, then 2 s
}

@DltHandler
public void onDeadLetter(OrderCreatedEvent event) {
    log.error("order {} sent to the dead letter topic", event.orderId());
}

Retry only failures that can succeed later (a database briefly unavailable). A message that can never be processed — invalid JSON, an unknown product ID — should go straight to the DLQ; retrying it only delays the rest of the partition.

When to Use (and Not Use) Event-Driven

Connections

• Hands-on 9 — Kafka in the store: OrderCreated published by the order service and consumed by the notification service, with a DLQ.
• Microservices — database per service, the reason sagas and outboxes are needed.
• Distributed Systems — eventual consistency, ordering and consensus (KRaft).
• Reliability — retries, backoff and idempotency apply to consumers too.
• Observability and Hands-on 10 — tracing a request across topics.

Key takeaways

• Use events when the caller does not need the answer to continue; use a synchronous request when it does.
• Kafka is a replicated, partitioned log: order holds within a partition, the key decides the partition, and each consumer group keeps its own offsets — different groups fan out, the same group shares the work.
• Delivery is realistically at-least-once: commit after processing and make every consumer idempotent.
• Never dual-write: the transactional outbox publishes events reliably; sagas with compensations replace distributed transactions.
• Event sourcing and CQRS are powerful for specific domains, not defaults; a DLQ keeps poison messages from blocking a partition.