Kafka & Event Logs

Six months after BoxOffice adopted pub/sub, three requests arrive in the same week. The data team wants to load every order event of the last seven days into the warehouse — including events published before their loader existed. The search team shipped a bug that indexed wrong prices for two hours and wants to re-read those two hours after the fix. And the fraud team wants all events of the same order, in order, to detect "hold, release, hold, release" patterns from bots. A queue cannot do any of this: once a message is acknowledged, it is gone, and competing consumers scramble the order.

All three requests ask for the same thing: keep the messages, in order, and let each reader decide where to read from. That is a log, and Apache Kafka is the system that made the log the backbone of modern data architectures¹².

The log

A log here is not a text file of debug messages. It is the simplest storage abstraction there is: an append-only, totally ordered sequence of records. Writers can only add at the end; each record gets the next sequential number, its offset. Readers never remove anything — each reader just remembers the offset it has reached.

flowchart LR
    subgraph log ["order-events (one partition)"]
        direction LR
        r0["0<br/>OrderPlaced<br/>o-1"] --- r1["1<br/>OrderPlaced<br/>o-2"] --- r2["2<br/>OrderConfirmed<br/>o-1"] --- r3["3<br/>OrderCancelled<br/>o-2"] --- r4["4<br/>OrderPlaced<br/>o-3"] --- r5["5<br/>(next write)"]
    end
    n["notification<br/>committed offset 4"] -.-> r4
    a["analytics<br/>committed offset 2"] -.-> r2
    w["warehouse loader<br/>replaying from 0"] -.-> r0

This tiny change — reading does not delete — has big consequences:

• Any number of independent readers. notification, analytics and the warehouse loader read the same data at their own pace. A slow reader only falls behind; it slows down nobody.
• Replay. A reader can move its offset backwards and re-read history (the search team's fix), and a new reader can start from the beginning (the data team's loader).
• Order. Records are read in the order they were written — within one log.
• The log is the source of truth for "what happened". Databases, caches and search indexes can be rebuilt from it — the idea behind change data capture and event sourcing (Patterns).

Topics, partitions and offsets

A single log on a single machine would limit throughput to one disk and one machine. Kafka splits each topic into partitions: each partition is an independent log, stored on a different broker, with its own offsets. The topic is just the name of a set of partitions.

flowchart LR
    p1["order (replica 1)"] -->|"key o-7731"| P1
    p2["order (replica 2)"] -->|"key o-1002"| P0
    p2 -->|"key o-5120"| P2
    subgraph topic ["topic order-events"]
        P0["partition 0 · broker 1<br/>0 1 2 3 4 5 →"]
        P1["partition 1 · broker 2<br/>0 1 2 3 →"]
        P2["partition 2 · broker 3<br/>0 1 2 3 4 →"]
    end

• A record is identified by (topic, partition, offset). Offsets are per partition: offset 3 in partition 0 and offset 3 in partition 1 are unrelated records.
• Order is guaranteed only inside a partition. Two records in different partitions have no defined order.
• Partitions are the unit of parallelism: different partitions are written by different brokers and read by different consumers at the same time.
• On disk, a partition is a sequence of segment files (e.g. 1 GB each). Kafka only appends to the active segment and deletes or compacts whole old segments — this is what makes writes sequential and cheap.

Keys decide the partition

Each record may have a key. The default partitioner hashes the key (murmur2) and takes the result modulo the number of partitions:

Same key → same partition → in order. So the key is how you say what must stay in order. For BoxOffice order events, the key is the orderId: OrderPlaced, OrderConfirmed and OrderCancelled of the same order are always read in sequence, while different orders spread across partitions and are processed in parallel. Records without a key are spread across partitions by the sticky partitioner (it fills a batch for one partition, then moves to another), with no ordering between them.

The key also decides load balance. The simulation below hashes one minute of illustrative on-sale traffic into 12 partitions with Kafka's own hash function, using two different keys. In the on-sale minute, 60% of the purchases are for the stadium show.

Keyed by orderId (left), each partition gets about 1/12 of the traffic. Keyed by the show (right), every purchase of the stadium show lands in one hot partition: one broker and one consumer do most of the work while the others sit idle, and adding consumers does not help. The lesson: choose the key with the finest granularity that still preserves the ordering the business needs. Order events need per-order ordering, not per-show ordering.

Consumer groups

A consumer group is a set of consumer instances that share the work of reading a topic. Kafka assigns each partition to exactly one consumer of the group, so each record is processed by one member — competing consumers, but with ordering preserved per partition. Different groups are completely independent: each has its own committed offsets, so each group receives every record — that is the fan-out.

flowchart LR
    subgraph topic ["order-events · 6 partitions"]
        P0["P0"]
        P1["P1"]
        P2["P2"]
        P3["P3"]
        P4["P4"]
        P5["P5"]
    end
    subgraph gn ["group notification · 3 consumers"]
        N1["notification #1"]
        N2["notification #2"]
        N3["notification #3"]
    end
    subgraph ga ["group analytics · 1 consumer"]
        A1["analytics #1"]
    end
    P0 --> N1
    P1 --> N1
    P2 --> N2
    P3 --> N2
    P4 --> N3
    P5 --> N3
    P0 --> A1
    P1 --> A1
    P2 --> A1
    P3 --> A1
    P4 --> A1
    P5 --> A1

Rules that follow from "one partition, one consumer per group":

• Maximum parallelism of a group = number of partitions. With 6 partitions, a 7th notification instance gets nothing and sits idle.
• Order per key is kept as long as a consumer processes a partition's records sequentially. If it hands records to a thread pool, it breaks ordering again (unless it parallelizes by key).
• When a consumer joins, leaves or stops polling for too long (max.poll.interval.ms), the group rebalances: partitions are reassigned. Classic rebalances paused the whole group; the newer protocol (KIP-848, generally available in Kafka 4.0) and cooperative assignment move only the partitions that need to move.

Offsets and consumer lag

A consumer commits the offset of the next record it will read, per partition; Kafka stores commits in an internal compacted topic (__consumer_offsets). After a restart or a rebalance, the new owner of the partition resumes from the last committed offset. When you commit — before or after processing — decides whether a crash loses or duplicates records (Delivery semantics).

Consumer lag is the distance between the end of the partition and the committed offset — how many records are written but not yet processed. It is the health metric of a consumer:

$ kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group notification

GROUP         TOPIC         PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG    CONSUMER-ID
notification  order-events  0          182340          182355          15     notification-1-...
notification  order-events  1          181902          181910          8      notification-1-...
notification  order-events  2          120004          184771          64767  notification-2-...

Partition 2 is 64,767 records behind while the others are near zero: a hot key, a stuck consumer or a poison message in that partition — not a general capacity problem. Lag in records is easy to read from Kafka; lag in seconds (now minus the timestamp of the last processed record) is what you put in a freshness SLO.

Replication and durability

Each partition is replicated on several brokers — the replication factor (RF), typically 3, with the replicas in different availability zones. One replica is the leader: it receives all writes (and, by default, serves reads). The followers continuously fetch from the leader. Followers that are caught up form the in-sync replica set (ISR). If the leader's broker fails, the controller elects a new leader from the ISR.

sequenceDiagram
    participant P as producer (order)
    participant L as leader P1 (broker 2, AZ-b)
    participant F1 as follower (broker 1, AZ-a)
    participant F2 as follower (broker 3, AZ-c)
    P->>L: produce OrderConfirmed (acks=all)
    L->>L: append at offset 4211
    F1->>L: fetch from 4211
    L-->>F1: record 4211
    F2->>L: fetch from 4211
    L-->>F2: record 4211
    Note over L: all in-sync replicas have it: high watermark = 4212
    L-->>P: ack (offset 4211)
    Note over L,F2: consumers only see records below the high watermark

The producer setting acks chooses how much of this the producer waits for:

acks=all alone is not enough: if the ISR shrinks to just the leader, "all in-sync replicas" is one replica. min.insync.replicas=2 makes the broker refuse writes when fewer than 2 replicas are in sync. With RF = 3 and min ISR = 2, the topic keeps accepting writes with one broker down and never acknowledges a write that exists on only one machine. Keep unclean.leader.election.enable=false (the default) so an out-of-date replica is never elected leader. Since Kafka 3.0, producers default to acks=all with idempotence enabled.

Retention, compaction and tiered storage

A log cannot grow forever. Each topic has a cleanup policy:

A compacted topic behaves like a table that is also a stream: a new consumer reads it from the start and ends up with the latest value of every key. Deleting a key is done by writing a tombstone (the key with a null value). Kafka's own __consumer_offsets is a compacted topic.

Tiered storage (production-ready since Kafka 3.9) keeps only recent segments on the brokers' disks and moves older segments to object storage (S3, GCS, Azure Blob). Long retention — months of events for replay — becomes cheap, and brokers become lighter to replace.

Why Kafka is fast

A Kafka broker routinely moves hundreds of megabytes per second on commodity hardware. The reasons are worth knowing because they reappear in many systems³:

• Sequential I/O. Appends go to the end of a segment file; reads by caught-up consumers come from the same recent region. Sequential disk access is orders of magnitude faster than random access (see Numbers).
• The OS page cache. Kafka does not keep its own cache; it lets the operating system cache recent segments in memory, so consumers that keep up read from RAM.
• Zero-copy. To serve a consumer, the broker asks the kernel to send file bytes straight to the socket (sendfile), without copying them through the application.
• Batching and compression. Producers group records into batches per partition (linger.ms, batch.size) and compress whole batches (zstd, lz4). Fewer, bigger requests; fewer bytes on disk and on the network.
• Partitioning. Load spreads across many brokers and many consumers.

The trade-off is latency: batching adds a few milliseconds (linger.ms), and acks=all adds a replication round trip. Typical end-to-end latency is a few to tens of milliseconds — fine for events, not a replacement for an in-memory RPC.

Sizing partitions

Partitions are cheap but not free (more files, more leader elections, longer rebalances, more memory in clients). A common starting rule⁴: with a target throughput \(T\), a measured per-partition producer throughput \(p\) and a per-partition consumer throughput \(c\),

then round up with headroom for growth, because increasing partitions later moves keys.

The topic is created explicitly (never rely on auto-creation in production):

kafka-topics.sh --bootstrap-server kafka:9092 --create --topic order-events \
  --partitions 24 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000 \
  --config compression.type=zstd

And the Spring Boot services configure their clients accordingly:

spring:
  kafka:
    bootstrap-servers: kafka:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
      acks: all                    # wait for every in-sync replica
      properties:
        enable.idempotence: true   # producer retries do not duplicate records
        linger.ms: 5               # wait up to 5 ms to fill a batch
        compression.type: zstd

spring:
  kafka:
    bootstrap-servers: kafka:9092
    consumer:
      group-id: notification
      auto-offset-reset: earliest  # a brand-new group starts from the oldest record
      enable-auto-commit: false    # the container commits after processing
      max-poll-records: 100
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
      properties:
        spring.json.trusted.packages: "boxoffice.events"
    listener:
      ack-mode: record             # commit the offset after each record
      concurrency: 3               # 3 consumer threads in this instance

kafka:
  image: apache/kafka:4.3.1
  hostname: kafka
  ports:
    - "9094:9094"
  environment:
    KAFKA_NODE_ID: 1
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
    KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
kafka-init:
  image: apache/kafka:4.3.1
  depends_on:
    kafka:
      condition: service_healthy
  entrypoint: ["/bin/sh", "-c"]
  command:
    - |
      /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
        --topic order-events --partitions 6 --replication-factor 1 &&
      /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
        --topic order-events-dlt --partitions 6 --replication-factor 1
  restart: "no"

Kafka and its alternatives

Redpanda and Azure Event Hubs expose the Kafka protocol, so Kafka clients work against them unchanged. Kafka itself is also growing queue semantics: share groups ("Queues for Kafka", KIP-932, introduced in Kafka 4.0) let several consumers share a partition with per-record acknowledgement, like a classic queue.

How it connects

flowchart LR
    LOG["append-only log<br/>offsets"] --> PART["partitions<br/>unit of parallelism"]
    KEY["key = orderId<br/>murmur2 mod N"] --> PART
    PART --> ORD["order per key<br/>→ Delivery semantics"]
    PART --> CG["consumer groups<br/>one consumer per partition"]
    CG --> LAG["consumer lag<br/>→ Observability"]
    PART --> REP["RF 3, acks=all,<br/>min.insync.replicas=2"]
    REP --> KRAFT["KRaft controllers<br/>→ Consensus (class 3)"]
    LOG --> RET["retention, compaction,<br/>tiered storage"]
    LOG --> REPLAY["replay: CDC, event sourcing<br/>→ Patterns"]
    PART --> SIZE["N ≥ max(T/p, T/c)<br/>→ Estimation"]
    KEY --> HOT["hot key, hash mod N moves keys<br/>→ Databases partitioning"]

Palette components: kafka · stream (the managed equivalent: Kinesis, Event Hubs) · objstore (tiered storage) · dwh (the warehouse loader replays the log) · notifications · analytics · search (consumer groups reading at their own pace). Scenarios: event-fanout. Continues in: Delivery semantics, Patterns, Distributed — consensus (KRaft is Raft), Databases — partitioning (hot keys, resharding), Observability — signals (lag as a metric), Hands-on 7 — Kafka.