Kafka & Event Logs
Six months after BoxOffice adopted pub/sub, three requests arrive in the same week. The data team wants to load every order event of the last seven days into the warehouse β including events published before their loader existed. The search team shipped a bug that indexed wrong prices for two hours and wants to re-read those two hours after the fix. And the fraud team wants all events of the same order, in order, to detect "hold, release, hold, release" patterns from bots. A queue cannot do any of this: once a message is acknowledged, it is gone, and competing consumers scramble the order.
All three requests ask for the same thing: keep the messages, in order, and let each reader decide where to read from. That is a log, and Apache Kafka is the system that made the log the backbone of modern data architectures12.
The log
A log here is not a text file of debug messages. It is the simplest storage abstraction there is: an append-only, totally ordered sequence of records. Writers can only add at the end; each record gets the next sequential number, its offset. Readers never remove anything β each reader just remembers the offset it has reached.
flowchart LR
subgraph log ["order-events (one partition)"]
direction LR
r0["0<br/>OrderPlaced<br/>o-1"] --- r1["1<br/>OrderPlaced<br/>o-2"] --- r2["2<br/>OrderConfirmed<br/>o-1"] --- r3["3<br/>OrderCancelled<br/>o-2"] --- r4["4<br/>OrderPlaced<br/>o-3"] --- r5["5<br/>(next write)"]
end
n["notification<br/>committed offset 4"] -.-> r4
a["analytics<br/>committed offset 2"] -.-> r2
w["warehouse loader<br/>replaying from 0"] -.-> r0 This tiny change β reading does not delete β has big consequences:
- Any number of independent readers.
notification,analyticsand the warehouse loader read the same data at their own pace. A slow reader only falls behind; it slows down nobody. - Replay. A reader can move its offset backwards and re-read history (the search team's fix), and a new reader can start from the beginning (the data team's loader).
- Order. Records are read in the order they were written β within one log.
- The log is the source of truth for "what happened". Databases, caches and search indexes can be rebuilt from it β the idea behind change data capture and event sourcing (Patterns).
Topics, partitions and offsets
A single log on a single machine would limit throughput to one disk and one machine. Kafka splits each topic into partitions: each partition is an independent log, stored on a different broker, with its own offsets. The topic is just the name of a set of partitions.
flowchart LR
p1["order (replica 1)"] -->|"key o-7731"| P1
p2["order (replica 2)"] -->|"key o-1002"| P0
p2 -->|"key o-5120"| P2
subgraph topic ["topic order-events"]
P0["partition 0 Β· broker 1<br/>0 1 2 3 4 5 β"]
P1["partition 1 Β· broker 2<br/>0 1 2 3 β"]
P2["partition 2 Β· broker 3<br/>0 1 2 3 4 β"]
end - A record is identified by (topic, partition, offset). Offsets are per partition: offset 3 in partition 0 and offset 3 in partition 1 are unrelated records.
- Order is guaranteed only inside a partition. Two records in different partitions have no defined order.
- Partitions are the unit of parallelism: different partitions are written by different brokers and read by different consumers at the same time.
- On disk, a partition is a sequence of segment files (e.g. 1 GB each). Kafka only appends to the active segment and deletes or compacts whole old segments β this is what makes writes sequential and cheap.
Keys decide the partition
Each record may have a key. The default partitioner hashes the key (murmur2) and takes the result modulo the number of partitions:
Same key β same partition β in order. So the key is how you say what must stay in order. For BoxOffice order events, the key is the orderId: OrderPlaced, OrderConfirmed and OrderCancelled of the same order are always read in sequence, while different orders spread across partitions and are processed in parallel. Records without a key are spread across partitions by the sticky partitioner (it fills a batch for one partition, then moves to another), with no ordering between them.
The key also decides load balance. The simulation below hashes one minute of illustrative on-sale traffic into 12 partitions with Kafka's own hash function, using two different keys. In the on-sale minute, 60% of the purchases are for the stadium show.
Keyed by orderId (left), each partition gets about 1/12 of the traffic. Keyed by the show (right), every purchase of the stadium show lands in one hot partition: one broker and one consumer do most of the work while the others sit idle, and adding consumers does not help. The lesson: choose the key with the finest granularity that still preserves the ordering the business needs. Order events need per-order ordering, not per-show ordering.
Changing the number of partitions moves keys
hash(key) mod N changes when \(N\) changes. If you grow order-events from 12 to 24 partitions, about half of the keys move to a different partition, and for a while new events of an order can be read before old events still waiting in the old partition. Size partitions with headroom up front; if you must grow, do it when traffic is low, or create a new topic and migrate consumers.
Consumer groups
A consumer group is a set of consumer instances that share the work of reading a topic. Kafka assigns each partition to exactly one consumer of the group, so each record is processed by one member β competing consumers, but with ordering preserved per partition. Different groups are completely independent: each has its own committed offsets, so each group receives every record β that is the fan-out.
flowchart LR
subgraph topic ["order-events Β· 6 partitions"]
P0["P0"]
P1["P1"]
P2["P2"]
P3["P3"]
P4["P4"]
P5["P5"]
end
subgraph gn ["group notification Β· 3 consumers"]
N1["notification #1"]
N2["notification #2"]
N3["notification #3"]
end
subgraph ga ["group analytics Β· 1 consumer"]
A1["analytics #1"]
end
P0 --> N1
P1 --> N1
P2 --> N2
P3 --> N2
P4 --> N3
P5 --> N3
P0 --> A1
P1 --> A1
P2 --> A1
P3 --> A1
P4 --> A1
P5 --> A1 Rules that follow from "one partition, one consumer per group":
- Maximum parallelism of a group = number of partitions. With 6 partitions, a 7th
notificationinstance gets nothing and sits idle. - Order per key is kept as long as a consumer processes a partition's records sequentially. If it hands records to a thread pool, it breaks ordering again (unless it parallelizes by key).
- When a consumer joins, leaves or stops polling for too long (
max.poll.interval.ms), the group rebalances: partitions are reassigned. Classic rebalances paused the whole group; the newer protocol (Kafka Improvement Proposal KIP-848, generally available in Kafka 4.0) and cooperative assignment move only the partitions that need to move.
Try it
The defaults are lab 7's: order-events has 6 partitions and notification runs 3 consumers (NOTIFY_CONCURRENCY). Press play and the lag stays near zero. Then:
- Raise Consumers to 8: 2 of them get no partition and sit idle.
- Switch Key to none: OrderPlaced and OrderConfirmed of the same order land in different partitions, and some orders are read confirmed-before-placed (22 in 200 ticks with 3 consumers, 107 with 2). Back to orderId: never.
- Switch Key to show: the stadium show
ev-1hashes to P5, which falls behind (183 records after 200 ticks) while the other partitions stay near zero β and with 6 consumers it is still 142 behind. - Kill a consumer: the whole group stops for the rebalance, lag grows on every partition, then the 2 survivors take 3 partitions each and resume from the committed offsets.
Offsets and consumer lag
A consumer commits the offset of the next record it will read, per partition; Kafka stores commits in an internal compacted topic (__consumer_offsets). After a restart or a rebalance, the new owner of the partition resumes from the last committed offset. When you commit β before or after processing β decides whether a crash loses or duplicates records (Delivery semantics).
Consumer lag is the distance between the end of the partition and the committed offset β how many records are written but not yet processed. It is the health metric of a consumer:
$ kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group notification
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID
notification order-events 0 182340 182355 15 notification-1-...
notification order-events 1 181902 181910 8 notification-1-...
notification order-events 2 120004 184771 64767 notification-2-...
Partition 2 is 64,767 records behind while the others are near zero: a hot key, a stuck consumer or a poison message in that partition β not a general capacity problem. Lag in records is easy to read from Kafka; lag in seconds (now minus the timestamp of the last processed record) is what you put in a freshness service level objective (SLO).
Try it in the playground
The event-fanout scenario publishes order events to Kafka with several consumer services. Run it with the load profile and watch the per-consumer lag. Then slow down one consumer (raise its processing time) and confirm that only its lag grows β the producer and the other consumers are unaffected. Finally add consumer instances to the slow service and observe that it stops improving once instances equal partitions.
The boxoffice-outbox preset is lab 7 in numbers: notification sends each e-mail in 50 ms, so 3 consumers send at most 60 e-mails/s, and a burst of 200 orders/s lasts 30 s. Put Kafka (6 partitions) between order and notification: with 6 consumers the e-mails catch up with about 30 s of lag, with 3 the lag passes 60 s, and 12 consumers lag exactly like 6 β the partitions cap the group.
Replication and durability
Each partition is replicated on several brokers β the replication factor (RF), typically 3, with the replicas in different availability zones. One replica is the leader: it receives all writes (and, by default, serves reads). The followers continuously fetch from the leader. Followers that are caught up form the in-sync replica set (ISR). If the leader's broker fails, the controller elects a new leader from the ISR.
sequenceDiagram
participant P as producer (order)
participant L as leader P1 (broker 2, AZ-b)
participant F1 as follower (broker 1, AZ-a)
participant F2 as follower (broker 3, AZ-c)
P->>L: produce OrderConfirmed (acks=all)
L->>L: append at offset 4211
F1->>L: fetch from 4211
L-->>F1: record 4211
F2->>L: fetch from 4211
L-->>F2: record 4211
Note over L: all in-sync replicas have it: high watermark = 4212
L-->>P: ack (offset 4211)
Note over L,F2: consumers only see records below the high watermark The producer setting acks chooses how much of this the producer waits for:
acks | Producer gets the confirmation when⦠| A broker failure can⦠| Latency |
|---|---|---|---|
0 | the record is sent (no confirmation) | β lose records silently | lowest |
1 | the leader has written it | β οΈ lose acknowledged records if the leader dies before followers copy them | low |
all | every in-sync replica has it | β
not lose acknowledged records, if min.insync.replicas β₯ 2 | highest (one extra round trip inside the cluster) |
acks=all alone is not enough: if the ISR shrinks to just the leader, "all in-sync replicas" is one replica. min.insync.replicas=2 makes the broker refuse writes when fewer than 2 replicas are in sync. With RF = 3 and min ISR = 2, the topic keeps accepting writes with one broker down and never acknowledges a write that exists on only one machine. Keep unclean.leader.election.enable=false (the default) so an out-of-date replica is never elected leader. Since Kafka 3.0, producers default to acks=all with idempotence enabled.
KRaft: Kafka's own consensus
Cluster metadata β which topics exist, which broker leads each partition, the ISR β is kept by a small quorum of controllers using KRaft, a Raft-based protocol (see Consensus). Older clusters used ZooKeeper; Kafka 4.0 (2025) removed ZooKeeper entirely. The course labs run Kafka in KRaft mode: in lab 7 a single node is both broker and controller.
Retention, compaction and tiered storage
A log cannot grow forever. Each topic has a cleanup policy:
| Policy | Configuration | Keeps | BoxOffice use |
|---|---|---|---|
| Delete by time | retention.ms (default 7 days) | everything newer than the limit | order-events: 7 days for replay and the warehouse loader |
| Delete by size | retention.bytes (per partition; default unlimited) | the newest N bytes of each partition | High-volume click-stream with a disk budget |
| Compact | cleanup.policy=compact | at least the latest record of each key | event-catalog: the current state of each event (name, venue, date) |
| Both | cleanup.policy=compact,delete | latest per key, but not older than the retention | Latest seat-map snapshot per show, for 30 days |
A compacted topic behaves like a table that is also a stream: a new consumer reads it from the start and ends up with the latest value of every key. Deleting a key is done by writing a tombstone (the key with a null value). Kafka's own __consumer_offsets is a compacted topic.
flowchart LR
b0["0 Β· ev-88<br/>21:00"]:::problem --- b1["1 Β· ev-90<br/>Jazz"]:::problem --- b2["2 Β· ev-88<br/>20:00"] --- b3["3 Β· ev-91<br/>Theater"] --- b4["4 Β· ev-90<br/>tombstone"]
b4 ==>|"compaction keeps the latest record of each key"| a2["2 Β· ev-88<br/>20:00"]:::comp
a2 --- a3["3 Β· ev-91<br/>Theater"]:::comp Compaction removes records that a newer record of the same key replaced (offsets 0 and 1), and later the tombstone itself (after delete.retention.ms, one day by default). The survivors keep their offsets: a consumer that stored offset 3 is still right.
Tiered storage (production-ready since Kafka 3.9) keeps only recent segments on the brokers' disks and moves older segments to object storage (S3, GCS, Azure Blob). Long retention β months of events for replay β becomes cheap, and brokers become lighter to replace.
Why Kafka is fast
A Kafka broker routinely moves hundreds of megabytes per second on commodity hardware. The reasons are worth knowing because they reappear in many systems3:
- Sequential I/O. Appends go to the end of a segment file; reads by caught-up consumers come from the same recent region. Sequential disk access is orders of magnitude faster than random access (see Numbers).
- The OS page cache. Kafka does not keep its own cache; it lets the operating system cache recent segments in memory, so consumers that keep up read from RAM.
- Zero-copy. To serve a consumer, the broker asks the kernel to send file bytes straight to the socket (
sendfile), without copying them through the application. - Batching and compression. Producers group records into batches per partition (
linger.ms,batch.size) and compress whole batches (zstd,lz4). Fewer, bigger requests; fewer bytes on disk and on the network. - Partitioning. Load spreads across many brokers and many consumers.
The trade-off is latency: batching adds a few milliseconds (linger.ms), and acks=all adds a replication round trip. Typical end-to-end latency is a few to tens of milliseconds β fine for events, not a replacement for an in-memory remote procedure call (RPC).
Sizing partitions
Partitions are cheap but not free (more files, more leader elections, longer rebalances, more memory in clients). A common starting rule4: with a target throughput \(T\), a measured per-partition producer throughput \(p\) and a per-partition consumer throughput \(c\),
then round up with headroom for growth, because increasing partitions later moves keys.
BoxOffice: partitions for order-events
Illustrative peak: 30,000 orders in the first 3 minutes of a big on-sale, each producing about 10 events (placed, held, payment authorized, confirmed, ticket issuedβ¦): \(T \approx 30\,000 \times 10 / 180 \approx 1{,}700\) events/s. The records are small (~1 KB), so the producer side is trivial: a single partition handles far more than 1.7 MB/s.
The consumer side decides. notification calls the e-mail provider for each OrderConfirmed, ~50 ms per call, processing one record at a time per partition: \(c = 1 / 0.05 = 20\) records/s per partition. For its share of the traffic (the 30,000 OrderConfirmed in 180 s β 167/s):
With headroom for bigger shows and growth, BoxOffice creates order-events with 24 partitions. If one day that is not enough, notification can batch e-mails or process records concurrently per key inside each partition instead of adding partitions.
That is the production cluster (three brokers, replication factor 3). Hands-on 7 runs a single broker on a laptop and creates order-events with 6 partitions and replication factor 1: its load is far smaller, and the same rule sets its ceiling β at 20 e-mails/s per partition, 6 partitions deliver at most 120 confirmations/s, and a seventh notification thread would sit idle.
The topic is created explicitly (never rely on auto-creation in production):
kafka-topics.sh --bootstrap-server kafka:9092 --create --topic order-events \
--partitions 24 --replication-factor 3 \
--config min.insync.replicas=2 \
--config retention.ms=604800000 \
--config compression.type=zstd
And the Spring Boot services configure their clients accordingly:
spring:
kafka:
bootstrap-servers: kafka:9092
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
acks: all # wait for every in-sync replica
properties:
enable.idempotence: true # producer retries do not duplicate records
linger.ms: 5 # wait up to 5 ms to fill a batch
compression.type: zstd
spring:
kafka:
bootstrap-servers: kafka:9092
consumer:
group-id: notification
auto-offset-reset: earliest # a brand-new group starts from the oldest record
enable-auto-commit: false # the container commits after processing
max-poll-records: 100
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
properties:
spring.json.trusted.packages: "boxoffice.events"
listener:
ack-mode: record # commit the offset after each record
concurrency: 3 # 3 consumer threads in this instance
In the BoxOffice code: Kafka in Compose (lab 7)
Hands-on 7 β Kafka in Compose runs one node that is broker and KRaft controller at once, with auto-creation off and both topics created by a one-shot container: order-events and order-events-dlt with 6 partitions and replication factor 1. One laptop cannot hold three brokers in three zones, so this is the place where the lab is not production: acks: all in order waits for "all in-sync replicas", and with RF = 1 that is one replica β exactly the trap described in Replication and durability. The 6 partitions bound the consumer side: notification runs 3 threads (NOTIFY_CONCURRENCY), and more than 6 would sit idle. The lab's burst-and-drain experiment applies this page's arithmetic: one thread at 50 ms per e-mail handles 20/s, so 100 orders/s for 60 s leave about 4,800 messages of lag that take about 240 s to drain; six threads keep the lag near zero.
kafka:
image: apache/kafka:4.3.1
hostname: kafka
ports:
- "9094:9094"
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
kafka-init:
image: apache/kafka:4.3.1
depends_on:
kafka:
condition: service_healthy
entrypoint: ["/bin/sh", "-c"]
command:
- |
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
--topic order-events --partitions 6 --replication-factor 1 &&
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
--topic order-events-dlt --partitions 6 --replication-factor 1
restart: "no"
Kafka and its alternatives
| Apache Kafka | Amazon Kinesis Data Streams | Google Cloud Pub/Sub | Apache Pulsar | |
|---|---|---|---|---|
| Model | partitioned replicated log | partitioned log (shards) | pub/sub with per-message ack | pub/sub + log (segments in BookKeeper) |
| Ordering | per partition | per shard | per ordering key (optional) | per partition / key |
| Replay | yes, any offset in retention | yes, within retention (up to 365 days) | seek to a time or snapshot | yes |
| Operations | self-managed or managed (Amazon MSK, Confluent Cloud, Aiven) | fully managed | fully managed, global | self-managed or managed |
| Good fit | event backbone, change data capture (CDC), stream processing, very high throughput | AWS-native streaming | Google Cloud-native fan-out, push to serverless | multi-tenant messaging and streaming |
Redpanda and Azure Event Hubs expose the Kafka protocol, so Kafka clients work against them unchanged. Kafka itself is also growing queue semantics: share groups ("Queues for Kafka", KIP-932, introduced in Kafka 4.0) let several consumers share a partition with per-record acknowledgement, like a classic queue.
In the interview
When you put Kafka on the board, the interviewer expects four decisions, said out loud: (1) the topic and the key ("order-events, keyed by orderId, because we need per-order ordering and an even spread"); (2) partitions from a throughput estimate with the consumer as the usual bottleneck; (3) durability (RF 3 across zones, acks=all, min.insync.replicas=2); (4) retention (7 days to allow replay, or compaction for state). Mention consumer lag as the metric you alert on. Be ready for "what happens if a broker dies?" (leader election from the ISR, producers retry, no acknowledged data lost) and "how do you scale consumers?" (up to the number of partitions).
Best practices
- Create every topic explicitly, with partitions, replication and retention reviewed like code β auto-creation makes topics with broker defaults nobody chose. In BoxOffice:
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"and akafka-initcontainer (Compose box). - Place the three replicas of each partition in three availability zones (
broker.rack) β losing one zone leaves 2 replicas in sync, somin.insync.replicas=2keeps accepting writes. - Set retention by the replay you need, then size the disk β at least the longest consumer outage you plan to recover from, plus a margin (7 days for
order-events); disk β daily volume Γ retention days Γ replication factor. - Watch lag per partition, not only per group β one hot key (60% of the purchases are for
ev-1) hides inside a healthy group total. In BoxOffice: lab 7'skafka-lag.shprints the group total;kafka-consumer-groups.sh --describeshows each partition (lab 7 β measure). - Batch and compress on the producer β
linger.msof a few milliseconds andcompression.type=zstdcut requests and bytes; 5 ms is invisible next to a 300 ms SLO. In BoxOffice:ordersetslinger.ms: 5,acks: allandenable.idempotence: true. - Use incremental (cooperative) rebalancing for consumer groups β with eager rebalancing, a rolling deploy of 6 consumers stops the whole group at every restart.
Common pitfalls
- A low-cardinality key (country, show,
true/false) creates hot partitions that no number of consumers can fix. - Too few partitions: the group cannot scale beyond them, and adding partitions later remaps keys.
acks=1ormin.insync.replicas=1for business events: "acknowledged" does not mean "safe".- Processing a partition with a thread pool without keying the work: ordering is silently lost.
- Slow processing inside the poll loop beyond
max.poll.interval.ms: the consumer is kicked out, the group rebalances, another consumer gets the same records β a rebalance storm with duplicates. - Using Kafka as a database for queries by arbitrary fields. Kafka is read sequentially by offset; build query models in a database from it (Command Query Responsibility Segregation, CQRS).
How it connects
flowchart LR
LOG["append-only log<br/>offsets"] --> PART["partitions<br/>unit of parallelism"]
KEY["key = orderId<br/>murmur2 mod N"] --> PART
PART --> ORD["order per key<br/>β Delivery semantics"]
PART --> CG["consumer groups<br/>one consumer per partition"]
CG --> LAG["consumer lag<br/>β Observability"]
PART --> REP["RF 3, acks=all,<br/>min.insync.replicas=2"]
REP --> KRAFT["KRaft controllers<br/>β Consensus (class 3)"]
LOG --> RET["retention, compaction,<br/>tiered storage"]
LOG --> REPLAY["replay: CDC, event sourcing<br/>β Patterns"]
PART --> SIZE["N β₯ max(T/p, T/c)<br/>β Estimation"]
KEY --> HOT["hot key, hash mod N moves keys<br/>β Databases partitioning"] Palette components: kafka Β· stream (the managed equivalent: Kinesis, Event Hubs) Β· objstore (tiered storage) Β· dwh (the warehouse loader replays the log) Β· notifications Β· analytics Β· search (consumer groups reading at their own pace). Scenarios: event-fanout. Continues in: Delivery semantics, Patterns, Distributed β consensus (KRaft is Raft), Databases β partitioning (hot keys, resharding), Observability β signals (lag as a metric), Hands-on 7 β Kafka.
-
KREPS, J.; NARKHEDE, N.; RAO, J. Kafka: a Distributed Messaging System for Log Processing. In: NetDB Workshop, 2011.Β β©
-
KREPS, J. The Log: What every software engineer should know about real-time data's unifying abstraction. LinkedIn Engineering, 2013.Β β©
-
Apache Kafka Documentation β Design. Apache Software Foundation.Β β©
-
SHAPIRA, G.; PALINO, T.; SIVARAM, R.; PETTY, K. Kafka: The Definitive Guide, 2nd ed. O'Reilly, 2021. Chapters on producers, consumers and reliable delivery.Β β©