Skip to content

Kafka & Event Logs

Six months after BoxOffice adopted pub/sub, three requests arrive in the same week. The data team wants to load every order event of the last seven days into the warehouse β€” including events published before their loader existed. The search team shipped a bug that indexed wrong prices for two hours and wants to re-read those two hours after the fix. And the fraud team wants all events of the same order, in order, to detect "hold, release, hold, release" patterns from bots. A queue cannot do any of this: once a message is acknowledged, it is gone, and competing consumers scramble the order.

All three requests ask for the same thing: keep the messages, in order, and let each reader decide where to read from. That is a log, and Apache Kafka is the system that made the log the backbone of modern data architectures12.


The log

A log here is not a text file of debug messages. It is the simplest storage abstraction there is: an append-only, totally ordered sequence of records. Writers can only add at the end; each record gets the next sequential number, its offset. Readers never remove anything β€” each reader just remembers the offset it has reached.

flowchart LR
    subgraph log ["order-events (one partition)"]
        direction LR
        r0["0<br/>OrderPlaced<br/>o-1"] --- r1["1<br/>OrderPlaced<br/>o-2"] --- r2["2<br/>OrderConfirmed<br/>o-1"] --- r3["3<br/>OrderCancelled<br/>o-2"] --- r4["4<br/>OrderPlaced<br/>o-3"] --- r5["5<br/>(next write)"]
    end
    n["notification<br/>committed offset 4"] -.-> r4
    a["analytics<br/>committed offset 2"] -.-> r2
    w["warehouse loader<br/>replaying from 0"] -.-> r0

This tiny change β€” reading does not delete β€” has big consequences:

  • Any number of independent readers. notification, analytics and the warehouse loader read the same data at their own pace. A slow reader only falls behind; it slows down nobody.
  • Replay. A reader can move its offset backwards and re-read history (the search team's fix), and a new reader can start from the beginning (the data team's loader).
  • Order. Records are read in the order they were written β€” within one log.
  • The log is the source of truth for "what happened". Databases, caches and search indexes can be rebuilt from it β€” the idea behind change data capture and event sourcing (Patterns).

Topics, partitions and offsets

A single log on a single machine would limit throughput to one disk and one machine. Kafka splits each topic into partitions: each partition is an independent log, stored on a different broker, with its own offsets. The topic is just the name of a set of partitions.

flowchart LR
    p1["order (replica 1)"] -->|"key o-7731"| P1
    p2["order (replica 2)"] -->|"key o-1002"| P0
    p2 -->|"key o-5120"| P2
    subgraph topic ["topic order-events"]
        P0["partition 0 Β· broker 1<br/>0 1 2 3 4 5 β†’"]
        P1["partition 1 Β· broker 2<br/>0 1 2 3 β†’"]
        P2["partition 2 Β· broker 3<br/>0 1 2 3 4 β†’"]
    end
  • A record is identified by (topic, partition, offset). Offsets are per partition: offset 3 in partition 0 and offset 3 in partition 1 are unrelated records.
  • Order is guaranteed only inside a partition. Two records in different partitions have no defined order.
  • Partitions are the unit of parallelism: different partitions are written by different brokers and read by different consumers at the same time.
  • On disk, a partition is a sequence of segment files (e.g. 1 GB each). Kafka only appends to the active segment and deletes or compacts whole old segments β€” this is what makes writes sequential and cheap.

Keys decide the partition

Each record may have a key. The default partitioner hashes the key (murmur2) and takes the result modulo the number of partitions:

\[ \text{partition} = \text{murmur2}(\text{key}) \bmod N_{\text{partitions}} \]

Same key β†’ same partition β†’ in order. So the key is how you say what must stay in order. For BoxOffice order events, the key is the orderId: OrderPlaced, OrderConfirmed and OrderCancelled of the same order are always read in sequence, while different orders spread across partitions and are processed in parallel. Records without a key are spread across partitions by the sticky partitioner (it fills a batch for one partition, then moves to another), with no ordering between them.

The key also decides load balance. The simulation below hashes one minute of illustrative on-sale traffic into 12 partitions with Kafka's own hash function, using two different keys. In the on-sale minute, 60% of the purchases are for the stadium show.

2026-11-18T00:40:11.496071 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 1 2 3 4 5 6 7 8 9 10 11 partition 0 10 20 30 40 50 60 % of the messages key = orderId: busiest partition 8% 0 1 2 3 4 5 6 7 8 9 10 11 partition key = eventId (the show): busiest partition 64%

Keyed by orderId (left), each partition gets about 1/12 of the traffic. Keyed by the show (right), every purchase of the stadium show lands in one hot partition: one broker and one consumer do most of the work while the others sit idle, and adding consumers does not help. The lesson: choose the key with the finest granularity that still preserves the ordering the business needs. Order events need per-order ordering, not per-show ordering.

Changing the number of partitions moves keys

hash(key) mod N changes when \(N\) changes. If you grow order-events from 12 to 24 partitions, about half of the keys move to a different partition, and for a while new events of an order can be read before old events still waiting in the old partition. Size partitions with headroom up front; if you must grow, do it when traffic is low, or create a new topic and migrate consumers.


Consumer groups

A consumer group is a set of consumer instances that share the work of reading a topic. Kafka assigns each partition to exactly one consumer of the group, so each record is processed by one member β€” competing consumers, but with ordering preserved per partition. Different groups are completely independent: each has its own committed offsets, so each group receives every record β€” that is the fan-out.

flowchart LR
    subgraph topic ["order-events Β· 6 partitions"]
        P0["P0"]
        P1["P1"]
        P2["P2"]
        P3["P3"]
        P4["P4"]
        P5["P5"]
    end
    subgraph gn ["group notification Β· 3 consumers"]
        N1["notification #1"]
        N2["notification #2"]
        N3["notification #3"]
    end
    subgraph ga ["group analytics Β· 1 consumer"]
        A1["analytics #1"]
    end
    P0 --> N1
    P1 --> N1
    P2 --> N2
    P3 --> N2
    P4 --> N3
    P5 --> N3
    P0 --> A1
    P1 --> A1
    P2 --> A1
    P3 --> A1
    P4 --> A1
    P5 --> A1

Rules that follow from "one partition, one consumer per group":

  • Maximum parallelism of a group = number of partitions. With 6 partitions, a 7th notification instance gets nothing and sits idle.
  • Order per key is kept as long as a consumer processes a partition's records sequentially. If it hands records to a thread pool, it breaks ordering again (unless it parallelizes by key).
  • When a consumer joins, leaves or stops polling for too long (max.poll.interval.ms), the group rebalances: partitions are reassigned. Classic rebalances paused the whole group; the newer protocol (Kafka Improvement Proposal KIP-848, generally available in Kafka 4.0) and cooperative assignment move only the partitions that need to move.

Try it

The defaults are lab 7's: order-events has 6 partitions and notification runs 3 consumers (NOTIFY_CONCURRENCY). Press play and the lag stays near zero. Then:

  • Raise Consumers to 8: 2 of them get no partition and sit idle.
  • Switch Key to none: OrderPlaced and OrderConfirmed of the same order land in different partitions, and some orders are read confirmed-before-placed (22 in 200 ticks with 3 consumers, 107 with 2). Back to orderId: never.
  • Switch Key to show: the stadium show ev-1 hashes to P5, which falls behind (183 records after 200 ticks) while the other partitions stay near zero β€” and with 6 consumers it is still 142 behind.
  • Kill a consumer: the whole group stops for the rebalance, lag grows on every partition, then the 2 survivors take 3 partitions each and resume from the committed offsets.

Offsets and consumer lag

A consumer commits the offset of the next record it will read, per partition; Kafka stores commits in an internal compacted topic (__consumer_offsets). After a restart or a rebalance, the new owner of the partition resumes from the last committed offset. When you commit β€” before or after processing β€” decides whether a crash loses or duplicates records (Delivery semantics).

Consumer lag is the distance between the end of the partition and the committed offset β€” how many records are written but not yet processed. It is the health metric of a consumer:

$ kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group notification

GROUP         TOPIC         PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG    CONSUMER-ID
notification  order-events  0          182340          182355          15     notification-1-...
notification  order-events  1          181902          181910          8      notification-1-...
notification  order-events  2          120004          184771          64767  notification-2-...

Partition 2 is 64,767 records behind while the others are near zero: a hot key, a stuck consumer or a poison message in that partition β€” not a general capacity problem. Lag in records is easy to read from Kafka; lag in seconds (now minus the timestamp of the last processed record) is what you put in a freshness service level objective (SLO).

Try it in the playground

The event-fanout scenario publishes order events to Kafka with several consumer services. Run it with the load profile and watch the per-consumer lag. Then slow down one consumer (raise its processing time) and confirm that only its lag grows β€” the producer and the other consumers are unaffected. Finally add consumer instances to the slow service and observe that it stops improving once instances equal partitions.

The boxoffice-outbox preset is lab 7 in numbers: notification sends each e-mail in 50 ms, so 3 consumers send at most 60 e-mails/s, and a burst of 200 orders/s lasts 30 s. Put Kafka (6 partitions) between order and notification: with 6 consumers the e-mails catch up with about 30 s of lag, with 3 the lag passes 60 s, and 12 consumers lag exactly like 6 β€” the partitions cap the group.


Replication and durability

Each partition is replicated on several brokers β€” the replication factor (RF), typically 3, with the replicas in different availability zones. One replica is the leader: it receives all writes (and, by default, serves reads). The followers continuously fetch from the leader. Followers that are caught up form the in-sync replica set (ISR). If the leader's broker fails, the controller elects a new leader from the ISR.

sequenceDiagram
    participant P as producer (order)
    participant L as leader P1 (broker 2, AZ-b)
    participant F1 as follower (broker 1, AZ-a)
    participant F2 as follower (broker 3, AZ-c)
    P->>L: produce OrderConfirmed (acks=all)
    L->>L: append at offset 4211
    F1->>L: fetch from 4211
    L-->>F1: record 4211
    F2->>L: fetch from 4211
    L-->>F2: record 4211
    Note over L: all in-sync replicas have it: high watermark = 4212
    L-->>P: ack (offset 4211)
    Note over L,F2: consumers only see records below the high watermark

The producer setting acks chooses how much of this the producer waits for:

acks Producer gets the confirmation when… A broker failure can… Latency
0 the record is sent (no confirmation) ❌ lose records silently lowest
1 the leader has written it ⚠️ lose acknowledged records if the leader dies before followers copy them low
all every in-sync replica has it βœ… not lose acknowledged records, if min.insync.replicas β‰₯ 2 highest (one extra round trip inside the cluster)

acks=all alone is not enough: if the ISR shrinks to just the leader, "all in-sync replicas" is one replica. min.insync.replicas=2 makes the broker refuse writes when fewer than 2 replicas are in sync. With RF = 3 and min ISR = 2, the topic keeps accepting writes with one broker down and never acknowledges a write that exists on only one machine. Keep unclean.leader.election.enable=false (the default) so an out-of-date replica is never elected leader. Since Kafka 3.0, producers default to acks=all with idempotence enabled.

KRaft: Kafka's own consensus

Cluster metadata β€” which topics exist, which broker leads each partition, the ISR β€” is kept by a small quorum of controllers using KRaft, a Raft-based protocol (see Consensus). Older clusters used ZooKeeper; Kafka 4.0 (2025) removed ZooKeeper entirely. The course labs run Kafka in KRaft mode: in lab 7 a single node is both broker and controller.


Retention, compaction and tiered storage

A log cannot grow forever. Each topic has a cleanup policy:

Policy Configuration Keeps BoxOffice use
Delete by time retention.ms (default 7 days) everything newer than the limit order-events: 7 days for replay and the warehouse loader
Delete by size retention.bytes (per partition; default unlimited) the newest N bytes of each partition High-volume click-stream with a disk budget
Compact cleanup.policy=compact at least the latest record of each key event-catalog: the current state of each event (name, venue, date)
Both cleanup.policy=compact,delete latest per key, but not older than the retention Latest seat-map snapshot per show, for 30 days

A compacted topic behaves like a table that is also a stream: a new consumer reads it from the start and ends up with the latest value of every key. Deleting a key is done by writing a tombstone (the key with a null value). Kafka's own __consumer_offsets is a compacted topic.

flowchart LR
    b0["0 Β· ev-88<br/>21:00"]:::problem --- b1["1 Β· ev-90<br/>Jazz"]:::problem --- b2["2 Β· ev-88<br/>20:00"] --- b3["3 Β· ev-91<br/>Theater"] --- b4["4 Β· ev-90<br/>tombstone"]
    b4 ==>|"compaction keeps the latest record of each key"| a2["2 Β· ev-88<br/>20:00"]:::comp
    a2 --- a3["3 Β· ev-91<br/>Theater"]:::comp

Compaction removes records that a newer record of the same key replaced (offsets 0 and 1), and later the tombstone itself (after delete.retention.ms, one day by default). The survivors keep their offsets: a consumer that stored offset 3 is still right.

Tiered storage (production-ready since Kafka 3.9) keeps only recent segments on the brokers' disks and moves older segments to object storage (S3, GCS, Azure Blob). Long retention β€” months of events for replay β€” becomes cheap, and brokers become lighter to replace.


Why Kafka is fast

A Kafka broker routinely moves hundreds of megabytes per second on commodity hardware. The reasons are worth knowing because they reappear in many systems3:

  • Sequential I/O. Appends go to the end of a segment file; reads by caught-up consumers come from the same recent region. Sequential disk access is orders of magnitude faster than random access (see Numbers).
  • The OS page cache. Kafka does not keep its own cache; it lets the operating system cache recent segments in memory, so consumers that keep up read from RAM.
  • Zero-copy. To serve a consumer, the broker asks the kernel to send file bytes straight to the socket (sendfile), without copying them through the application.
  • Batching and compression. Producers group records into batches per partition (linger.ms, batch.size) and compress whole batches (zstd, lz4). Fewer, bigger requests; fewer bytes on disk and on the network.
  • Partitioning. Load spreads across many brokers and many consumers.

The trade-off is latency: batching adds a few milliseconds (linger.ms), and acks=all adds a replication round trip. Typical end-to-end latency is a few to tens of milliseconds β€” fine for events, not a replacement for an in-memory remote procedure call (RPC).


Sizing partitions

Partitions are cheap but not free (more files, more leader elections, longer rebalances, more memory in clients). A common starting rule4: with a target throughput \(T\), a measured per-partition producer throughput \(p\) and a per-partition consumer throughput \(c\),

\[ N_{\text{partitions}} \ge \max\left(\frac{T}{p},\ \frac{T}{c}\right) \]

then round up with headroom for growth, because increasing partitions later moves keys.

BoxOffice: partitions for order-events

Illustrative peak: 30,000 orders in the first 3 minutes of a big on-sale, each producing about 10 events (placed, held, payment authorized, confirmed, ticket issued…): \(T \approx 30\,000 \times 10 / 180 \approx 1{,}700\) events/s. The records are small (~1 KB), so the producer side is trivial: a single partition handles far more than 1.7 MB/s.

The consumer side decides. notification calls the e-mail provider for each OrderConfirmed, ~50 ms per call, processing one record at a time per partition: \(c = 1 / 0.05 = 20\) records/s per partition. For its share of the traffic (the 30,000 OrderConfirmed in 180 s β‰ˆ 167/s):

\[ N \ge \frac{167}{20} \approx 9 \text{ partitions} \]

With headroom for bigger shows and growth, BoxOffice creates order-events with 24 partitions. If one day that is not enough, notification can batch e-mails or process records concurrently per key inside each partition instead of adding partitions.

That is the production cluster (three brokers, replication factor 3). Hands-on 7 runs a single broker on a laptop and creates order-events with 6 partitions and replication factor 1: its load is far smaller, and the same rule sets its ceiling β€” at 20 e-mails/s per partition, 6 partitions deliver at most 120 confirmations/s, and a seventh notification thread would sit idle.

The topic is created explicitly (never rely on auto-creation in production):

kafka-topics.sh --bootstrap-server kafka:9092 --create --topic order-events \
  --partitions 24 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000 \
  --config compression.type=zstd

And the Spring Boot services configure their clients accordingly:

spring:
  kafka:
    bootstrap-servers: kafka:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JacksonJsonSerializer
      acks: all                    # wait for every in-sync replica
      properties:
        enable.idempotence: true   # producer retries do not duplicate records
        linger.ms: 5               # wait up to 5 ms to fill a batch
        compression.type: zstd
spring:
  kafka:
    bootstrap-servers: kafka:9092
    consumer:
      group-id: notification
      auto-offset-reset: earliest  # a brand-new group starts from the oldest record
      enable-auto-commit: false    # the container commits after processing
      max-poll-records: 100
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JacksonJsonDeserializer
      properties:
        spring.json.trusted.packages: "boxoffice.events"
    listener:
      ack-mode: record             # commit the offset after each record
      concurrency: 3               # 3 consumer threads in this instance

In the BoxOffice code: Kafka in Compose (lab 7)

Hands-on 7 β€” Kafka in Compose runs one node that is broker and KRaft controller at once, with auto-creation off and both topics created by a one-shot container: order-events and order-events-dlt with 6 partitions and replication factor 1. One laptop cannot hold three brokers in three zones, so this is the place where the lab is not production: acks: all in order waits for "all in-sync replicas", and with RF = 1 that is one replica β€” exactly the trap described in Replication and durability. The 6 partitions bound the consumer side: notification runs 3 threads (NOTIFY_CONCURRENCY), and more than 6 would sit idle. The lab's burst-and-drain experiment applies this page's arithmetic: one thread at 50 ms per e-mail handles 20/s, so 100 orders/s for 60 s leave about 4,800 messages of lag that take about 240 s to drain; six threads keep the lag near zero.

compose.yaml (one KRaft node, topics created explicitly)
kafka:
  image: apache/kafka:4.3.1
  hostname: kafka
  ports:
    - "9094:9094"
  environment:
    KAFKA_NODE_ID: 1
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
    KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
kafka-init:
  image: apache/kafka:4.3.1
  depends_on:
    kafka:
      condition: service_healthy
  entrypoint: ["/bin/sh", "-c"]
  command:
    - |
      /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
        --topic order-events --partitions 6 --replication-factor 1 &&
      /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
        --topic order-events-dlt --partitions 6 --replication-factor 1
  restart: "no"

Kafka and its alternatives

Apache Kafka Amazon Kinesis Data Streams Google Cloud Pub/Sub Apache Pulsar
Model partitioned replicated log partitioned log (shards) pub/sub with per-message ack pub/sub + log (segments in BookKeeper)
Ordering per partition per shard per ordering key (optional) per partition / key
Replay yes, any offset in retention yes, within retention (up to 365 days) seek to a time or snapshot yes
Operations self-managed or managed (Amazon MSK, Confluent Cloud, Aiven) fully managed fully managed, global self-managed or managed
Good fit event backbone, change data capture (CDC), stream processing, very high throughput AWS-native streaming Google Cloud-native fan-out, push to serverless multi-tenant messaging and streaming

Redpanda and Azure Event Hubs expose the Kafka protocol, so Kafka clients work against them unchanged. Kafka itself is also growing queue semantics: share groups ("Queues for Kafka", KIP-932, introduced in Kafka 4.0) let several consumers share a partition with per-record acknowledgement, like a classic queue.

In the interview

When you put Kafka on the board, the interviewer expects four decisions, said out loud: (1) the topic and the key ("order-events, keyed by orderId, because we need per-order ordering and an even spread"); (2) partitions from a throughput estimate with the consumer as the usual bottleneck; (3) durability (RF 3 across zones, acks=all, min.insync.replicas=2); (4) retention (7 days to allow replay, or compaction for state). Mention consumer lag as the metric you alert on. Be ready for "what happens if a broker dies?" (leader election from the ISR, producers retry, no acknowledged data lost) and "how do you scale consumers?" (up to the number of partitions).

Best practices

  • Create every topic explicitly, with partitions, replication and retention reviewed like code β€” auto-creation makes topics with broker defaults nobody chose. In BoxOffice: KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false" and a kafka-init container (Compose box).
  • Place the three replicas of each partition in three availability zones (broker.rack) β€” losing one zone leaves 2 replicas in sync, so min.insync.replicas=2 keeps accepting writes.
  • Set retention by the replay you need, then size the disk β€” at least the longest consumer outage you plan to recover from, plus a margin (7 days for order-events); disk β‰ˆ daily volume Γ— retention days Γ— replication factor.
  • Watch lag per partition, not only per group β€” one hot key (60% of the purchases are for ev-1) hides inside a healthy group total. In BoxOffice: lab 7's kafka-lag.sh prints the group total; kafka-consumer-groups.sh --describe shows each partition (lab 7 β€” measure).
  • Batch and compress on the producer β€” linger.ms of a few milliseconds and compression.type=zstd cut requests and bytes; 5 ms is invisible next to a 300 ms SLO. In BoxOffice: order sets linger.ms: 5, acks: all and enable.idempotence: true.
  • Use incremental (cooperative) rebalancing for consumer groups β€” with eager rebalancing, a rolling deploy of 6 consumers stops the whole group at every restart.

Common pitfalls

  • A low-cardinality key (country, show, true/false) creates hot partitions that no number of consumers can fix.
  • Too few partitions: the group cannot scale beyond them, and adding partitions later remaps keys.
  • acks=1 or min.insync.replicas=1 for business events: "acknowledged" does not mean "safe".
  • Processing a partition with a thread pool without keying the work: ordering is silently lost.
  • Slow processing inside the poll loop beyond max.poll.interval.ms: the consumer is kicked out, the group rebalances, another consumer gets the same records β€” a rebalance storm with duplicates.
  • Using Kafka as a database for queries by arbitrary fields. Kafka is read sequentially by offset; build query models in a database from it (Command Query Responsibility Segregation, CQRS).

How it connects

flowchart LR
    LOG["append-only log<br/>offsets"] --> PART["partitions<br/>unit of parallelism"]
    KEY["key = orderId<br/>murmur2 mod N"] --> PART
    PART --> ORD["order per key<br/>β†’ Delivery semantics"]
    PART --> CG["consumer groups<br/>one consumer per partition"]
    CG --> LAG["consumer lag<br/>β†’ Observability"]
    PART --> REP["RF 3, acks=all,<br/>min.insync.replicas=2"]
    REP --> KRAFT["KRaft controllers<br/>β†’ Consensus (class 3)"]
    LOG --> RET["retention, compaction,<br/>tiered storage"]
    LOG --> REPLAY["replay: CDC, event sourcing<br/>β†’ Patterns"]
    PART --> SIZE["N β‰₯ max(T/p, T/c)<br/>β†’ Estimation"]
    KEY --> HOT["hot key, hash mod N moves keys<br/>β†’ Databases partitioning"]

Palette components: kafka Β· stream (the managed equivalent: Kinesis, Event Hubs) Β· objstore (tiered storage) Β· dwh (the warehouse loader replays the log) Β· notifications Β· analytics Β· search (consumer groups reading at their own pace). Scenarios: event-fanout. Continues in: Delivery semantics, Patterns, Distributed β€” consensus (KRaft is Raft), Databases β€” partitioning (hot keys, resharding), Observability β€” signals (lag as a metric), Hands-on 7 β€” Kafka.



  1. KREPS, J.; NARKHEDE, N.; RAO, J. Kafka: a Distributed Messaging System for Log Processing. In: NetDB Workshop, 2011. ↩

  2. KREPS, J. The Log: What every software engineer should know about real-time data's unifying abstraction. LinkedIn Engineering, 2013. ↩

  3. Apache Kafka Documentation β€” Design. Apache Software Foundation. ↩

  4. SHAPIRA, G.; PALINO, T.; SIVARAM, R.; PETTY, K. Kafka: The Definitive Guide, 2nd ed. O'Reilly, 2021. Chapters on producers, consumers and reliable delivery. ↩