Platforms, Microservices, DevOps and APIs · Hands-on 10

Hands-on 10 — Kafka Messaging

A producer appends to a log and answers; a consumer reads it on its own time — first the reference pair, then the store

Should the email block the customer?

Every call the order-service makes happens before it answers. Now it must send a confirmation email. One more REST call?

REST
POST /orders
→
account, product
→
email serviceslow or down: the order waits or fails
→
201
Kafka
POST /orders
→
account, product
→
append to a topica durable log
→
201 at once
later, on its own time: the notification-service reads the topic and sends the email

Two steps: the reference, then the store

Reference
GET /producer/{msg}
→
producer-service
send →
store-producer-exampleno key
→
consumer-servicegroup store
Store
POST /orders
→
order-serviceafter saving
send →
order-eventskey = order id
→
notification-servicegroup notification-group
  • Same pattern twice: two Spring Boot services that never call each other.
  • The producer does not know who reads; a new reader (analytics) is a new group, not a code change.

A topic is a log

partition 0
0hello
1world
2a
3b
4c
5next write
read and committed by group store
waiting: lag = 5 − 2 = 3
log-end offset 5
  • The producer appends; a record gets the next offset and stays — reading does not delete it.
  • Kafka keeps, per group and partition, the committed offset: the next record that group will read.
  • Lag = log-end offset − committed offset. Another group reads the same log from its own offset.

2 · Kafka in Docker Compose

Docker network
producer · consumerbootstrap-servers: kafka:9092
PLAINTEXT →
kafka · one node, KRaftbroker + controller · :9092 clients · :9093 controller quorum
no ports →
your machinetools run inside the container
  • KRaft: Kafka keeps its metadata on its own Raft quorum; Kafka 4.0 removed ZooKeeper.
  • ADVERTISED_LISTENERS: kafka:9092 — the service name; in a container, localhost is the container itself.
  • A topic is created on first use with KAFKA_NUM_PARTITIONS: 1; one broker → replication factor 1.

The reference uses apache/kafka:latest (4.3.1 when written); pin it for reproducible builds.

3 · The producer-service

GET /producer/hello
→
ProducerResource
→
ProducerServiceMap → JSON (ObjectMapper)
→
kafkaTemplate.send(topic, json) · no key
→
200handed to the client buffer
Map<String, String> map = Map.of("id", id, "message", msg);
kafkaTemplate.send("store-producer-example", objectMapper.writeValueAsString(map));
  • KafkaProducerConfig: the broker address + StringSerializer for key and value — nothing else.
  • send is asynchronous: 200 = "handed to the Kafka client", not "stored".
  • Since Kafka 3.0 the defaults are safe: acks=all and idempotence on.

3.1 · Through the gateway

curl
→
lb :8080
→
gatewayroute Path=/producer/**
→
RouterValidator"GET /producer/**" is open: no cookie
→
producer:8080
  • Without the route: 404. Without the open entry: 401 (the route needs a login).

An open GET that publishes is a demo only: anyone can write to the topic, and a crawler or a prefetch would publish too. In the store the event comes out of POST /orders, a secured route.

4 · The consumer-service

listener containerpolls store-producer-example
→
@KafkaListenercalled once per record
→
all records of the poll done
→
commit the offsetsAckMode.BATCH
↺
@KafkaListener(topics = "store-producer-example", groupId = "store")
public void consumerMsg(String message) { … logger.debug("enqueue {}", map); }
  • The method never calls poll() or commit(): the container does, with enable.auto.commit=false.
  • A crash before the commit → those records come again: at-least-once (section 8).

pom.xml is the producer's with artifactId consumer-service — keep the old one and the jar gets the producer's name.

5 · Run the reference

cd api/producer-service && mvn clean package -DskipTests && cd ../..
cd api/consumer-service && mvn clean package -DskipTests && cd ../..
cd api && docker compose up -d --build
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/producer/hello     # 200
producer-1queue {id=e295…, message=hello}
→
store-producer-examplepartition 0 · offset 0
→
consumer-1enqueue {id=e295…, message=hello}
Tool (inside the kafka container) Shows
kafka-topics.sh --describe --topic … partitions, leader, replicas
kafka-console-consumer.sh --from-beginning the records, still there after being read
kafka-consumer-groups.sh --describe --group store committed offset, log-end offset, lag

5.2 · The consumer was down

docker compose stop consumer
→
3 × curl /producer/…200, 200, 200
→
--describeno active members · lag 3
→
start consumerresumes at offset 2: a, b, c · lag 0
  • The producer did not notice: it answered 200 three times.
  • With REST, those three calls would have failed; with a log in between, the consumer catches up.

Asynchronous messaging turns "the other service is down" from an error into a delay.

6 · In the store: order-events

POST /orders
→
account, productREST, synchronous
→
save the order
→
send(order-events, key = order id)
→
201
next poll
→
notification-servicesending confirmation email: order …
  • Publish after saving: never announce an order that does not exist.
  • The order-service never calls the notification-service, and does not wait for the email.

6 · Same pattern, three changes

reference
store
producer
ProducerServicesend(topic, json)
OrderEventPublishersend("order-events", id, json): with a key
config
KafkaProducerConfig
the same class, in package store.order
consumer
consumer-service
notification-servicenew package, group notification-group, topic order-events
message
{id, message}
{id, idAccount, total}
  • Test: log in, POST /orders → 201 at once → the email line in docker compose logs notification.

7 · Partitions: the key picks one

order-servicekey = order id
murmur2(key) % 3 →
order-events · 3 partitions
P0events of some orders, in order
P1…
P2…
→
notification-group
notification-1reads P0, P1
notification-2reads P2
  • Same key → same partition → the events of one order stay in order.
  • In a group, each partition has one reader; joining or leaving → rebalance, resuming at the committed offset.
  • Kafka can add partitions (--alter --partitions 3), never remove them.

7 · How many instances help?

1 instance
2
3
4
1 partition (the lab)
reads P0
1 idle
2 idle
3 idle
3 partitions
reads all 3
P0, P1 · P2
one each
1 idle
  • The number of partitions caps the parallelism of a group.
  • docker compose up -d --scale notification=2 + kafka-consumer-groups --describe --members: #PARTITIONS 1 and 0.
  • Another group (analytics) reads every message, at its own offsets.

7 · Without a key, partitions do not spread

9 × curl /producer/…no key
sticky partitioner →
store-producer-example · 3 partitions
P0nothing new
P1all nine: offsets 0 to 8
P2nothing
→
group store
consumer-1reads P0, P1: does all the work
consumer-2reads P2: idle
  • No key → the producer fills one partition before moving on (KIP-794, a Kafka Improvement Proposal): big batches, no spreading.
  • A key spreads the load and keeps each key in order: that is why order-events uses the order id.

Simulator: partitions and groups

The store's part of the lab without Docker: order-events with 1 partition, one notification-service. Play runs a page; the buttons work at any time.

6 orders, 1 partitionall in P0 · notification-2 gets nothing
→
--partitions 3notification-1 ← P0, P1 · notification-2 ← P2
→
6 more orderssplit 2, 2 and 2 by key
  • Stop one instance: its partition moves to the other, which drains the lag one message per poll.
  • Free mode: 4 instances on 3 partitions — one sits idle.

Page section 7 · simulator order-events, script "7 · Partitions and groups".

8.1 · At-least-once

record at offset 0
→
email sent
→
killed before the commit
→
restart / rebalanceoffset 0 again
→
the same email twice
  • The container commits after the listener returns: a crash in between means redelivery.
  • Kafka delivers at least once — the consumer must be idempotent.

Record the notified order ids in a durable store (INSERT … ON CONFLICT DO NOTHING) and email only when the insert added a row. A set in memory dies in the very crash that causes the redelivery.

8.2 · A message that cannot be processed

echo 'oops' | console producernot JSON
→
readValue throws10 attempts, no pause
→
ERROR · Backoff … exhaustedfor order-events-0@1
→
commit past itthe message is gone
  • No error handler of your own → Spring Kafka's DefaultErrorHandler: FixedBackOff(0, 9), log, skip.
  • Nothing blocks — the next orders flow, lag back to 0 — but one log line is all that is left.

For an order whose event had a bug, that is a customer who never gets the email — and nobody notices.

8.3 · A dead-letter topic (DLT)

bad record
→
3 attempts, 1 s apartRecord in retry and not yet recovered
→
DeadLetterPublishingRecoverer
→
order-events-dltoffset 0 · oops · inspect, fix, replay
new DefaultErrorHandler(new DeadLetterPublishingRecoverer(kafkaTemplate(),
    (record, ex) -> new TopicPartition(record.topic() + "-dlt", -1)), new FixedBackOff(1000L, 2L))
  • Default destination: <topic>-dlt, same partition → the DLT would need as many partitions; -1 lets Kafka choose.
  • The DLT record carries headers: original topic, partition, offset, exception.

Simulator: failures

duplicates
kill before the commit1 duplicate email
→
idempotent consumerthe repeat is skipped
bad message
default handler10 attempts, skipped · 1 lost · lag 0
→
dead-letter topic3 attempts → order-events-dlt · 0 lost
  • The tiles count it: emails sent, duplicate emails, group lag, messages lost, order-events-dlt.
  • One record per poll: each step of the page is one click.

Page section 8 · simulator order-events, script "8 · Failures".

Pitfalls

Symptom Cause
401 / 404 on /producer/hello route not open in the RouterValidator / missing in the gateway
COPY target/*.jar fails mvn clean package not run in that service
consumer jar named producer-service-1.0.0.jar copied pom.xml still has the producer's artifactId
a second instance never works 1 partition — or no key: everything in one partition
an email sent twice crash before the commit: make the consumer idempotent
a bad event vanished default error handler skipped it: add a dead-letter topic

Takeaways

  1. A topic is a log: the producer appends and answers; consumers read at their own offsets.
  2. Spring Kafka: KafkaTemplate.send to publish, @KafkaListener + a group to consume; the container commits.
  3. A consumer that is down means lag, not errors: it catches up.
  4. Partitions cap a group's parallelism; a key spreads the load and keeps each key in order.
  5. Delivery is at-least-once: idempotent consumers, with a durable record.
  6. Without a dead-letter topic, a bad message is retried 10 times and lost.

Outline: (1) why not one more REST call; (2) Kafka in KRaft mode in compose; (3) the producer-service and its gateway route; (4) the consumer-service; (5) run it, look inside Kafka, stop the consumer; (6) the store: order-events and the notification-service; (7) partitions, groups, keys; (8) duplicates, bad messages, the dead-letter topic; the simulator.

The page's table "Reference / In the store" lists the same pieces: producer, topic, key, consumer, message.

Measured in the lab's stack: after --partitions 3 and a restart, nine curl messages all went to partition 1.