Skip to content

10. Kafka Messaging

Goal

Add asynchronous messaging to the platform with Apache Kafka. First the reference pair of the class: a producer-service that publishes what it receives on GET /producer/{msg} to the topic store-producer-example, and a consumer-service that reads that topic β€” two Spring Boot services that never call each other. Then the same pattern in the store: the order-service publishes an OrderCreated event to order-events, and a notification-service consumes it and sends the confirmation email, without the order-service knowing it exists.

The platform uses synchronous REST calls for every interaction between services. Every call the order-service makes β€” to account-service, to product-service β€” happens inline, before the response goes back to the client. That fits operations whose caller needs the result now. Others are fire-and-forget: sending the confirmation email of an order does not need to block the customer. Apache Kafka is the messaging backbone for these cases: a producer appends a message to a topic, a durable log, and answers at once; the consumers read the log on their own time.

Slides

Click the slides and use ← β†’ to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

1. Architecture

flowchart LR
    client[Client] -->|"GET /producer/{msg}"| gw[Gateway]
    gw -->|REST| producer[producer-service]
    producer -->|"send\nstore-producer-example"| kafka[(Kafka)]
    kafka -->|"@KafkaListener\ngroup store"| consumer[consumer-service]
flowchart LR
    client[Client] -->|"POST /orders"| gw[Gateway]
    gw -->|REST| order[order-service]
    order -->|REST sync| account[account-service]
    order -->|REST sync| product[product-service]
    order -->|"Kafka (async)\norder-events"| kafka[(Kafka)]
    kafka -->|"Kafka (async)\nnotification-group"| notification[notification-service]

Decoupling

The producer publishes to a topic; it has no knowledge of who reads it. The producer-service never calls the consumer-service, and the order-service never calls the notification-service. Tomorrow an analytics-service can read order-events too, in a group of its own, without touching a single line of the order-service.

Piece Reference (sections 2–5) In the store (sections 6–8)
Producer producer-service, GET /producer/{msg} order-service, after saving an order
Topic store-producer-example order-events
Key none the order id
Consumer consumer-service, group store notification-service, group notification-group
Message {"id": "…", "message": "…"} {"id": "…", "idAccount": "…", "total": "…"}

2. Kafka in Docker Compose

Add the broker and the two reference services to the compose.yaml of the platform (the api/ folder, next to the other services). A single Kafka 4 node runs in KRaft mode β€” no ZooKeeper container.

compose.yaml (additions)
# add to the services of your compose.yaml
  kafka:
    hostname: kafka
    image: apache/kafka:latest
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_NUM_PARTITIONS: 1

  producer:
    build:
      context: ./producer-service
      dockerfile: Dockerfile
    deploy:
      replicas: 1

  consumer:
    build:
      context: ./consumer-service
      dockerfile: Dockerfile
    deploy:
      replicas: 1

KRaft β€” no ZooKeeper

Kafka keeps its cluster metadata (topics, partitions, leaders) in its own Raft-based controller quorum, KRaft. Before, this metadata lived in a separate ZooKeeper ensemble; Kafka 4.0 (2025) removed ZooKeeper support entirely. Here the single node is both broker and controller (KAFKA_PROCESS_ROLES: broker,controller): clients talk to the PLAINTEXT listener on kafka:9092, and the controller quorum uses the CONTROLLER listener on port 9093. See Event-Driven Architecture.

  • KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 is the address the broker hands to clients: the service name inside the Docker network. Inside a container, localhost:9092 is the container itself, not the broker β€” and the broker publishes no port to your machine, so every Kafka command below runs inside the kafka container.
  • A single broker cannot keep more than one copy of the internal topics: hence the three …_REPLICATION_FACTOR: 1 / MIN_ISR: 1.
  • A topic is created on first use (the broker's auto.create.topics.enable is true by default) with KAFKA_NUM_PARTITIONS: 1 partition. Section 7 adds partitions.
  • KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 lets a consumer group start at once instead of waiting 3 s for more members β€” convenient in development.

Pin the image

apache/kafka:latest follows the newest release (Kafka 4.3.1 when this page was written). For a reproducible build, pin the version: image: apache/kafka:4.3.1.


3. The producer-service

A Spring Boot service with two endpoints: GET /producer/health-check answers 200 for the orchestrator, and GET /producer/{msg} wraps the text in a small JSON with a random id and publishes it to store-producer-example.

api/
    producer-service/
        src/main/java/store/kafka/producer/
            KafkaProducerConfig.java
            ProducerResource.java
            ProducerService.java
            ProducerServiceApplication.java
        src/main/resources/application.yaml
        Dockerfile
        pom.xml
producer-service/src/main/java/store/kafka/producer/ProducerService.java
package store.kafka.producer;

import java.util.Map;
import java.util.UUID;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

import tools.jackson.databind.ObjectMapper;

@Service
public class ProducerService {

    private static Logger logger = LoggerFactory.getLogger(ProducerService.class);

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    private final ObjectMapper objectMapper = new ObjectMapper();

    public void sendMessage(String msg) {
        final String id = UUID.randomUUID().toString();
        Map<String, String> map = Map.of(
            "id", String.valueOf(id),
            "message", msg
        );
        logger.debug("queue {}", map);
        kafkaTemplate.send("store-producer-example", objectMapper.writeValueAsString(map));
    }

}
producer-service/src/main/java/store/kafka/producer/KafkaProducerConfig.java
package store.kafka.producer;

import java.util.HashMap;
import java.util.Map;

import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;

@Configuration
public class KafkaProducerConfig {

    @Value(value = "${spring.kafka.bootstrap-servers}")
    private String bootstrapAddress;

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        Map<String, Object> configProps = new HashMap<>();
        configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,  bootstrapAddress);
        configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return new DefaultKafkaProducerFactory<>(configProps);
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

}
producer-service/src/main/java/store/kafka/producer/ProducerResource.java
package store.kafka.producer;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;

@Controller
public class ProducerResource {

    @Autowired
    private ProducerService producerService;

    @GetMapping("/producer/{msg}")
    public ResponseEntity<Void> create(
        @PathVariable String msg
    ) {
        producerService.sendMessage(msg);
        return ResponseEntity.ok().build();
    }

    // a literal path wins over /producer/{msg}, so this never sends a message
    @GetMapping("/producer/health-check")
    public ResponseEntity<Void> healthCheck() {
        return ResponseEntity.ok().build();
    }

}
producer-service/src/main/java/store/kafka/producer/ProducerServiceApplication.java
package store.kafka.producer;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ProducerServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(ProducerServiceApplication.class, args);
    }

}
producer-service/src/main/resources/application.yaml
spring:

  application:
    name: producer

  kafka:
    bootstrap-servers: kafka:9092

logging:
  level:
    store: debug
producer-service/pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.1.1</version>
    </parent>
    <groupId>store</groupId>
    <artifactId>producer-service</artifactId>
    <version>1.0.0</version>

    <properties>
        <java.version>25</java.version>
        <spring-cloud.version>2025.1.3</spring-cloud.version>
        <maven.compiler.proc>full</maven.compiler.proc>
        <lombok.version>1.18.48</lombok.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webmvc</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>

        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>

    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <id>default-compile</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <configuration>
                            <annotationProcessorPaths>
                                <path>
                                    <groupId>org.projectlombok</groupId>
                                    <artifactId>lombok</artifactId>
                                </path>
                            </annotationProcessorPaths>
                        </configuration>
                    </execution>
                    <execution>
                        <id>default-testCompile</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <annotationProcessorPaths>
                                <path>
                                    <groupId>org.projectlombok</groupId>
                                    <artifactId>lombok</artifactId>
                                </path>
                            </annotationProcessorPaths>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>
producer-service/Dockerfile
1
2
3
4
FROM eclipse-temurin:25
VOLUME /tmp
COPY target/*.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
  • KafkaProducerConfig builds the ProducerFactory (where the broker is, how to turn key and value into bytes: StringSerializer for both) and the KafkaTemplate, the object the code sends with.
  • ProducerService turns the map into JSON with Jackson's ObjectMapper and calls kafkaTemplate.send(topic, value). There is no key: section 7 shows what that means for partitions.
  • send is asynchronous: it hands the record to the client's buffer and returns a CompletableFuture; the 200 of the endpoint means "handed to the Kafka client", not "stored in Kafka". Since Kafka 3.0 the producer's defaults are already the safe ones β€” acks=all (the leader waits for every in-sync replica) and idempotence on (a retry never writes a duplicate) β€” so the factory sets only the address and the serializers.
sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant P as producer-service
    participant K as Kafka (store-producer-example)
    participant S as consumer-service
    C->>G: GET /producer/hello
    G->>P: GET /producer/hello (open route, no cookie)
    P->>K: send(store-producer-example, {"id": "…", "message": "hello"})
    P-->>C: 200 OK (does not wait for any consumer)
    K->>S: the record, on the next poll
    S->>S: log enqueue {…}
    S->>K: commit the offset

3.1 Through the gateway

Add the route to the gateway's application.yaml, next to the others:

gateway-service/src/main/resources/application.yaml
            - id: producer-service
              uri: http://producer:8080
              predicates:
                - Path=/producer/**

and open it in the RouterValidator (see Hands-on 3), so that it needs no login:

gateway-service/src/main/java/store/gateway/security/RouterValidator.java
    private List<String> openApiEndpoints = List.of(
            "GET /producer/**",
            "POST /auth/register",
            "GET /auth/logout",
            "POST /auth/login"
        );

An open route is a demo

GET /producer/** lets anyone publish to the topic β€” fine for the reference, never for real events. And a GET that changes state breaks HTTP semantics: a crawler or a browser prefetch would publish messages. In the store, the event comes out of POST /orders, a secured route.


4. The consumer-service

api/
    consumer-service/
        src/main/java/store/kafka/consumer/
            ConsumerResource.java
            ConsumerService.java
            ConsumerServiceApplication.java
            KafkaConsumerConfig.java
        src/main/resources/application.yaml
        Dockerfile
        pom.xml
consumer-service/src/main/java/store/kafka/consumer/ConsumerService.java
package store.kafka.consumer;

import java.util.HashMap;
import java.util.Map;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

import tools.jackson.core.type.TypeReference;
import tools.jackson.databind.ObjectMapper;

@Service
public class ConsumerService {

    private static Logger logger = LoggerFactory.getLogger(ConsumerService.class);

    private final ObjectMapper objectMapper = new ObjectMapper();

    @KafkaListener(topics = "store-producer-example", groupId = "store")
    public void consumerMsg(String message) {
        Map<String, String> map = objectMapper.readValue(message, new TypeReference<HashMap<String, String>>() {});
        logger.debug("enqueue {}", map);
    }

}
consumer-service/src/main/java/store/kafka/consumer/ConsumerResource.java
package store.kafka.consumer;

import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;

@Controller
public class ConsumerResource {

    // the consumer has no route in the gateway: this is for the orchestrator
    // (compose healthcheck, Kubernetes probe), called inside the network
    @GetMapping("/consumer/health-check")
    public ResponseEntity<Void> healthCheck() {
        return ResponseEntity.ok().build();
    }

}
consumer-service/src/main/java/store/kafka/consumer/KafkaConsumerConfig.java
package store.kafka.consumer;

import java.util.HashMap;
import java.util.Map;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;

@EnableKafka
@Configuration
public class KafkaConsumerConfig {

    @Value(value = "${spring.kafka.bootstrap-servers}")
    private String bootstrapAddress;

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "store");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props);
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }

}
consumer-service/src/main/java/store/kafka/consumer/ConsumerServiceApplication.java
package store.kafka.consumer;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ConsumerServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(ConsumerServiceApplication.class, args);
    }

}
consumer-service/src/main/resources/application.yaml
spring:

  application:
    name: consumer

  kafka:
    bootstrap-servers: kafka:9092

logging:
  level:
    store: debug
consumer-service/pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.1.1</version>
    </parent>
    <groupId>store</groupId>
    <artifactId>consumer-service</artifactId>
    <version>1.0.0</version>

    <properties>
        <java.version>25</java.version>
        <spring-cloud.version>2025.1.3</spring-cloud.version>
        <maven.compiler.proc>full</maven.compiler.proc>
        <lombok.version>1.18.48</lombok.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webmvc</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>

        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>

    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <id>default-compile</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <configuration>
                            <annotationProcessorPaths>
                                <path>
                                    <groupId>org.projectlombok</groupId>
                                    <artifactId>lombok</artifactId>
                                </path>
                            </annotationProcessorPaths>
                        </configuration>
                    </execution>
                    <execution>
                        <id>default-testCompile</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <annotationProcessorPaths>
                                <path>
                                    <groupId>org.projectlombok</groupId>
                                    <artifactId>lombok</artifactId>
                                </path>
                            </annotationProcessorPaths>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>
consumer-service/Dockerfile
1
2
3
4
FROM eclipse-temurin:25
VOLUME /tmp
COPY target/*.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
  • @KafkaListener(topics = "store-producer-example", groupId = "store"): Spring runs a listener container that polls the topic and calls the method once per record. The method never calls poll() nor commit().
  • The group (store) is the consumer's identity in Kafka: the broker keeps, per group and per partition, the committed offset β€” the next record the group will read. Another group reading the same topic gets every message again, at its own offsets.
  • When the offset is committed: the container sets enable.auto.commit=false and, by default (AckMode.BATCH), commits after the method has returned for every record of a poll. A crash in the middle means those records come again: at-least-once (section 8).
  • pom.xml is the producer's with <artifactId>consumer-service</artifactId>: a copy that keeps producer-service builds a jar with the wrong name.

5. Run the reference

Build both jars (the Dockerfile copies target/*.jar), then start the stack:

cd api/producer-service && mvn clean package -DskipTests && cd ../..
cd api/consumer-service && mvn clean package -DskipTests && cd ../..
cd api
docker compose up -d --build

Send two messages through the gateway (the load balancer on port 8080):

$ curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/producer/hello
200
$ curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/producer/world
200

You should see each message twice β€” published by the producer, read by the consumer:

$ docker compose logs producer | grep queue
producer-1  | … DEBUG 1 --- [producer] [nio-8080-exec-1] store.kafka.producer.ProducerService     : queue {id=e29517a4-e00a-4ff6-9d0b-b64825763623, message=hello}
producer-1  | … DEBUG 1 --- [producer] [nio-8080-exec-3] store.kafka.producer.ProducerService     : queue {id=139b7a01-76ee-4fc4-9b65-a15467e35c81, message=world}
$ docker compose logs consumer | grep -E "assigned|enqueue"
consumer-1  | … INFO 1 --- [consumer] [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : store: partitions assigned: [store-producer-example-0]
consumer-1  | … DEBUG 1 --- [consumer] [ntainer#0-0-C-1] store.kafka.consumer.ConsumerService     : enqueue {id=e29517a4-e00a-4ff6-9d0b-b64825763623, message=hello}
consumer-1  | … DEBUG 1 --- [consumer] [ntainer#0-0-C-1] store.kafka.consumer.ConsumerService     : enqueue {id=139b7a01-76ee-4fc4-9b65-a15467e35c81, message=world}

(The order of id and message in the producer's line may differ: Map.of has no fixed order.)

5.1 Look inside Kafka

The broker ships its command-line tools in /opt/kafka/bin. The topic, created on first use with one partition:

$ docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --describe --topic store-producer-example
Topic: store-producer-example   TopicId: FMKKtZWTTTWB-zZHi-aYOA PartitionCount: 1   ReplicationFactor: 1    Configs: min.insync.replicas=1
    Topic: store-producer-example   Partition: 0    Leader: 1   Replicas: 1 Isr: 1  Elr:    LastKnownElr: 

The messages are still there after being read β€” a consumer only moves its offset:

$ docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic store-producer-example --from-beginning --max-messages 2 --formatter-property print.partition=true --formatter-property print.offset=true
Partition:0 Offset:0    {"id":"e29517a4-e00a-4ff6-9d0b-b64825763623","message":"hello"}
Partition:0 Offset:1    {"id":"139b7a01-76ee-4fc4-9b65-a15467e35c81","message":"world"}
Processed a total of 2 messages

And the group: where store is, per partition. Lag is the log-end offset minus the committed offset β€” the messages waiting.

docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group store

5.2 The consumer was down

Stop the consumer, publish three messages and look at the group:

$ docker compose stop consumer
$ for m in a b c; do curl -s -o /dev/null http://localhost:8080/producer/$m; done
$ docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group store
Consumer group 'store' has no active members.

GROUP  TOPIC                  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID
store  store-producer-example 0          2               5               3    -            -     -

The producer did not notice β€” it answered 200 three times. Start the consumer again: it resumes at offset 2 and logs enqueue for a, b and c; --describe shows lag 0. With REST, those three calls would have failed; with a log in between, the consumer simply catches up.

Troubleshooting

Symptom Cause and fix
Connection to node -1 (kafka/…:9092) could not be established in the first seconds the broker is still starting; the client retries on its own
{store-producer-example=UNKNOWN_TOPIC_OR_PARTITION} once, at start the topic does not exist yet; it is created on that first request
401 on /producer/hello the route is missing from openApiEndpoints in the RouterValidator
404 on /producer/hello the gateway has no producer-service route, or the gateway was not rebuilt
COPY target/*.jar fails in the build run mvn clean package in that service first
the consumer jar is called producer-service-1.0.0.jar the consumer's pom.xml still has the producer's artifactId

6. In the store: order-events

The reference shows the mechanics; now the platform uses them. When an order is saved, the order-service publishes OrderCreated to order-events; a new notification-service, written exactly like the consumer-service, sends the confirmation email (here: a log line).

6.1 The order-service publishes

Add spring-kafka to the order-service's pom.xml and the broker to its application.yaml:

order-service/pom.xml
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
order-service/src/main/resources/application.yaml
spring:
  kafka:
    bootstrap-servers: kafka:9092

Then two classes in the order-service's package β€” the producer's KafkaProducerConfig unchanged (only the package), and a publisher like the ProducerService, now with a key:

order-service/src/main/java/store/order/OrderEventPublisher.java
package store.order;

import java.util.Map;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

import tools.jackson.databind.ObjectMapper;

@Service
public class OrderEventPublisher {

    private static Logger logger = LoggerFactory.getLogger(OrderEventPublisher.class);

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    private final ObjectMapper objectMapper = new ObjectMapper();

    // call it after the order is saved: never announce an order that does not exist
    public void orderCreated(String id, String idAccount, Double total) {
        Map<String, String> event = Map.of(
            "id", id,
            "idAccount", idAccount,
            "total", String.valueOf(total)
        );
        logger.debug("order-events {}", event);
        // the key is the order id: every event of an order goes to the same partition, in order
        kafkaTemplate.send("order-events", id, objectMapper.writeValueAsString(event));
    }

}
order-service/src/main/java/store/order/KafkaProducerConfig.java
package store.order;

import java.util.HashMap;
import java.util.Map;

import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;

@Configuration
public class KafkaProducerConfig {

    @Value(value = "${spring.kafka.bootstrap-servers}")
    private String bootstrapAddress;

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        Map<String, Object> configProps = new HashMap<>();
        configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,  bootstrapAddress);
        configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return new DefaultKafkaProducerFactory<>(configProps);
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }

}

In OrderService, inject the publisher and call it after the order is saved β€” never announce an order that does not exist (adapt the accessors to your Order):

order-service: OrderService.java
@Autowired
private OrderEventPublisher orderEventPublisher;

// in create(), after saving:
orderEventPublisher.orderCreated(order.id(), idAccount, order.total());

6.2 The notification-service

A copy of the consumer-service with four changes: the package, the health-check path (/notification/health-check), the group (notification-group) and the listener, which reads order-events.

api/
    notification-service/
        src/main/java/store/notification/
            KafkaConsumerConfig.java
            NotificationResource.java
            NotificationService.java
            NotificationServiceApplication.java
        src/main/resources/application.yaml
        Dockerfile
        pom.xml
notification-service/src/main/java/store/notification/NotificationService.java
package store.notification;

import java.util.HashMap;
import java.util.Map;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

import tools.jackson.core.type.TypeReference;
import tools.jackson.databind.ObjectMapper;

@Service
public class NotificationService {

    private static Logger logger = LoggerFactory.getLogger(NotificationService.class);

    private final ObjectMapper objectMapper = new ObjectMapper();

    @KafkaListener(topics = "order-events", groupId = "notification-group")
    public void orderCreated(String message) {
        Map<String, String> event = objectMapper.readValue(message, new TypeReference<HashMap<String, String>>() {});
        logger.info("sending confirmation email: order {} (account {}, total {})",
            event.get("id"), event.get("idAccount"), event.get("total"));
    }

}
notification-service/src/main/java/store/notification/NotificationResource.java
package store.notification;

import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;

@Controller
public class NotificationResource {

    // the notification has no route in the gateway: this is for the orchestrator
    // (compose healthcheck, Kubernetes probe), called inside the network
    @GetMapping("/notification/health-check")
    public ResponseEntity<Void> healthCheck() {
        return ResponseEntity.ok().build();
    }

}
notification-service/src/main/java/store/notification/KafkaConsumerConfig.java
package store.notification;

import java.util.HashMap;
import java.util.Map;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;

@EnableKafka
@Configuration
public class KafkaConsumerConfig {

    @Value(value = "${spring.kafka.bootstrap-servers}")
    private String bootstrapAddress;

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "notification-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props);
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }

}
notification-service/src/main/java/store/notification/NotificationServiceApplication.java
package store.notification;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class NotificationServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(NotificationServiceApplication.class, args);
    }

}
notification-service/src/main/resources/application.yaml
spring:

  application:
    name: notification

  kafka:
    bootstrap-servers: kafka:9092

logging:
  level:
    store: debug
notification-service/pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.1.1</version>
    </parent>
    <groupId>store</groupId>
    <artifactId>notification-service</artifactId>
    <version>1.0.0</version>

    <properties>
        <java.version>25</java.version>
        <spring-cloud.version>2025.1.3</spring-cloud.version>
        <maven.compiler.proc>full</maven.compiler.proc>
        <lombok.version>1.18.48</lombok.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webmvc</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>

        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>

    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <id>default-compile</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <configuration>
                            <annotationProcessorPaths>
                                <path>
                                    <groupId>org.projectlombok</groupId>
                                    <artifactId>lombok</artifactId>
                                </path>
                            </annotationProcessorPaths>
                        </configuration>
                    </execution>
                    <execution>
                        <id>default-testCompile</id>
                        <phase>test-compile</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <annotationProcessorPaths>
                                <path>
                                    <groupId>org.projectlombok</groupId>
                                    <artifactId>lombok</artifactId>
                                </path>
                            </annotationProcessorPaths>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>
notification-service/Dockerfile
1
2
3
4
FROM eclipse-temurin:25
VOLUME /tmp
COPY target/*.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
compose.yaml (addition)
# add to the services of your compose.yaml
  notification:
    build:
      context: ./notification-service
      dockerfile: Dockerfile
    deploy:
      replicas: 1

6.3 Create an order

Build the order-service and the notification-service (mvn clean package -DskipTests in each), docker compose up -d --build, log in and create an order (see Hands-on 3 for the cookie):

curl -c cookies.txt -X POST http://localhost:8080/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "acme@insper.edu.br", "password": "123@321"}'
curl -b cookies.txt -X POST http://localhost:8080/orders \
  -H "Content-Type: application/json" \
  -d '{"items": [{"idProduct": "0195abfb-7074-73a9-9d26-b4b9fbaab0a8", "quantity": 2}]}'

The order comes back with 201 at once. You should see the email in the notification-service:

$ docker compose logs notification | grep NotificationService
notification-1  | … INFO 1 --- [notification] [ntainer#0-0-C-1] store.notification.NotificationService   : sending confirmation email: order 0195ac33-73e5-7cb3-90ca-7b5e7e549569 (account 0195abf8-…, total 20.24)

The request goes like this β€” the synchronous part for the customer, the email later, on the consumer's own time:

sequenceDiagram
    participant C as Client
    participant O as order-service
    participant DB as Database
    participant K as Kafka (order-events)
    participant N as notification-service
    C->>O: POST /orders
    O->>O: REST calls to account-service and product-service
    O->>DB: save the order
    O->>K: send(order-events, key = order id, event)
    O-->>C: 201 CREATED (does not wait for the email)
    K->>N: OrderCreated, on the next poll
    N->>N: log the confirmation email
    N->>K: commit the offset

7. Partitions and consumer groups

A topic is split into partitions. A record with a key goes to partition hash(key) % partitions (murmur2), so all events of one order stay in order in one partition. Inside a consumer group, each partition is read by one member at a time, and Kafka keeps the committed offset per partition. When a member joins or leaves, the group rebalances: partitions change owner, and the new owner resumes at the committed offset.

order-events was created with KAFKA_NUM_PARTITIONS: 1. Start a second notification-service and look at the group:

docker compose up -d --scale notification=2
docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group notification-group --members

One member shows #PARTITIONS 1, the other 0: the second instance is in the group with nothing to read. Add partitions β€” Kafka can add them, never remove them:

docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --alter --topic order-events --partitions 3

Once the clients refresh their metadata (up to five minutes, metadata.max.age.ms; docker compose restart notification order makes it immediate), the group rebalances: one instance reads partitions 0 and 1, the other partition 2 (the range assignor, the Java client's first strategy). Create a few orders and run --describe again: each partition shows its CURRENT-OFFSET, LOG-END-OFFSET and LAG.

flowchart LR
    os["order-service\nkey = order id"]
    subgraph topic ["order-events"]
        p0["partition 0"]
        p1["partition 1"]
        p2["partition 2"]
    end
    subgraph group ["notification-group"]
        n1["notification-1"]
        n2["notification-2"]
    end
    os --> p0
    os --> p1
    os --> p2
    p0 --> n1
    p1 --> n1
    p2 --> n2
    p0 & p1 & p2 -.->|"another group gets every message"| an["analytics-group"]
Instances in notification-group With 1 partition (the lab) With 3 partitions
1 βœ… reads partition 0 βœ… reads 0, 1 and 2
2 ⚠️ one instance is idle βœ… 0 and 1 Β· 2
3 ⚠️ two instances are idle βœ… one partition each
4 ⚠️ three instances are idle ⚠️ one instance is idle

The number of partitions caps the parallelism of a group. A different group (an analytics-service, say) reads every message independently, at its own offsets.

Without a key, partitions do not spread the work

Do the same with the reference: --partitions 3 on store-producer-example, docker compose up -d --scale consumer=2, restart both services and send nine messages with curl. All nine land in one partition (in our run, partition 1: offsets 0 to 8), so one consumer works and the other idles. A record without a key goes to a partition chosen by the producer's sticky partitioner, which fills one partition before moving on (KIP-794, a Kafka Improvement Proposal) β€” good for batching, useless for spreading a trickle of messages. A key spreads the load and keeps each key's events in order: that is why order-events uses the order id.

Try it

The simulator starts where the store's part of the lab starts: order-events with 1 partition, one notification-service in notification-group; play or β†’ runs a page, and the buttons work at any time. In 7 Partitions and groups, six orders land in partition 0 and a second instance gets nothing; after --partitions 3, notification-1 reads partitions 0 and 1, notification-2 reads partition 2, and the next six orders split 2, 2 and 2 by key. In free mode, scale to 4 instances with 3 partitions and watch one sit idle.


8. Failures: duplicates and bad messages

8.1 At-least-once

The container commits the offset after the listener returns. If the instance dies between sending the email and the commit, the offset never moved: whoever reads that partition next β€” the same instance after a restart, or another one after the rebalance β€” gets the message again and sends the email a second time.

sequenceDiagram
    participant K as Kafka (order-events)
    participant N as notification-service
    K->>N: record at offset 0
    N->>N: send the email
    N-xK: killed before the commit
    Note over K,N: offset 0 was never committed
    K->>N: after the restart, offset 0 again
    N->>N: the same email a second time
    N->>K: commit, the group now points at offset 1

Make the consumer idempotent

Kafka delivers at least once: a message may come twice. Processing the same event twice must give the same result as once. For the notification-service: record the order ids already notified in a durable store β€” a table with the order id as primary key, INSERT … ON CONFLICT DO NOTHING, and send the email only when the insert added a row. A set in memory does not help: it disappears in the very crash that causes the redelivery.

8.2 A message that cannot be processed

Publish something that is not JSON straight to the topic, with the console producer:

echo 'oops' | docker compose exec -T kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server kafka:9092 --topic order-events

readValue throws on every attempt. With no error handler of its own, the container uses Spring Kafka's DefaultErrorHandler: it retries the record 10 times with no pause, logs an ERROR and commits past it:

$ docker compose logs notification | grep -E "ERROR|Caused by"
notification-1  | … ERROR 1 --- [notification] [ntainer#0-0-C-1] o.s.kafka.listener.DefaultErrorHandler   : Backoff FixedBackOffExecution[interval=0, currentAttempts=10, maxAttempts=9] exhausted for order-events-0@1
notification-1  | Caused by: tools.jackson.core.exc.StreamReadException: Unrecognized token 'oops': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')

The stream does not block β€” the next orders are processed and the lag goes back to 0 β€” but the bad message is gone: one log line is all that is left. For an order whose event had a bug, that is a customer who never gets the email.

8.3 A dead-letter topic

A dead-letter topic (DLT) keeps the messages that failed, for someone to inspect, fix and replay. Replace the notification-service's KafkaConsumerConfig with this one: a KafkaTemplate to publish, and an error handler that tries 3 times, 1 s apart, then hands the record to a DeadLetterPublishingRecoverer:

notification-service/src/main/java/store/notification/KafkaConsumerConfig.java
package store.notification;

import java.util.HashMap;
import java.util.Map;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.EnableKafka;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.kafka.core.DefaultKafkaProducerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.FixedBackOff;

@EnableKafka
@Configuration
public class KafkaConsumerConfig {

    @Value(value = "${spring.kafka.bootstrap-servers}")
    private String bootstrapAddress;

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "notification-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props);
    }

    // publishes the failed records to the dead-letter topic
    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(props));
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        // 3 attempts, 1 s apart; then the record goes to order-events-dlt (partition chosen by Kafka)
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate(),
            (record, ex) -> new TopicPartition(record.topic() + "-dlt", -1));
        factory.setCommonErrorHandler(new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2L)));
        return factory;
    }

}

By default the recoverer writes to <topic>-dlt in the same partition as the original, so the DLT would need as many partitions as order-events; the resolver above returns partition -1 and lets Kafka choose. Rebuild, publish oops again and read the DLT:

$ docker compose logs notification | grep "not yet recovered"
notification-1  | … INFO 1 --- [notification] [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : Record in retry and not yet recovered
notification-1  | … INFO 1 --- [notification] [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : Record in retry and not yet recovered
$ docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic order-events-dlt --from-beginning --max-messages 1 --formatter-property print.offset=true
Offset:0    oops
Processed a total of 1 messages
flowchart LR
    m["record from\norder-events"] --> p{"orderCreated()"}
    p -->|ok| c["commit"]
    p -->|"exception\n(3 attempts, 1 s apart)"| d["DeadLetterPublishingRecoverer\n→ order-events-dlt"]
    d --> c
    c --> n["next record"]
    d -.->|"inspected and replayed later"| ops["operations"]

Either way the offset advances and the main stream continues; with the DLT, nothing is lost. (The recoverer also adds headers to the DLT record β€” the original topic, partition, offset and the exception β€” which --formatter-property print.headers=true shows.)

Try it

The simulator replays this section, one record per poll. A kill before the commit sends 1 duplicate email; the idempotent consumer skips the repeat. With the default error handler, a bad message is retried 10 times and skipped: the lag drops back to 0 and 1 message is lost. With the dead-letter topic, the next bad message ends in order-events-dlt after 3 attempts and nothing is lost.


9. Checklist

  • Kafka (KRaft mode) added to compose.yaml
  • producer-service publishes GET /producer/{msg} to store-producer-example; the route is in the gateway and open in the RouterValidator
  • consumer-service logs enqueue for every message; kafka-consumer-groups --describe --group store shows lag 0
  • with the consumer stopped, the lag grows; started again, it catches up
  • the order-service publishes OrderCreated to order-events, keyed by the order id, after saving
  • notification-service (group notification-group) logs the confirmation email for a new order
  • with 3 partitions, two notification instances share the work
  • a bad message ends in order-events-dlt; idempotency understood

Next: Quiz