Hands-on 9 — Kafka Messaging

Should the email block the customer?

• The customer does not need the email to finish the order: it is fire-and-forget.
• So publish an OrderCreated event to Kafka and return.
• A new notification-service consumes it asynchronously — the order-service does not even know it exists.

Architecture

The steps

1. Docker Compose — add Kafka (KRaft mode, no ZooKeeper) and Kafka UI.
2. order-service — spring-kafka, the OrderCreatedEvent record, producer config, publisher.
3. notification-service — a new Python / FastAPI consumer.
4. Test — create an order, see the event in Kafka UI and in the notification logs.
5. Reliability — dead letter queue and idempotent consumers.

1 · Docker Compose

Key Kafka settings:

• KAFKA_PROCESS_ROLES: broker,controller — KRaft: Kafka keeps its own metadata; ZooKeeper was removed in Kafka 4.0.
• KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 — clients connect by the service name.
• KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" — order-events is created on first publish.
• KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 — a single broker in development.

2 · order-service: the event

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

public record OrderCreatedEvent(
    String orderId,
    String customerId,
    Double totalAmount,
    String timestamp
) {}

A record: immutable, like the fact it represents.

2 · Producer configuration

spring:
  kafka:
    bootstrap-servers: kafka:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all
      retries: 3
      properties:
        enable.idempotence: true

acks: all waits for every in-sync replica before confirming. No effect on a single broker; in production it prevents data loss on broker failover.

2 · Publishing

@Component
@RequiredArgsConstructor
public class OrderEventPublisher {
    private final KafkaTemplate<String, OrderCreatedEvent> kafkaTemplate;

    public void publishOrderCreated(Order order) {
        var event = new OrderCreatedEvent(order.getId().toString(),
            order.getCustomerId(), order.getTotalAmount(), Instant.now().toString());
        kafkaTemplate.send("order-events", order.getId().toString(), event);
    }
}

In OrderService, after saving: eventPublisher.publishOrderCreated(savedOrder);

Topic order-events, key = order id, value = the event as JSON.

3 · notification-service (Python / FastAPI)

notification-service/
    main.py
    Dockerfile
    requirements.txt

fastapi==0.110.0
kafka-python==2.3.2
uvicorn==0.29.0

3 · The consumer

consumer = KafkaConsumer(
    "order-events",
    bootstrap_servers="kafka:9092",
    group_id="notification-group",
    value_deserializer=lambda m: json.loads(m.decode("utf-8")),
    auto_offset_reset="earliest",
    enable_auto_commit=False,
)
for message in consumer:
    event = message.value
    logger.info(f"[NOTIFICATION] Order {event['orderId']} ... sending confirmation email")
    consumer.commit()

enable_auto_commit=False + commit() after processing: a crash mid-processing means the message is consumed again — at-least-once.

5 · When processing fails: DLQ

A message that keeps failing must not block the stream. Send it to order-events-dlq instead:

try:
    process_event(event)
    consumer.commit()
except Exception as e:
    logger.error(f"Failed to process {event['orderId']}: {e} — sending to DLQ")
    dlq_producer.send("order-events-dlq", value=event)
    consumer.commit()

Either way the offset advances: the main stream continues, the DLQ is inspected later.

Pitfalls

• Publish after saving the order — never announce an order that does not exist.
• Inside the Docker network the broker is kafka:9092, not localhost:9092.
• acks: all only protects data when there is more than one broker.
• Use Kafka UI to check topics, messages and consumer group lag while debugging.

Takeaways

1. Use REST when the caller needs the answer now; use events for fire-and-forget work.
2. The order-service publishes OrderCreated to order-events and does not know its consumers.
3. Kafka runs in KRaft mode in compose (no ZooKeeper); Kafka UI shows topics, messages and lag.
4. Spring: KafkaTemplate.send(topic, key, event) with a JSON serializer and acks: all.
5. Python: a KafkaConsumer in group notification-group with manual commit → at-least-once.
6. Failing messages go to a DLQ; consumers must be idempotent.