1. Kafka in Compose

In this step you replace your docker-compose.yaml with the complete file of this lab, start the four new infrastructure containers and get to know Kafka from the command line before any Java code uses it. All commands run from the root of your boxoffice repository.

Container	Image	Host port	Role
`kafka`	`apache/kafka:4.3.1`	9094	one broker that is also its own controller (KRaft)
`kafka-init`	`apache/kafka:4.3.1`	—	one-shot job: creates the topics and exits
`kafka-ui`	`kafbat/kafka-ui:v1.5.0`	8089	browse topics, messages, headers and consumer lag
`mailpit`	`axllent/mailpit:v1.31`	8025 (web), 1025 (SMTP)	fake e-mail provider with an inbox and an API

Kafka in KRaft mode

Kafka keeps its own metadata (topics, partitions, leaders) in a Raft quorum of controllers — KRaft. ZooKeeper is gone since Kafka 4.0. Here a single node plays both roles, broker,controller. This is the new part of docker-compose.yaml:

docker-compose.yaml (excerpt)

  kafka:
    image: apache/kafka:4.3.1
    hostname: kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
      interval: 10s
      timeout: 10s
      start_period: 60s      # a busy laptop can take a minute to start the broker
      retries: 20

  kafka-init:
    image: apache/kafka:4.3.1
    depends_on:
      kafka:
        condition: service_healthy
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
          --topic order-events --partitions 6 --replication-factor 1 &&
        /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
          --topic order-events-dlt --partitions 6 --replication-factor 1
    restart: "no"

  kafka-ui:
    image: kafbat/kafka-ui:v1.5.0
    depends_on:
      kafka:
        condition: service_healthy
    ports:
      - "8089:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: boxoffice
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092

  mailpit:
    image: axllent/mailpit:v1.31
    ports:
      - "8025:8025"
      - "1025:1025"

Get the complete file

Run from the root of your boxoffice repository (it replaces the lab 6 version; keep a copy if you changed it):

cp docker-compose.yaml docker-compose.lab6.yaml
curl -fsSL -o docker-compose.yaml https://insper.github.io/platform/2027.1/hands-on/7/code/docker-compose.yaml
docker compose config --services

You should see the lab 6 services plus kafka, kafka-init, kafka-ui, mailpit, order and notification. order and notification are built in steps 2 and 3; until then, start only the services named in each command. Delete docker-compose.lab6.yaml before you commit: Git already keeps the lab 6 version (git show lab6:docker-compose.yaml).

docker-compose.yaml — the complete file at the end of this lab

docker-compose.yaml

name: boxoffice

# settings shared by the three catalog replicas (a YAML anchor)
x-catalog: &catalog
  build: ./services/catalog
  image: boxoffice-catalog
  environment:
    DB_HOST: db
    DB_PORT: 5432
    DB_NAME: ${DB_NAME:-boxoffice}
    DB_USER: ${DB_USER:-boxoffice}
    DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
    DB_POOL_SIZE: ${DB_POOL_SIZE:-10}
    REDIS_HOST: redis
    REDIS_PORT: 6379
    CACHE_ENABLED: ${CACHE_ENABLED:-true}
    CACHE_TTL_EVENT: ${CACHE_TTL_EVENT:-60s}
    CACHE_TTL_LIST: ${CACHE_TTL_LIST:-10s}
    CACHE_STAMPEDE_PROTECTION: ${CACHE_STAMPEDE_PROTECTION:-false}
  depends_on:
    db:
      condition: service_healthy
    redis:
      condition: service_healthy
  deploy:
    resources:
      limits:
        cpus: "1.0"
        memory: 768M
  healthcheck:
    test: ["CMD-SHELL", "wget -qO- http://localhost:8082/actuator/health/readiness || exit 1"]
    interval: 10s
    timeout: 3s
    start_period: 40s
    retries: 3

services:

  db:
    image: postgres:17
    command: ["postgres", "-c", "max_connections=${DB_MAX_CONNECTIONS:-100}"]
    environment:
      POSTGRES_DB: ${DB_NAME:-boxoffice}
      POSTGRES_USER: ${DB_USER:-boxoffice}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-boxoffice}
      REPLICATION_PASSWORD: ${REPLICATION_PASSWORD:-replicator}
    ports:
      - "5432:5432"
    volumes:
      - db-data:/var/lib/postgresql/data
      # runs only when the volume is new: creates the "replicator" user
      - ./infra/postgres/10-replication.sh:/docker-entrypoint-initdb.d/10-replication.sh:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 3s
      retries: 10

  # read-only streaming replica of db (hot standby)
  db-replica:
    image: postgres:17
    user: postgres
    entrypoint: ["/usr/local/bin/replica-entrypoint.sh"]
    environment:
      PGPASSWORD: ${REPLICATION_PASSWORD:-replicator}     # used by pg_basebackup
      REPLICA_DELAY: ${REPLICA_DELAY:-0}                   # e.g. 2s: artificial lag
    ports:
      - "5433:5432"
    volumes:
      - replica-data:/var/lib/postgresql/data
      - ./infra/postgres/replica-entrypoint.sh:/usr/local/bin/replica-entrypoint.sh:ro
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U boxoffice -d boxoffice"]
      interval: 5s
      timeout: 3s
      retries: 20

  pgbouncer:
    image: edoburu/pgbouncer:v1.26.0-p0
    ports:
      - "6432:6432"
    volumes:
      - ./infra/pgbouncer/pgbouncer.ini:/etc/pgbouncer/pgbouncer.ini:ro
      - ./infra/pgbouncer/userlist.txt:/etc/pgbouncer/userlist.txt:ro
    depends_on:
      db:
        condition: service_healthy

  redis:
    image: redis:8-alpine
    # a pure cache: bounded memory, evict least-recently-used keys, no persistence
    command: ["redis-server", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru",
              "--save", "", "--appendonly", "no"]
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

  # three identical replicas; no "ports:" — only the edge is reachable from outside
  catalog-1:
    <<: *catalog
    hostname: catalog-1
  catalog-2:
    <<: *catalog
    hostname: catalog-2
  catalog-3:
    <<: *catalog
    hostname: catalog-3

  inventory:
    build: ./services/inventory
    environment:
      DB_HOST: ${INVENTORY_DB_HOST:-pgbouncer}      # writes go through PgBouncer...
      DB_PORT: ${INVENTORY_DB_PORT:-6432}
      DB_MIGRATION_HOST: db                         # ...migrations straight to the primary
      DB_REPLICA_HOST: ${DB_REPLICA_HOST:-db-replica}
      DB_REPLICA_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      DB_POOL_SIZE: ${INVENTORY_POOL_SIZE:-20}
      INVENTORY_LOCKING: ${INVENTORY_LOCKING:-pessimistic}
      INVENTORY_HOLD_TTL: ${INVENTORY_HOLD_TTL:-PT10M}
    ports:
      - "8083:8083"            # published for experiments; clients go through the edge
    depends_on:
      db:
        condition: service_healthy
      db-replica:
        condition: service_healthy
      pgbouncer:
        condition: service_started
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8083/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 40s
      retries: 3

  auth:
    build: ./services/auth
    environment:
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      JWT_ISSUER: ${JWT_ISSUER:-boxoffice-auth}
      JWT_TTL: ${JWT_TTL:-PT1H}
      ADMIN_EMAIL: ${ADMIN_EMAIL:-admin@boxoffice.dev}
      ADMIN_PASSWORD: ${ADMIN_PASSWORD:-boxoffice-admin}
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8081/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 40s
      retries: 3

  gateway:
    build: ./services/gateway
    environment:
      AUTH_URL: http://auth:8081
      INVENTORY_URL: http://inventory:8083
      AUTH_JWKS_URI: http://auth:8081/auth/.well-known/jwks.json
      JWT_ISSUER: ${JWT_ISSUER:-boxoffice-auth}
    ports:
      - "8080:8080"          # published for experiments; clients use the edge (port 80)
    depends_on:
      - auth
      - inventory
      - catalog-1
      - catalog-2
      - catalog-3
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8080/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 40s
      retries: 3

  edge:
    image: nginx:1.30-alpine
    ports:
      - "80:80"
    volumes:
      - ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - gateway

  # ---- lab 7: Kafka (KRaft), Kafka UI, Mailpit, order and notification ----

  kafka:
    image: apache/kafka:4.3.1
    hostname: kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
      interval: 10s
      timeout: 10s
      start_period: 60s      # a busy laptop can take a minute to start the broker
      retries: 20

  kafka-init:
    image: apache/kafka:4.3.1
    depends_on:
      kafka:
        condition: service_healthy
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
          --topic order-events --partitions 6 --replication-factor 1 &&
        /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
          --topic order-events-dlt --partitions 6 --replication-factor 1
    restart: "no"

  kafka-ui:
    image: kafbat/kafka-ui:v1.5.0
    depends_on:
      kafka:
        condition: service_healthy
    ports:
      - "8089:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: boxoffice
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092

  mailpit:
    image: axllent/mailpit:v1.31
    ports:
      - "8025:8025"
      - "1025:1025"

  order:
    build: ./services/order
    environment:
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      REDIS_HOST: redis
      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      NOTIFICATION_URL: http://notification:8086
      NOTIFY_MODE: ${NOTIFY_MODE:-outbox}            # outbox (after) | sync (before)
      ADMIT_PER_SECOND: ${ADMIT_PER_SECOND:-20}      # waiting-room admission rate
    ports:
      - "8084:8084"
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
      kafka-init:
        condition: service_completed_successfully
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8084/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 40s
      retries: 3

  notification:
    build: ./services/notification
    environment:
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      MAIL_HOST: mailpit
      NOTIFY_CONCURRENCY: ${NOTIFY_CONCURRENCY:-3}   # consumer threads (max useful: 6 partitions)
      EMAIL_LATENCY: ${EMAIL_LATENCY:-50ms}          # simulated e-mail provider latency
      NOTIFY_IDEMPOTENT: ${NOTIFY_IDEMPOTENT:-true}  # false = no dedupe (to see duplicates)
    ports:
      - "8086:8086"
    depends_on:
      db:
        condition: service_healthy
      mailpit:
        condition: service_started
      kafka-init:
        condition: service_completed_successfully
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8086/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 40s
      retries: 3

volumes:
  db-data:
  replica-data:

Three details matter:

Two client listeners. Containers reach the broker as kafka:9092 (PLAINTEXT). A program on your laptop (an IDE run, a CLI) uses localhost:9094 (EXTERNAL). A client first connects to any address, then receives the advertised address of each partition leader and reconnects there — so each listener must advertise a name that its clients can resolve. Mixing them up is the classic "connects, then times out" bug. That is why Kafka is the one exception to the "same port inside and outside" rule of lab 0: services use kafka:9092, your machine uses localhost:9094.
Topics are created explicitly (KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"). Auto-creation uses the broker default of 1 partition, which would silently cap the consumer parallelism at 1.
The dead-letter topic has as many partitions as the main topic: Spring's dead-letter publisher writes a failed record to the same partition number in order-events-dlt.

Why 6 partitions?

A partition is consumed by at most one consumer of a group at a time, so the number of partitions is the ceiling of consumer parallelism. Six lets the notification group grow to six consumers. Partitions are cheap to start with and painful to add later: adding partitions changes which partition a key maps to, breaking per-key ordering for keys already in flight.

Start and check

docker compose up -d kafka kafka-init kafka-ui mailpit ✔ Container boxoffice-mailpit-1 Started
✔ Container boxoffice-kafka-1 Healthy
✔ Container boxoffice-kafka-init-1 Started
✔ Container boxoffice-kafka-ui-1 Starteddocker compose logs kafka-initkafka-init-1 | Created topic order-events.
kafka-init-1 | Created topic order-events-dlt.docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --describe --topic order-eventsTopic: order-events TopicId: hwjrAir1Q8m_ZV4fyhnctw PartitionCount: 6 ReplicationFactor: 1 Configs: min.insync.replicas=1
Topic: order-events Partition: 0 Leader: 1 Replicas: 1 Isr: 1 Elr: LastKnownElr:
Topic: order-events Partition: 1 Leader: 1 Replicas: 1 Isr: 1 Elr: LastKnownElr:
Topic: order-events Partition: 2 Leader: 1 Replicas: 1 Isr: 1 Elr: LastKnownElr:
Topic: order-events Partition: 3 Leader: 1 Replicas: 1 Isr: 1 Elr: LastKnownElr:
Topic: order-events Partition: 4 Leader: 1 Replicas: 1 Isr: 1 Elr: LastKnownElr:
Topic: order-events Partition: 5 Leader: 1 Replicas: 1 Isr: 1 Elr: LastKnownElr:

You should see kafka, kafka-ui and mailpit running and kafka-init exited with code 0 (docker compose ps -a), six partitions for order-events, Kafka UI at http://localhost:8089 (cluster boxoffice online, topics order-events and order-events-dlt) and an empty Mailpit inbox at http://localhost:8025. The TopicId differs on every machine. If kafka stays starting for more than a minute, see Troubleshooting in the overview.

A five-minute tour of the log

Before Spring hides it, feel how a log differs from a queue. Open two terminals.

In terminal A, start a consumer in group tour:

docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server kafka:9092 --topic order-events --group tour \
  --formatter-property print.key=true --formatter-property print.partition=true

In terminal B, produce three keyed messages (key:value):

docker compose exec -T kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server kafka:9092 --topic order-events \
  --reader-property parse.key=true --reader-property key.separator=: <<'EOF'
order-1:{"hello":"first"}
order-2:{"hello":"second"}
order-1:{"hello":"third"}
EOF

Terminal A prints each message with its partition, for example:

Partition:4 order-1 {"hello":"first"}
Partition:3 order-2 {"hello":"second"}
Partition:4 order-1 {"hello":"third"}

Both order-1 messages land on the same partition — the default partitioner hashes the key (murmur2) modulo the number of partitions — and that is the only ordering Kafka promises: per partition, hence per key.

Now stop the consumer (Ctrl+C) and look at the group:

docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group tour
Consumer group 'tour' has no active members.

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
tour order-events 0 0 0 0 - - -
tour order-events 1 0 0 0 - - -
tour order-events 2 0 0 0 - - -
tour order-events 3 1 1 0 - - -
tour order-events 4 2 2 0 - - -
tour order-events 5 0 0 0 - - -

The messages are still in the log: consuming does not delete. Each group only stores its offset per partition (CURRENT-OFFSET), and lag = LOG-END-OFFSET − CURRENT-OFFSET is how far behind it is. Rewind the group and read everything again:

docker compose exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --group tour --topic order-events --reset-offsets --to-earliest --execute

That replay ability is what makes Kafka a good backbone for new consumers (analytics can read the whole history) — and also why every consumer must tolerate seeing a message twice.

Keep the tour messages

The three test messages are not valid BoxOffice events. They will be useful later: in step 3 they are exactly the kind of poison message the consumer must survive. Leave them there.

Think first: a consumer group with 8 instances on a 6-partition topic

How many instances receive messages? What do the other ones do?

Six. Each partition is assigned to one member of the group, so two instances stay idle (they take over if another member dies). To use more consumers, the topic needs more partitions.

Next: Transactional outbox