1. Payment, Provider and Toxiproxy

BoxOffice does not move money itself: the payment service is an adapter to an external payment service provider (PSP). That makes it the most dangerous dependency of the checkout — slow on a busy day, outside your control, and the place where a retry can charge a fan twice. This step builds the three pieces the rest of the lab breaks on purpose:

Container	Image / build	Port	Role
`psp`	`wiremock/wiremock:3.13.2`	8080 (internal)	a fake external provider: `POST /v1/charges` answers `approved`
`payment`	`services/payment` (Spring Boot)	8085	adapter: idempotent `POST /payments`, calls the PSP with timeouts
`toxiproxy`	`ghcr.io/shopify/toxiproxy:2.13.0`	8474 (API)	TCP proxy in front of `payment` and `psp`; injects latency, resets, outages

flowchart LR
    order[order] -->|"PAYMENT_URL = toxiproxy:18085"| tp1["toxiproxy proxy: payment"]
    tp1 --> payment[payment :8085]
    payment -->|"PSP_URL = toxiproxy:18095"| tp2["toxiproxy proxy: psp"]
    tp2 --> psp["psp (WireMock)"]
    payment --> db[(db: schema payment)]
    you[you / game day] -->|"curl :8474 add toxic"| tp1

Every call crosses a Toxiproxy proxy, so you can make any hop slow or broken without touching code: Toxiproxy works at the TCP level, below HTTP, exactly where real networks fail.

The fake provider

WireMock answers HTTP from JSON stub files. One stub is enough: approve every charge and invent a provider reference.

infra/psp/mappings/charge.json

{
  "request": {
    "method": "POST",
    "urlPath": "/v1/charges"
  },
  "response": {
    "status": 201,
    "headers": { "Content-Type": "application/json" },
    "jsonBody": {
      "id": "ch_{{randomValue length=12 type='ALPHANUMERIC'}}",
      "status": "approved"
    },
    "transformers": ["response-template"]
  }
}

The `payment` service

A Spring Boot service in services/payment/, port 8085, serving under /payments, schema payment. Its build files follow the same pattern as notification in lab 7, without Kafka and mail.

services/payment/
├── Dockerfile
├── .dockerignore
├── pom.xml
└── src/main/
    ├── java/boxoffice/payment/
    │   ├── PaymentApplication.java
    │   ├── PaymentController.java
    │   ├── PaymentService.java
    │   ├── PspClient.java
    │   ├── ChargeRequest.java
    │   └── PaymentOut.java
    └── resources/
        ├── application.yaml
        └── db/migration/V1__create_payments.sql

pom.xmlDockerfile.dockerignoreapplication.yamlV1__create_payments.sqlChargeRequest.javaPaymentOut.javaPaymentApplication.java

services/payment/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>4.1.1</version>
        <relativePath/>
    </parent>

    <groupId>boxoffice</groupId>
    <artifactId>payment</artifactId>
    <version>1.0.0</version>

    <properties>
        <java.version>21</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webmvc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-restclient</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-validation</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jdbc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-flyway</artifactId>
        </dependency>
        <dependency>
            <groupId>org.flywaydb</groupId>
            <artifactId>flyway-database-postgresql</artifactId>
        </dependency>
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <scope>runtime</scope>
        </dependency>
    </dependencies>

    <build>
        <finalName>app</finalName>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

services/payment/Dockerfile

# syntax=docker/dockerfile:1
# same Dockerfile as catalog (lab 1); only the port changes
FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /src
COPY pom.xml .
COPY src ./src
RUN --mount=type=cache,target=/root/.m2 mvn -B -q package -DskipTests

FROM eclipse-temurin:21-jre
WORKDIR /app
RUN useradd --system --uid 10001 app
COPY --from=build /src/target/app.jar app.jar
USER app
EXPOSE 8085
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=75", "-jar", "app.jar"]

services/payment/.dockerignore

target/
.idea/
.vscode/
*.iml

services/payment/src/main/resources/application.yaml

server:
  port: 8085

spring:
  application:
    name: payment
  mvc:
    problemdetails:
      enabled: true
  datasource:
    url: jdbc:postgresql://${DB_HOST:db}:${DB_PORT:5432}/${DB_NAME:boxoffice}?currentSchema=payment
    username: ${DB_USER:boxoffice}
    password: ${DB_PASSWORD:boxoffice}
  flyway:
    schemas: payment

boxoffice:
  psp:
    url: ${PSP_URL:http://psp:8080}
    connect-timeout: 500ms
    read-timeout: ${PSP_READ_TIMEOUT:2s}

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics
  endpoint:
    health:
      probes:
        enabled: true

services/payment/src/main/resources/db/migration/V1__create_payments.sql

CREATE TABLE payments (
    id              UUID          PRIMARY KEY,
    idempotency_key VARCHAR(100)  NOT NULL UNIQUE,  -- one charge per key, whatever the retries
    order_id        UUID          NOT NULL,
    amount          NUMERIC(10,2) NOT NULL,
    status          VARCHAR(20)   NOT NULL,          -- AUTHORIZED | DECLINED
    provider_ref    VARCHAR(100),
    created_at      TIMESTAMPTZ   NOT NULL DEFAULT now()
);

services/payment/src/main/java/boxoffice/payment/ChargeRequest.java

package boxoffice.payment;

import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Positive;
import java.math.BigDecimal;
import java.util.UUID;

public record ChargeRequest(@NotNull UUID orderId, @NotNull @Positive BigDecimal amount) {
}

services/payment/src/main/java/boxoffice/payment/PaymentOut.java

package boxoffice.payment;

import java.math.BigDecimal;
import java.time.OffsetDateTime;
import java.util.UUID;

public record PaymentOut(
        UUID id,
        UUID orderId,
        BigDecimal amount,
        String status,
        String providerRef,
        OffsetDateTime createdAt) {
}

services/payment/src/main/java/boxoffice/payment/PaymentApplication.java

package boxoffice.payment;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class PaymentApplication {

    public static void main(String[] args) {
        SpringApplication.run(PaymentApplication.class, args);
    }
}

The provider client sets both timeouts explicitly. Without them, a Java HTTP client waits forever for a response — the root cause of most cascading failures.

PspClient.javaPaymentService.javaPaymentController.java

services/payment/src/main/java/boxoffice/payment/PspClient.java

package boxoffice.payment;

import java.math.BigDecimal;
import java.time.Duration;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

@Component
public class PspClient {

    record ChargeBody(BigDecimal amount, String currency, String reference) {
    }

    record Charge(String id, String status) {
    }

    private final RestClient http;

    public PspClient(RestClient.Builder builder,
                     @Value("${boxoffice.psp.url}") String url,
                     @Value("${boxoffice.psp.connect-timeout}") Duration connectTimeout,
                     @Value("${boxoffice.psp.read-timeout}") Duration readTimeout) {
        SimpleClientHttpRequestFactory factory = new SimpleClientHttpRequestFactory();
        factory.setConnectTimeout(connectTimeout); // time to open the TCP connection
        factory.setReadTimeout(readTimeout);       // max silence while waiting for the response
        this.http = builder.baseUrl(url).requestFactory(factory).build();
    }

    /** The provider deduplicates by Idempotency-Key, like real PSPs do. */
    public Charge charge(String idempotencyKey, BigDecimal amount, String reference) {
        return http.post()
                .uri("/v1/charges")
                .header("Idempotency-Key", idempotencyKey)
                .body(new ChargeBody(amount, "BRL", reference))
                .retrieve()
                .body(Charge.class);
    }
}

services/payment/src/main/java/boxoffice/payment/PaymentService.java

package boxoffice.payment;

import java.util.Optional;
import java.util.UUID;
import org.springframework.jdbc.core.simple.JdbcClient;
import org.springframework.stereotype.Service;

@Service
public class PaymentService {

    public record Result(PaymentOut payment, boolean created) {
    }

    private final JdbcClient jdbc;
    private final PspClient psp;

    public PaymentService(JdbcClient jdbc, PspClient psp) {
        this.jdbc = jdbc;
        this.psp = psp;
    }

    public Result charge(String idempotencyKey, ChargeRequest req) {
        Optional<PaymentOut> done = find(idempotencyKey);
        if (done.isPresent()) {
            return new Result(done.get(), false); // a retry: answer, do not charge again
        }
        // remote call OUTSIDE any database transaction: never hold a connection while waiting
        PspClient.Charge charge = psp.charge(idempotencyKey, req.amount(), "order-" + req.orderId());
        String status = "approved".equals(charge.status()) ? "AUTHORIZED" : "DECLINED";
        jdbc.sql("""
                INSERT INTO payments (id, idempotency_key, order_id, amount, status, provider_ref)
                VALUES (?, ?, ?, ?, ?, ?)
                ON CONFLICT (idempotency_key) DO NOTHING
                """)
                .params(UUID.randomUUID(), idempotencyKey, req.orderId(), req.amount(), status, charge.id())
                .update();
        return new Result(find(idempotencyKey).orElseThrow(), true);
    }

    private Optional<PaymentOut> find(String idempotencyKey) {
        return jdbc.sql("""
                SELECT id, order_id, amount, status, provider_ref, created_at
                FROM payments WHERE idempotency_key = ?
                """)
                .param(idempotencyKey)
                .query(PaymentOut.class)
                .optional();
    }
}

services/payment/src/main/java/boxoffice/payment/PaymentController.java

package boxoffice.payment;

import jakarta.validation.Valid;
import java.net.URI;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/payments")
public class PaymentController {

    private final PaymentService payments;

    public PaymentController(PaymentService payments) {
        this.payments = payments;
    }

    @PostMapping
    public ResponseEntity<PaymentOut> charge(@RequestHeader("Idempotency-Key") String idempotencyKey,
                                             @Valid @RequestBody ChargeRequest req) {
        PaymentService.Result r = payments.charge(idempotencyKey, req);
        return r.created()
                ? ResponseEntity.created(URI.create("/payments/" + r.payment().id())).body(r.payment())
                : ResponseEntity.ok(r.payment());
    }
}

Idempotency end to end

The same key travels order → payment → PSP: order sends Idempotency-Key: order-<id>, payment stores it with a UNIQUE constraint and forwards it to the provider. A retry at any layer — Resilience4j in order, a timeout in the gateway, a fan double-clicking — finds the key and gets the first result back. Retries are only safe on top of idempotency; this is the payment-system case study in miniature.

Toxiproxy

Toxiproxy reads its proxies from a JSON file at startup:

infra/toxiproxy/toxiproxy.json

[
  { "name": "payment", "listen": "0.0.0.0:18085", "upstream": "payment:8085", "enabled": true },
  { "name": "psp",     "listen": "0.0.0.0:18095", "upstream": "psp:8080",     "enabled": true }
]

Compose

docker-compose.yaml (excerpt)

  psp:
    image: wiremock/wiremock:3.13.2
    volumes:
      - ./infra/psp:/home/wiremock:ro

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.13.0
    command: ["-host=0.0.0.0", "-config=/config/toxiproxy.json"]
    volumes:
      - ./infra/toxiproxy/toxiproxy.json:/config/toxiproxy.json:ro
    ports:
      - "8474:8474"

  payment:
    build: ./services/payment
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: payment
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      PSP_URL: http://toxiproxy:18095                 # the provider, through Toxiproxy
    ports:
      - "8085:8085"
    depends_on:
      db:
        condition: service_healthy
      toxiproxy:
        condition: service_started
      psp:
        condition: service_started
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8085/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

and the order service now reaches payment through the proxy (not directly), with the knobs of step 2 as variables:

docker-compose.yaml (excerpt of the order service)

      PAYMENT_URL: http://toxiproxy:18085            # lab 9: through Toxiproxy, not payment:8085
      PAYMENT_RESILIENCE: ${PAYMENT_RESILIENCE:-true}
      PAYMENT_READ_TIMEOUT: ${PAYMENT_READ_TIMEOUT:-1s}
      PAYMENT_RETRY_ATTEMPTS: ${PAYMENT_RETRY_ATTEMPTS:-3}
      PAYMENT_BULKHEAD: ${PAYMENT_BULKHEAD:-10}

Get the complete file (lab 8's plus these services and variables), from the repository root:

curl -fsSL -o docker-compose.yaml https://insper.github.io/platform/2027.1/hands-on/9/code/docker-compose.yaml

docker-compose.yaml — the complete file at the end of this lab

docker-compose.yaml

name: boxoffice

# lab 8: OpenTelemetry settings shared by every Java service (merged into each "environment")
x-otel: &otel
  JAVA_TOOL_OPTIONS: -javaagent:/otel/opentelemetry-javaagent.jar
  OTEL_JAVAAGENT_ENABLED: ${OTEL_ON:-true}          # false = same image, agent off
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-lgtm:4318
  OTEL_METRIC_EXPORT_INTERVAL: "10000"              # 10 s instead of 60 s
  OTEL_RESOURCE_ATTRIBUTES: service.namespace=boxoffice,deployment.environment.name=local
  OTEL_TRACES_SAMPLER: ${OTEL_TRACES_SAMPLER:-parentbased_always_on}
  OTEL_TRACES_SAMPLER_ARG: ${OTEL_TRACES_SAMPLER_ARG:-1.0}
  LOGGING_PATTERN_CORRELATION: "[%X{trace_id:-},%X{span_id:-}] "   # trace id in console logs

# settings shared by the three catalog replicas (a YAML anchor)
x-catalog: &catalog
  build: ./services/catalog
  image: boxoffice-catalog
  volumes:
    - ./infra/otel:/otel:ro          # lab 8: the Java agent
  environment:
    <<: *otel
    OTEL_SERVICE_NAME: catalog       # three replicas, ONE service
    DB_HOST: db
    DB_PORT: 5432
    DB_NAME: ${DB_NAME:-boxoffice}
    DB_USER: ${DB_USER:-boxoffice}
    DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
    DB_POOL_SIZE: ${DB_POOL_SIZE:-10}
    REDIS_HOST: redis
    REDIS_PORT: 6379
    CACHE_ENABLED: ${CACHE_ENABLED:-true}
    CACHE_TTL_EVENT: ${CACHE_TTL_EVENT:-60s}
    CACHE_TTL_LIST: ${CACHE_TTL_LIST:-10s}
    CACHE_STAMPEDE_PROTECTION: ${CACHE_STAMPEDE_PROTECTION:-false}
  depends_on:
    db:
      condition: service_healthy
    redis:
      condition: service_healthy
  deploy:
    resources:
      limits:
        cpus: "1.0"
        memory: 768M
  healthcheck:
    test: ["CMD-SHELL", "wget -qO- http://localhost:8082/actuator/health/readiness || exit 1"]
    interval: 10s
    timeout: 3s
    start_period: 120s       # lab 8: with the agent, JVMs starting together need up to ~90 s
    retries: 3

services:

  db:
    image: postgres:17
    command: ["postgres", "-c", "max_connections=${DB_MAX_CONNECTIONS:-100}"]
    environment:
      POSTGRES_DB: ${DB_NAME:-boxoffice}
      POSTGRES_USER: ${DB_USER:-boxoffice}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-boxoffice}
      REPLICATION_PASSWORD: ${REPLICATION_PASSWORD:-replicator}
    ports:
      - "5432:5432"
    volumes:
      - db-data:/var/lib/postgresql/data
      # runs only when the volume is new: creates the "replicator" user
      - ./infra/postgres/10-replication.sh:/docker-entrypoint-initdb.d/10-replication.sh:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 3s
      retries: 10

  # read-only streaming replica of db (hot standby)
  db-replica:
    image: postgres:17
    user: postgres
    entrypoint: ["/usr/local/bin/replica-entrypoint.sh"]
    environment:
      PGPASSWORD: ${REPLICATION_PASSWORD:-replicator}     # used by pg_basebackup
      REPLICA_DELAY: ${REPLICA_DELAY:-0}                   # e.g. 2s: artificial lag
    ports:
      - "5433:5432"
    volumes:
      - replica-data:/var/lib/postgresql/data
      - ./infra/postgres/replica-entrypoint.sh:/usr/local/bin/replica-entrypoint.sh:ro
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U boxoffice -d boxoffice"]
      interval: 5s
      timeout: 3s
      retries: 20

  pgbouncer:
    image: edoburu/pgbouncer:v1.26.0-p0
    ports:
      - "6432:6432"
    volumes:
      - ./infra/pgbouncer/pgbouncer.ini:/etc/pgbouncer/pgbouncer.ini:ro
      - ./infra/pgbouncer/userlist.txt:/etc/pgbouncer/userlist.txt:ro
    depends_on:
      db:
        condition: service_healthy

  redis:
    image: redis:8-alpine
    # a pure cache: bounded memory, evict least-recently-used keys, no persistence
    command: ["redis-server", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru",
              "--save", "", "--appendonly", "no"]
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

  # three identical replicas; no "ports:" — only the edge is reachable from outside
  catalog-1:
    <<: *catalog
    hostname: catalog-1
  catalog-2:
    <<: *catalog
    hostname: catalog-2
  catalog-3:
    <<: *catalog
    hostname: catalog-3

  inventory:
    build: ./services/inventory
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: inventory
      DB_HOST: ${INVENTORY_DB_HOST:-pgbouncer}      # writes go through PgBouncer...
      DB_PORT: ${INVENTORY_DB_PORT:-6432}
      DB_MIGRATION_HOST: db                         # ...migrations straight to the primary
      DB_REPLICA_HOST: ${DB_REPLICA_HOST:-db-replica}
      DB_REPLICA_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      DB_POOL_SIZE: ${INVENTORY_POOL_SIZE:-20}
      INVENTORY_LOCKING: ${INVENTORY_LOCKING:-pessimistic}
      INVENTORY_HOLD_TTL: ${INVENTORY_HOLD_TTL:-PT10M}
    ports:
      - "8083:8083"            # published for experiments; clients go through the edge
    depends_on:
      db:
        condition: service_healthy
      db-replica:
        condition: service_healthy
      pgbouncer:
        condition: service_started
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8083/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  auth:
    build: ./services/auth
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: auth
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      JWT_ISSUER: ${JWT_ISSUER:-boxoffice-auth}
      JWT_TTL: ${JWT_TTL:-PT1H}
      ADMIN_EMAIL: ${ADMIN_EMAIL:-admin@boxoffice.dev}
      ADMIN_PASSWORD: ${ADMIN_PASSWORD:-boxoffice-admin}
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8081/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  gateway:
    build: ./services/gateway
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: gateway
      AUTH_URL: http://auth:8081
      INVENTORY_URL: http://inventory:8083
      AUTH_JWKS_URI: http://auth:8081/auth/.well-known/jwks.json
      JWT_ISSUER: ${JWT_ISSUER:-boxoffice-auth}
    ports:
      - "8080:8080"          # published for experiments; clients use the edge (port 80)
    depends_on:
      - auth
      - inventory
      - catalog-1
      - catalog-2
      - catalog-3
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8080/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  edge:
    image: nginx:1.30-alpine
    ports:
      - "80:80"
    volumes:
      - ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - gateway

  # ---- lab 7: Kafka (KRaft), Kafka UI, Mailpit, order and notification ----

  kafka:
    image: apache/kafka:4.3.1
    hostname: kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
      interval: 10s
      timeout: 10s
      start_period: 60s      # a busy laptop can take a minute to start the broker
      retries: 20

  kafka-init:
    image: apache/kafka:4.3.1
    depends_on:
      kafka:
        condition: service_healthy
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
          --topic order-events --partitions 6 --replication-factor 1 &&
        /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
          --topic order-events-dlt --partitions 6 --replication-factor 1
    restart: "no"

  kafka-ui:
    image: kafbat/kafka-ui:v1.5.0
    depends_on:
      kafka:
        condition: service_healthy
    ports:
      - "8089:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: boxoffice
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092

  mailpit:
    image: axllent/mailpit:v1.31
    ports:
      - "8025:8025"
      - "1025:1025"

  order:
    build: ./services/order
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: order
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      REDIS_HOST: redis
      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      NOTIFICATION_URL: http://notification:8086
      PAYMENT_URL: http://toxiproxy:18085            # lab 9: through Toxiproxy, not payment:8085
      PAYMENT_RESILIENCE: ${PAYMENT_RESILIENCE:-true}
      PAYMENT_READ_TIMEOUT: ${PAYMENT_READ_TIMEOUT:-1s}
      PAYMENT_RETRY_ATTEMPTS: ${PAYMENT_RETRY_ATTEMPTS:-3}
      PAYMENT_BULKHEAD: ${PAYMENT_BULKHEAD:-10}
      NOTIFY_MODE: ${NOTIFY_MODE:-outbox}            # outbox (after) | sync (before)
      ADMIT_PER_SECOND: ${ADMIT_PER_SECOND:-20}      # waiting-room admission rate
    ports:
      - "8084:8084"
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
      kafka-init:
        condition: service_completed_successfully
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8084/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  notification:
    build: ./services/notification
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: notification
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      MAIL_HOST: mailpit
      NOTIFY_CONCURRENCY: ${NOTIFY_CONCURRENCY:-3}   # consumer threads (max useful: 6 partitions)
      EMAIL_LATENCY: ${EMAIL_LATENCY:-50ms}          # simulated e-mail provider latency
      NOTIFY_IDEMPOTENT: ${NOTIFY_IDEMPOTENT:-true}  # false = no dedupe (to see duplicates)
    ports:
      - "8086:8086"
    depends_on:
      db:
        condition: service_healthy
      mailpit:
        condition: service_started
      kafka-init:
        condition: service_completed_successfully
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8086/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  # ---- lab 8: observability backend (Grafana, Prometheus, Tempo, Loki, OTel Collector) ----

  otel-lgtm:
    image: grafana/otel-lgtm:0.37.0
    ports:
      - "3000:3000"   # Grafana (no login)
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
    volumes:
      - ./infra/grafana/dashboards.yaml:/otel-lgtm/grafana/conf/provisioning/dashboards/boxoffice.yaml:ro
      - ./infra/grafana/dashboards:/otel-lgtm/boxoffice-dashboards:ro
      - ./infra/grafana/alerting:/otel-lgtm/grafana/conf/provisioning/alerting:ro

  # ---- lab 9: payment adapter, fake provider and fault injection ----

  psp:
    image: wiremock/wiremock:3.13.2
    volumes:
      - ./infra/psp:/home/wiremock:ro

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.13.0
    command: ["-host=0.0.0.0", "-config=/config/toxiproxy.json"]
    volumes:
      - ./infra/toxiproxy/toxiproxy.json:/config/toxiproxy.json:ro
    ports:
      - "8474:8474"

  payment:
    build: ./services/payment
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: payment
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      PSP_URL: http://toxiproxy:18095                 # the provider, through Toxiproxy
    ports:
      - "8085:8085"
    depends_on:
      db:
        condition: service_healthy
      toxiproxy:
        condition: service_started
      psp:
        condition: service_started
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8085/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

volumes:
  db-data:
  replica-data:

Start the three new containers; --wait returns when payment is healthy (the first build takes a few minutes, the start about 30 s). Then list the proxies and charge once:

docker compose up -d --build --wait psp toxiproxy paymentcurl -s localhost:8474/proxies | jq 'keys'[
"payment",
"psp"
]curl -s -w '\n%{http_code}\n' -X POST localhost:8085/payments -H 'Content-Type: application/json' \ -H 'Idempotency-Key: test-1' -d '{"orderId":"5b0c2f9e-7d0a-4f43-9a55-1c1c2c8f8e10","amount":350.00}'{"id":"a8744598-4ef4-4d4d-85f2-105e5661233c","orderId":"5b0c2f9e-7d0a-4f43-9a55-1c1c2c8f8e10","amount":350.00,"status":"AUTHORIZED","providerRef":"ch_2zm5esh2yuqd","createdAt":"2026-10-18T18:44:51.939198Z"}
201

You should see 201 and a body like {"id":"a8744598-…","orderId":"5b0c2f9e-…","amount":350.00,"status":"AUTHORIZED","providerRef":"ch_2zm5esh2yuqd",…}. Send the same curl again: 200 instead of 201, the same id and the same providerRef.

Your first toxic

Add 3 seconds of latency to every response of the psp proxy, time a payment, remove the toxic:

curl -s -X POST localhost:8474/proxies/psp/toxics \
  -d '{"name":"slow","type":"latency","stream":"downstream","attributes":{"latency":3000}}'
time curl -s -X POST localhost:8085/payments -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: test-2' -d '{"orderId":"5b0c2f9e-7d0a-4f43-9a55-1c1c2c8f8e10","amount":350.00}'
curl -s -X DELETE localhost:8474/proxies/psp/toxics/slow

You should see {"…","status":500,"error":"Internal Server Error","path":"/payments"} after real 0m2.2s: the read timeout of PspClient (2 s) fired before the 3 s response arrived. Without it, the thread would have waited the full 3 s — or forever.

Toxic	Attributes	Simulates
`latency`	`latency`, `jitter` (ms)	a slow dependency, a congested network
`timeout`	`timeout` (ms; `0` = never close)	a black hole: packets go in, nothing comes back
`reset_peer`	`timeout` (ms)	connection reset by peer (a crash, a load balancer killing connections)
`bandwidth`	`rate` (KB/s)	a thin pipe
`limit_data`	`bytes`	a connection cut in the middle of a response
(proxy disabled)	`POST /proxies/<name>` with `{"enabled": false}`	the dependency is down: connection refused

Each toxic also takes a toxicity between 0 and 1: the probability that it applies to a new connection — perfect for "30% of the calls fail".

Next: Resilience patterns