Skip to content

1. The Service and the LLM Gateway

This step creates the assistant service — Python 3.12 + FastAPI, managed with uv — and its LLM gateway: the one module that knows which models exist, which one answers by default, what to do when a provider fails, and how much each answer costs. All commands run from the root of your boxoffice repository unless a cd says otherwise.

Infrastructure: a local LLM and a vector-capable PostgreSQL

Container Image Port Role
ollama ollama/ollama:0.40.6 11434 runs the local models: qwen2.5:1.5b (chat, with tool calling) and nomic-embed-text (embeddings, 768 dimensions)
ollama-init ollama/ollama:0.40.6 — one-shot: downloads both models (≈ 1.3 GB, once)
db and db-replica pgvector/pgvector:0.8.7-pg17-trixie 5432 / 5433 the same PostgreSQL 17, now with the vector extension available
assistant services/assistant 8088 the FastAPI service of this lab

The new part of docker-compose.yaml:

docker-compose.yaml (excerpt)
  ollama:
    image: ollama/ollama:0.40.6
    ports:
      - "11434:11434"
    volumes:
      - ollama-models:/root/.ollama      # models survive restarts (about 1.3 GB)
    healthcheck:                         # ollama-init must wait for the server, not just the container
      test: ["CMD", "ollama", "list"]
      interval: 5s
      timeout: 5s
      retries: 12

  # one-shot: download the chat model and the embedding model, then exit
  ollama-init:
    image: ollama/ollama:0.40.6
    environment:
      OLLAMA_HOST: ollama:11434
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        ollama pull ${LOCAL_MODEL_TAG:-qwen2.5:1.5b} && ollama pull nomic-embed-text
    depends_on:
      ollama:
        condition: service_healthy
    restart: "no"

Get the complete file (lab 9's plus the pgvector image for db/db-replica, ollama, ollama-init and assistant):

curl -fsSL -o docker-compose.yaml https://insper.github.io/platform/2027.1/hands-on/12/code/docker-compose.yaml
docker-compose.yaml — the complete file at the end of this lab
docker-compose.yaml
name: boxoffice

# lab 8: OpenTelemetry settings shared by every Java service (merged into each "environment")
x-otel: &otel
  JAVA_TOOL_OPTIONS: -javaagent:/otel/opentelemetry-javaagent.jar
  OTEL_JAVAAGENT_ENABLED: ${OTEL_ON:-true}          # false = same image, agent off
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-lgtm:4318
  OTEL_METRIC_EXPORT_INTERVAL: "10000"              # 10 s instead of 60 s
  OTEL_RESOURCE_ATTRIBUTES: service.namespace=boxoffice,deployment.environment.name=local
  OTEL_TRACES_SAMPLER: ${OTEL_TRACES_SAMPLER:-parentbased_always_on}
  OTEL_TRACES_SAMPLER_ARG: ${OTEL_TRACES_SAMPLER_ARG:-1.0}
  LOGGING_PATTERN_CORRELATION: "[%X{trace_id:-},%X{span_id:-}] "   # trace id in console logs

# settings shared by the three catalog replicas (a YAML anchor)
x-catalog: &catalog
  build: ./services/catalog
  image: boxoffice-catalog
  volumes:
    - ./infra/otel:/otel:ro          # lab 8: the Java agent
  environment:
    <<: *otel
    OTEL_SERVICE_NAME: catalog       # three replicas, ONE service
    DB_HOST: db
    DB_PORT: 5432
    DB_NAME: ${DB_NAME:-boxoffice}
    DB_USER: ${DB_USER:-boxoffice}
    DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
    DB_POOL_SIZE: ${DB_POOL_SIZE:-10}
    REDIS_HOST: redis
    REDIS_PORT: 6379
    CACHE_ENABLED: ${CACHE_ENABLED:-true}
    CACHE_TTL_EVENT: ${CACHE_TTL_EVENT:-60s}
    CACHE_TTL_LIST: ${CACHE_TTL_LIST:-10s}
    CACHE_STAMPEDE_PROTECTION: ${CACHE_STAMPEDE_PROTECTION:-false}
  depends_on:
    db:
      condition: service_healthy
    redis:
      condition: service_healthy
  deploy:
    resources:
      limits:
        cpus: "1.0"
        memory: 768M
  healthcheck:
    test: ["CMD-SHELL", "wget -qO- http://localhost:8082/actuator/health/readiness || exit 1"]
    interval: 10s
    timeout: 3s
    start_period: 120s       # lab 8: with the agent, JVMs starting together need up to ~90 s
    retries: 3

services:

  db:
    image: pgvector/pgvector:0.8.7-pg17-trixie   # lab 12: PostgreSQL 17 + the pgvector extension
    command: ["postgres", "-c", "max_connections=${DB_MAX_CONNECTIONS:-100}"]
    environment:
      POSTGRES_DB: ${DB_NAME:-boxoffice}
      POSTGRES_USER: ${DB_USER:-boxoffice}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-boxoffice}
      REPLICATION_PASSWORD: ${REPLICATION_PASSWORD:-replicator}
    ports:
      - "5432:5432"
    volumes:
      - db-data:/var/lib/postgresql/data
      # runs only when the volume is new: creates the "replicator" user
      - ./infra/postgres/10-replication.sh:/docker-entrypoint-initdb.d/10-replication.sh:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 3s
      retries: 10

  # read-only streaming replica of db (hot standby)
  db-replica:
    image: pgvector/pgvector:0.8.7-pg17-trixie   # lab 12: PostgreSQL 17 + the pgvector extension
    user: postgres
    entrypoint: ["/usr/local/bin/replica-entrypoint.sh"]
    environment:
      PGPASSWORD: ${REPLICATION_PASSWORD:-replicator}     # used by pg_basebackup
      REPLICA_DELAY: ${REPLICA_DELAY:-0}                   # e.g. 2s: artificial lag
    ports:
      - "5433:5432"
    volumes:
      - replica-data:/var/lib/postgresql/data
      - ./infra/postgres/replica-entrypoint.sh:/usr/local/bin/replica-entrypoint.sh:ro
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U boxoffice -d boxoffice"]
      interval: 5s
      timeout: 3s
      retries: 20

  pgbouncer:
    image: edoburu/pgbouncer:v1.26.0-p0
    ports:
      - "6432:6432"
    volumes:
      - ./infra/pgbouncer/pgbouncer.ini:/etc/pgbouncer/pgbouncer.ini:ro
      - ./infra/pgbouncer/userlist.txt:/etc/pgbouncer/userlist.txt:ro
    depends_on:
      db:
        condition: service_healthy

  redis:
    image: redis:8-alpine
    # a pure cache: bounded memory, evict least-recently-used keys, no persistence
    command: ["redis-server", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru",
              "--save", "", "--appendonly", "no"]
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

  # three identical replicas; no "ports:" — only the edge is reachable from outside
  catalog-1:
    <<: *catalog
    hostname: catalog-1
  catalog-2:
    <<: *catalog
    hostname: catalog-2
  catalog-3:
    <<: *catalog
    hostname: catalog-3

  inventory:
    build: ./services/inventory
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: inventory
      DB_HOST: ${INVENTORY_DB_HOST:-pgbouncer}      # writes go through PgBouncer...
      DB_PORT: ${INVENTORY_DB_PORT:-6432}
      DB_MIGRATION_HOST: db                         # ...migrations straight to the primary
      DB_REPLICA_HOST: ${DB_REPLICA_HOST:-db-replica}
      DB_REPLICA_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      DB_POOL_SIZE: ${INVENTORY_POOL_SIZE:-20}
      INVENTORY_LOCKING: ${INVENTORY_LOCKING:-pessimistic}
      INVENTORY_HOLD_TTL: ${INVENTORY_HOLD_TTL:-PT10M}
    ports:
      - "8083:8083"            # published for experiments; clients go through the edge
    depends_on:
      db:
        condition: service_healthy
      db-replica:
        condition: service_healthy
      pgbouncer:
        condition: service_started
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8083/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  auth:
    build: ./services/auth
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: auth
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      JWT_ISSUER: ${JWT_ISSUER:-boxoffice-auth}
      JWT_TTL: ${JWT_TTL:-PT1H}
      ADMIN_EMAIL: ${ADMIN_EMAIL:-admin@boxoffice.dev}
      ADMIN_PASSWORD: ${ADMIN_PASSWORD:-boxoffice-admin}
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8081/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  gateway:
    build: ./services/gateway
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: gateway
      AUTH_URL: http://auth:8081
      INVENTORY_URL: http://inventory:8083
      AUTH_JWKS_URI: http://auth:8081/auth/.well-known/jwks.json
      JWT_ISSUER: ${JWT_ISSUER:-boxoffice-auth}
    ports:
      - "8080:8080"          # published for experiments; clients use the edge (port 80)
    depends_on:
      - auth
      - inventory
      - catalog-1
      - catalog-2
      - catalog-3
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8080/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  edge:
    image: nginx:1.30-alpine
    ports:
      - "80:80"
    volumes:
      - ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - gateway

  # ---- lab 7: Kafka (KRaft), Kafka UI, Mailpit, order and notification ----

  kafka:
    image: apache/kafka:4.3.1
    hostname: kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"]
      interval: 10s
      timeout: 10s
      start_period: 60s      # a busy laptop can take a minute to start the broker
      retries: 20

  kafka-init:
    image: apache/kafka:4.3.1
    depends_on:
      kafka:
        condition: service_healthy
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
          --topic order-events --partitions 6 --replication-factor 1 &&
        /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists \
          --topic order-events-dlt --partitions 6 --replication-factor 1
    restart: "no"

  kafka-ui:
    image: kafbat/kafka-ui:v1.5.0
    depends_on:
      kafka:
        condition: service_healthy
    ports:
      - "8089:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: boxoffice
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092

  mailpit:
    image: axllent/mailpit:v1.31
    ports:
      - "8025:8025"
      - "1025:1025"

  order:
    build: ./services/order
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: order
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      REDIS_HOST: redis
      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      NOTIFICATION_URL: http://notification:8086
      PAYMENT_URL: http://toxiproxy:18085            # lab 9: through Toxiproxy, not payment:8085
      PAYMENT_RESILIENCE: ${PAYMENT_RESILIENCE:-true}
      PAYMENT_READ_TIMEOUT: ${PAYMENT_READ_TIMEOUT:-1s}
      PAYMENT_RETRY_ATTEMPTS: ${PAYMENT_RETRY_ATTEMPTS:-3}
      PAYMENT_BULKHEAD: ${PAYMENT_BULKHEAD:-10}
      NOTIFY_MODE: ${NOTIFY_MODE:-outbox}            # outbox (after) | sync (before)
      ADMIT_PER_SECOND: ${ADMIT_PER_SECOND:-20}      # waiting-room admission rate
    ports:
      - "8084:8084"
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
      kafka-init:
        condition: service_completed_successfully
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8084/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  notification:
    build: ./services/notification
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: notification
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      MAIL_HOST: mailpit
      NOTIFY_CONCURRENCY: ${NOTIFY_CONCURRENCY:-3}   # consumer threads (max useful: 6 partitions)
      EMAIL_LATENCY: ${EMAIL_LATENCY:-50ms}          # simulated e-mail provider latency
      NOTIFY_IDEMPOTENT: ${NOTIFY_IDEMPOTENT:-true}  # false = no dedupe (to see duplicates)
    ports:
      - "8086:8086"
    depends_on:
      db:
        condition: service_healthy
      mailpit:
        condition: service_started
      kafka-init:
        condition: service_completed_successfully
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8086/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  # ---- lab 8: observability backend (Grafana, Prometheus, Tempo, Loki, OTel Collector) ----

  otel-lgtm:
    image: grafana/otel-lgtm:0.37.0
    ports:
      - "3000:3000"   # Grafana (no login)
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
    volumes:
      - ./infra/grafana/dashboards.yaml:/otel-lgtm/grafana/conf/provisioning/dashboards/boxoffice.yaml:ro
      - ./infra/grafana/dashboards:/otel-lgtm/boxoffice-dashboards:ro
      - ./infra/grafana/alerting:/otel-lgtm/grafana/conf/provisioning/alerting:ro

  # ---- lab 9: payment adapter, fake provider and fault injection ----

  psp:
    image: wiremock/wiremock:3.13.2
    volumes:
      - ./infra/psp:/home/wiremock:ro

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.13.0
    command: ["-host=0.0.0.0", "-config=/config/toxiproxy.json"]
    volumes:
      - ./infra/toxiproxy/toxiproxy.json:/config/toxiproxy.json:ro
    ports:
      - "8474:8474"

  payment:
    build: ./services/payment
    volumes:
      - ./infra/otel:/otel:ro
    environment:
      <<: *otel
      OTEL_SERVICE_NAME: payment
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      PSP_URL: http://toxiproxy:18095                 # the provider, through Toxiproxy
    ports:
      - "8085:8085"
    depends_on:
      db:
        condition: service_healthy
      toxiproxy:
        condition: service_started
      psp:
        condition: service_started
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 768M
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8085/actuator/health/readiness || exit 1"]
      interval: 10s
      timeout: 3s
      start_period: 120s
      retries: 3

  # ---- lab 12: AI assistant (FastAPI + LiteLLM + pgvector) and a local LLM ----

  ollama:
    image: ollama/ollama:0.40.6
    ports:
      - "11434:11434"
    volumes:
      - ollama-models:/root/.ollama      # models survive restarts (about 1.3 GB)
    healthcheck:                         # ollama-init must wait for the server, not just the container
      test: ["CMD", "ollama", "list"]
      interval: 5s
      timeout: 5s
      retries: 12

  # one-shot: download the chat model and the embedding model, then exit
  ollama-init:
    image: ollama/ollama:0.40.6
    environment:
      OLLAMA_HOST: ollama:11434
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        ollama pull ${LOCAL_MODEL_TAG:-qwen2.5:1.5b} && ollama pull nomic-embed-text
    depends_on:
      ollama:
        condition: service_healthy
    restart: "no"

  assistant:
    build: ./services/assistant
    environment:
      DB_HOST: db
      DB_PORT: 5432
      DB_NAME: ${DB_NAME:-boxoffice}
      DB_USER: ${DB_USER:-boxoffice}
      DB_PASSWORD: ${DB_PASSWORD:-boxoffice}
      OLLAMA_URL: ${OLLAMA_URL:-http://ollama:11434}   # macOS: http://host.docker.internal:11434 (native Ollama)
      LOCAL_MODEL: ollama_chat/${LOCAL_MODEL_TAG:-qwen2.5:1.5b}
      CATALOG_URL: http://gateway:8080                  # GET /catalog/** is public at the gateway
      ORDER_URL: http://order:8084
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}          # optional: empty = local model only
      OTEL_SERVICE_NAME: assistant
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-lgtm:4318
      OTEL_METRIC_EXPORT_INTERVAL: "10000"
      OTEL_RESOURCE_ATTRIBUTES: service.namespace=boxoffice,deployment.environment.name=local
    ports:
      - "8088:8088"
    extra_hosts:
      - "host.docker.internal:host-gateway"             # reach a native Ollama on the host
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 1G
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8088/assistant/health')"]
      interval: 10s
      timeout: 5s
      start_period: 30s
      retries: 3

volumes:
  db-data:
  replica-data:
  ollama-models:

Where should the model run? (read before pulling)

A language model is a lot of arithmetic; where it runs decides your latency.

Docker on macOS has no GPU access: in a container the model runs on the CPU and a short answer can take a minute. Run Ollama natively instead (it uses the GPU through Metal) and point the assistant at it:

brew install ollama && ollama serve &          # or the app from ollama.com
ollama pull qwen2.5:1.5b && ollama pull nomic-embed-text
export OLLAMA_URL=http://host.docker.internal:11434

and skip the ollama and ollama-init containers below.

Install the NVIDIA Container Toolkit and add to the ollama service: deploy: { resources: { reservations: { devices: [{ driver: nvidia, count: all, capabilities: [gpu] }] } } }.

Use the containers as they are. It works, slowly: in a test run on a busy 8-core laptop the first token took tens of seconds. Prefer short questions, and compare with the optional Claude route of this page.

Start the infrastructure (the db image changes, so db restarts once):

docker compose up -d db db-replica ollama ollama-initdocker compose wait ollama-initcontainer "c8d93485…" exited with status code 0docker compose exec ollama ollama listNAME ID SIZE MODIFIED
nomic-embed-text:latest 0a109f422b47 274 MB 7 seconds ago
qwen2.5:1.5b 65ec06548149 986 MB 29 seconds ago

docker compose wait blocks until the one-shot ollama-init has downloaded both models (a minute or two the first time) and prints its exit code: 0. (docker compose logs -f ollama-init would not return if the container has already exited.) You should see both models in the list (ollama list on your machine if Ollama runs natively). With native Ollama (macOS), start only docker compose up -d db db-replica: an ollama container would fail with port is already allocated on 11434, which the native server already uses.

The project: uv

uv (from Astral) creates the virtual environment, resolves the dependencies and writes a lock file with the exact version and hash of every package, so that your laptop, CI and the container install the same bytes. The project was created with these commands (you can run them yourself in an empty services/assistant/, or download the files below):

mkdir -p services/assistant && cd services/assistant
uv init --app --name assistant --python 3.12 --no-readme --vcs none .
rm main.py
uv add "fastapi>=0.146" "uvicorn[standard]>=0.54" "litellm>=1.106" "psycopg[binary,pool]>=3.3" \
  "pgvector>=0.5" "numpy>=2.3" "httpx>=0.28" "pydantic>=2.14" "mcp>=1.20" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"
cd ../..

uv add edits pyproject.toml and writes uv.lock (≈ 3 000 lines: commit it, never edit it by hand; download it from https://insper.github.io/platform/2027.1/hands-on/12/code/services/assistant/uv.lock if you prefer the exact versions of this handout). uv does not write the Dockerfile and the .dockerignore: create them from the tabs below. If you skip uv init, also download pyproject.toml from the same address and create .python-version from its tab (the site does not serve files whose names start with a dot).

services/assistant/pyproject.toml
[project]
name = "assistant"
version = "1.0.0"
description = "BoxOffice AI support assistant (lab 12)"
requires-python = ">=3.12"
dependencies = [
    "fastapi>=0.146",
    "httpx>=0.28",
    "litellm>=1.106",
    "mcp>=1.20",
    "numpy>=2.3",
    "opentelemetry-exporter-otlp-proto-http>=1.45",
    "opentelemetry-sdk>=1.45",
    "pgvector>=0.5",
    "psycopg[binary,pool]>=3.3",
    "pydantic>=2.14",
    "uvicorn[standard]>=0.54",
]
services/assistant/.python-version
3.12
services/assistant/Dockerfile
# syntax=docker/dockerfile:1
FROM python:3.12-slim

# uv: the project's package and environment manager (lab 0); one static binary
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

WORKDIR /app
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy

# dependencies first (cached layer): exactly the versions of uv.lock, without dev tools
COPY pyproject.toml uv.lock .python-version ./
RUN --mount=type=cache,target=/root/.cache/uv uv sync --frozen --no-dev --no-install-project

COPY app ./app
RUN useradd --system --uid 10001 app
USER app
EXPOSE 8088
CMD ["/app/.venv/bin/uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8088"]
services/assistant/.dockerignore
.venv/
__pycache__/
*.pyc
.pytest_cache/

uv sync --frozen fails if uv.lock does not match pyproject.toml — the build never resolves versions on its own. (mcp is listed because LiteLLM imports it when a request carries tools.)

Running Python locally (step 4 does) writes __pycache__/ folders next to the code; add two lines to the root .gitignore so that git add -A leaves them, and any .venv/, out:

.gitignore
# build output
target/
*.class
# IDEs and OS
.idea/
.vscode/
*.iml
.DS_Store
# local secrets: never commit them
.env
# k6 HTML reports are big; commit the JSON summaries and the tables in results/*.md
results/*.html
# the OpenTelemetry Java agent is downloaded by scripts/get-otel-agent.sh (lab 8)
infra/otel/*.jar
# Python (lab 12): bytecode caches and local environments
__pycache__/
.venv/

Configuration and the gateway

services/assistant/app/config.py
"""All settings come from environment variables (the same names in Compose and in Kubernetes)."""
import os
from dataclasses import dataclass


@dataclass(frozen=True)
class Settings:
    db_dsn: str
    ollama_url: str
    local_model: str        # LiteLLM model name of the local chat model
    embed_model: str        # LiteLLM model name of the embedding model
    embed_dim: int          # dimension of the embedding vectors (nomic-embed-text: 768)
    claude_model: str | None  # set only when ANTHROPIC_API_KEY exists
    default_route: str      # "local" or "claude"
    catalog_url: str
    order_url: str
    max_steps: int          # agent loop: maximum LLM calls per question
    max_input_chars: int
    retrieve_k: int


def load() -> Settings:
    env = os.environ.get
    has_key = bool(env("ANTHROPIC_API_KEY"))
    return Settings(
        db_dsn=(
            f"host={env('DB_HOST', 'db')} port={env('DB_PORT', '5432')} "
            f"dbname={env('DB_NAME', 'boxoffice')} user={env('DB_USER', 'boxoffice')} "
            f"password={env('DB_PASSWORD', 'boxoffice')}"
        ),
        ollama_url=env("OLLAMA_URL", "http://ollama:11434"),
        local_model=env("LOCAL_MODEL", "ollama_chat/qwen2.5:1.5b"),
        embed_model=env("EMBED_MODEL", "ollama/nomic-embed-text"),
        embed_dim=int(env("EMBED_DIM", "768")),
        claude_model=env("CLAUDE_MODEL", "anthropic/claude-opus-5-5") if has_key else None,
        default_route=env("ASSISTANT_ROUTE", "claude" if has_key else "local"),
        catalog_url=env("CATALOG_URL", "http://gateway:8080"),
        order_url=env("ORDER_URL", "http://order:8084"),
        max_steps=int(env("MAX_STEPS", "4")),
        max_input_chars=int(env("MAX_INPUT_CHARS", "2000")),
        retrieve_k=int(env("RETRIEVE_K", "5")),
    )


settings = load()
services/assistant/app/llm.py
"""The LLM gateway: one place that knows models, routes, fallbacks, timeouts and costs."""
import litellm
from litellm import Router

from .config import settings

litellm.drop_params = True  # drop parameters a provider does not support instead of failing

model_list = [{
    "model_name": "local",
    "litellm_params": {"model": settings.local_model, "api_base": settings.ollama_url},
}]
fallbacks = []
if settings.claude_model:  # optional: only when ANTHROPIC_API_KEY is set
    model_list.append({"model_name": "claude", "litellm_params": {"model": settings.claude_model}})
    fallbacks = [{"claude": ["local"]}]  # provider down or out of quota: answer with the local model

router = Router(model_list=model_list, fallbacks=fallbacks, num_retries=1, timeout=120)


def real_model(route: str) -> str:
    """The provider model behind a route name (for metrics and costs)."""
    return settings.claude_model if route == "claude" and settings.claude_model else settings.local_model


async def embed(texts: list[str]) -> list[list[float]]:
    response = await litellm.aembedding(model=settings.embed_model, input=texts, api_base=settings.ollama_url)
    return [item["embedding"] for item in response.data]
services/assistant/app/__init__.py
"""BoxOffice AI support assistant (lab 12)."""

LiteLLM gives every provider the same OpenAI-style interface, and its Router is the gateway of the LLM gateway class:

Concern Here
Routing two routes: local (Ollama) always, claude only when ANTHROPIC_API_KEY is set; ASSISTANT_ROUTE or the request's route field chooses
Fallback claude → local when the provider fails (outage, rate limit, no credit)
Timeouts and retries 120 s per call, 1 retry — an LLM call is the slowest dependency BoxOffice has
Cost app/metrics.py (step 4) multiplies the tokens of each answer by the provider's price

Why an in-process SDK and not the LiteLLM proxy?

LiteLLM also runs as a separate proxy (an OpenAI-compatible HTTP server with keys, budgets and a UI) shared by many services. One service does not need it: the in-process Router has the same routing logic without another container. With several AI services, the proxy becomes the shared gateway — the same move as from per-service resilience (lab 9) to a service mesh.

The optional Claude route (costs money)

The lab works end to end with the local model. To compare with a frontier model, create an API key at console.anthropic.com (pay-as-you-go: you are charged per token — the default anthropic/claude-opus-5-5 costs US$ 4 per million input tokens and US$ 20 per million output tokens; one question of this lab uses about 1 000 input and 100 output tokens, ≈ US$ 0.006) and export it before starting the assistant:

export ANTHROPIC_API_KEY=sk-ant-...       # never commit it; docker-compose.yaml reads it from the shell

With the key, the default route becomes claude; send "route": "local" in a request to compare. Without it, nothing calls the paid API.

Start it

Steps 2 and 3 explain the remaining files of app/; download them all now so the service starts:

mkdir -p services/assistant/app
for f in __init__ config llm db rag tools guardrails agent metrics main; do
  curl -fsSL -o services/assistant/app/$f.py \
    https://insper.github.io/platform/2027.1/hands-on/12/code/services/assistant/app/$f.py
done
docker compose up -d --wait gateway order otel-lgtm      # what the assistant calls, with their dependencies
docker compose up -d --build --wait assistant
curl -s localhost:8088/assistant/health

You should see {"status":"UP","documents":0,"route":"local"} ("route":"claude" if you exported a key). documents is 0: the knowledge base is empty until step 2. Start the services by name, as above: a bare docker compose up -d also starts the ollama container, which fails next to a native Ollama.

Route claude without meaning to?

Then ANTHROPIC_API_KEY is already set in your shell (other tools use the same variable), and every question goes to the paid API. unset ANTHROPIC_API_KEY and run docker compose up -d --wait assistant again: the health check must show "route":"local".

Next: RAG with pgvector