12. AI Assistant

On sale day, support receives thousands of the same questions: when is the show? how much? where is my order? An assistant can answer them in seconds, at 3 a.m., in Portuguese and English — if it does not invent shows, leak data or cost more than the tickets. This lab builds one with the architecture of the AI systems class, running entirely on your machine with a local open model (Ollama), and optionally on Claude for comparison. In the story of the course, this lab builds chapter 13: it implements the decisions of the LLM gateway (routes and a fallback), the agent loop and its read-only tools, retrieval with pgvector and the safety mesh.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you start

Every command runs from the root of your boxoffice repository, in the state lab 11 left it:

boxoffice/
├── docker-compose.yaml     # the Compose stack of labs 1-9 (catalog behind gateway, order, ...)
├── .github/  k8s/          # labs 10-11
├── infra/  load/  scripts/
└── services/               # auth, catalog, gateway, inventory, notification, order, payment

Every file of this lab is shown in full on these pages and published under https://insper.github.io/platform/2027.1/hands-on/12/code/<path>.

Architecture

flowchart LR
    fan[fan / chat UI] -->|"POST /assistant/chat/stream"| api[assistant: FastAPI]
    api --> gin{input guard}
    gin --> orch[agent loop]
    orch -->|"search_query: ..."| emb[embedding model]
    emb --> vdb[("pgvector: assistant.documents")]
    vdb -->|top-k events| orch
    orch -->|LiteLLM router| gw[LLM gateway]
    gw -->|local| ollama["Ollama: qwen2.5:1.5b"]
    gw -.->|"optional, fallback to local"| claude[Claude API]
    orch -->|tool calls| reg[tool registry]
    reg -->|GET /catalog/events| cat[gateway -> catalog]
    reg -->|"GET /orders/id + X-User-Email"| ord[order]
    orch --> gout{output guard}
    gout -->|SSE tokens + done| fan
    api -.->|OTLP metrics| lgtm[otel-lgtm]

What you will measure

Prerequisites

• Labs 1 (catalog), 7 (order) and 8 (Grafana) done.
• Read the AI systems class: LLM gateway, agents, RAG and memory, safety and evals.

Steps

1. The service and the LLM gateway — Ollama, pgvector, the uv project, LiteLLM routes and fallback, the optional Claude route.
2. RAG with pgvector — embeddings, the HNSW index, ingestion of the catalog, retrieval evaluation.
3. Tools and the agent loop — the tool registry, the bounded loop, streaming, the API.
4. Guardrails and metrics — injection and PII on the way in, grounding on the way out; tokens, cost and TTFT in Grafana.
5. Measure it — TTFT under load, retrieval and answer quality, model choice.

Try it in the playground

Try it: run the ai-assistant scenario with the load profile. Raise the orchestrator's steps from 1 to 4 and watch the latency and the cost per request multiply — every step is another LLM call. Then enable the semantic cache of the LLM gateway and see which share of the traffic never reaches the model.

The full simulator is in the playground.

Measure it

The Measure it step has the scripts and the results table to keep in results/lab12.md.

Troubleshooting

Clean up

docker compose stop assistant ollama        # free the CPU/RAM of the model
docker compose down -v                      # or remove everything (including the downloaded models)

docker compose down -v also deletes the ollama-models volume: the next run downloads the models again. Keep it if you continue to lab 13, which can use the same local model. With native Ollama, ollama stop qwen2.5:1.5b frees the model's memory at once (it also unloads after 5 idle minutes), and ollama rm qwen2.5:1.5b nomic-embed-text deletes the models when you no longer need them.

End state

boxoffice/
├── .gitignore              # + __pycache__/, .venv/
├── docker-compose.yaml     # db/db-replica -> pgvector image; + ollama, ollama-init, assistant
├── scripts/
│   ├── assistant_load.py   # TTFT load test (uv inline script)
│   └── eval_retrieval.py   # hit@k / MRR (uv inline script)
├── results/lab12.md
└── services/assistant/     # new: pyproject.toml, uv.lock, .python-version, Dockerfile, .dockerignore,
    └── app/                # __init__, config, llm, db, rag, tools, agent, guardrails, metrics, main

Commit and tag: git add -A && git commit -m "lab 12: AI assistant" && git tag lab12.

Checklist

• ollama, ollama-init and assistant run in docker-compose.yaml (or Ollama natively, with OLLAMA_URL); db uses the pgvector image
• services/assistant is a uv project with pyproject.toml and uv.lock committed; the image builds with uv sync --frozen
• /assistant/ingest loaded at least 300 events; scripts/eval_retrieval.py reports hit@5 and MRR
• /assistant/chat and /assistant/chat/stream answer with citations checked by the output guard; an injection attempt returns 422
• get_order_status only answers for the user in X-User-Email
• assistant_* metrics (tokens, cost, TTFT) visible in Grafana
• results table filled in results/lab12.md, including where the model ran
• self-check: docker compose up -d --build assistant from a clean clone, then curl localhost:8088/assistant/health returns "status":"UP"; pushed with the tag lab12

Related

• Theory: AI systems and agents · LLM gateway · Agents and tools · RAG and memory · Safety and evals
• Case study: AI assistant
• Next lab: 13. AI in CI/CD uses language models inside the delivery pipeline of lab 11.

Start: The service and the LLM gateway