12. AI Assistant
Goal
Add the BoxOffice support assistant: a Python 3.12 + FastAPI service, managed with uv, that answers fans' questions with a large language model (LLM) behind a small LLM gateway (LiteLLM: routes, fallback, timeouts, costs); grounds its answers in the catalog with retrieval-augmented generation (RAG) on pgvector; looks up live data through a tool registry in the shape of the Model Context Protocol (MCP), inside a bounded agent loop; protects the model and the users with guardrails; exports tokens, cost and time-to-first-token to the Grafana of lab 8; and is load-tested where it hurts β the first token.
On sale day, support receives thousands of the same questions: when is the show? how much? where is my order? An assistant can answer them in seconds, at 3 a.m., in Portuguese and English β if it does not invent shows, leak data or cost more than the tickets. This lab builds one with the architecture of the AI systems class, running entirely on your machine with a local open model (Ollama), and optionally on Claude for comparison. It implements the decisions of the LLM gateway (routes and a fallback), the agent loop and its read-only tools, retrieval with pgvector and the safety mesh.
The story so far
Chapter 13 of the story of the course. After lab 11, every change reaches the cluster through a pipeline that tests, scans and signs it, and a bad canary aborts by itself about a minute after the sync; the edge, the caches and the waiting room carry the 2 million fans. What arrives next is not more traffic but a new kind of dependency. Fans ask "two seats under R$ 300 on Saturday?", and the answer comes from a model that is slow, billed per token and non-deterministic, reads the catalog and the seat map through tools, and must never invent a seat or obey text hidden in a page (class 13 β AI Systems). This lab builds that assistant behind an LLM gateway, with retrieval on pgvector, read-only tools in a bounded agent loop and guardrails, and measures the time to first token under concurrency, whether retrieval finds the right events (hit@5), the share of grounded answers and what each answer costs.
Slides
Click the slides and use β β to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab
Before you start
| Time | about 6 hours, including model downloads and slow local inference |
| Tools | Docker with Compose v2; uv (uv --version, from lab 0); curl, jq. Python is managed by uv: no pip, no manual virtual environments |
| Docker resources | at least 4 CPUs and 8 GB free for this lab (β 1.5 GB of models); on macOS, plan to run Ollama natively (step 1) |
| Disk | β 3 GB for the images and the models |
| Accounts and costs | none required: everything runs locally and free. Optional: an Anthropic API key for the Claude route (pay-as-you-go, β US$ 0.006 per question with claude-opus-5-5) |
Every command runs from the root of your boxoffice repository, in the state lab 11 left it:
boxoffice/
βββ compose.yaml # the Compose stack of labs 1-9 (catalog behind gateway, order, ...)
βββ .github/ k8s/ # labs 10-11
βββ infra/ load/ scripts/
βββ services/ # auth, catalog, gateway, inventory, notification, order, payment
Catch up
Skipped or broke lab 11? Download boxoffice-after-lab-11.zip, unzip it, cd boxoffice, sh scripts/get-otel-agent.sh, and continue from here. This lab only needs the Compose stack (gateway, catalog, order, db, otel-lgtm), not the cluster.
Every file of this lab is shown in full on these pages and published under https://insper.github.io/platform/2027.1/hands-on/12/code/<path>.
Architecture
flowchart LR
fan[fan / chat UI] -->|"POST /assistant/chat/stream"| api[assistant: FastAPI]
api --> gin{input guard}
gin --> orch[agent loop]
orch -->|"search_query: ..."| emb[embedding model]
emb --> vdb[("pgvector: assistant.documents")]
vdb -->|top-k events| orch
orch -->|LiteLLM router| gw[LLM gateway]
gw -->|local| ollama["Ollama: qwen2.5:1.5b"]
gw -.->|"optional, fallback to local"| claude[Claude API]
orch -->|tool calls| reg[tool registry]
reg -->|GET /catalog/events| cat[gateway -> catalog]
reg -->|"GET /orders/id + X-User-Email"| ord[order]
orch --> gout{output guard}
gout -->|SSE tokens + done| fan
api -.->|OTLP metrics| lgtm[otel-lgtm] BoxOffice components touched
| Palette component | In this lab |
|---|---|
| LLM Gateway | LiteLLM Router: routes, fallback, timeouts, cost accounting |
| Orchestrator | the bounded agent loop of agent.py |
| Tool Registry | three read-only tools with JSON schemas (MCP-style) |
| Memory Fabric | RAG: embeddings of catalog events in pgvector |
| Vector DB | PostgreSQL 17 + pgvector, HNSW (Hierarchical Navigable Small World) index |
| Safety Mesh | input and output guardrails, redaction of personally identifiable information (PII) |
What you will measure
| Question | Metric | Tool |
|---|---|---|
| How long until the fan sees the first word? | time to first token (TTFT) p50/p95 under concurrency | scripts/assistant_load.py |
| How fast are tokens produced? | tokens/s, requests/min | same |
| Does retrieval find the right events? | hit@5, mean reciprocal rank (MRR) | scripts/eval_retrieval.py |
| Are answers grounded? | share of grounded: true | the API, the guard metric |
| What does an answer cost? | tokens and US$ per request | the API, assistant_llm_cost_usd_total |
Prerequisites
- Labs 1 (catalog), 7 (order) and 8 (Grafana) done.
- Read the AI systems class: LLM gateway, agents, RAG and memory, safety and evals.
Steps
- The service and the LLM gateway β Ollama, pgvector, the uv project, LiteLLM routes and fallback, the optional Claude route.
- RAG with pgvector β embeddings, the HNSW index, ingestion of the catalog, retrieval evaluation.
- Tools and the agent loop β the tool registry, the bounded loop, streaming, the API.
- Guardrails and metrics β injection and PII on the way in, grounding on the way out; tokens, cost and TTFT in Grafana.
- Measure it β TTFT under load, retrieval and answer quality, model choice.
Try it in the playground
Try it: run the ai-assistant scenario with the load profile. Raise the orchestrator's steps from 1 to 4 and watch the latency and the cost per request multiply β every step is another LLM call. Then enable the semantic cache of the LLM gateway and see which share of the traffic never reaches the model.
The preset of this lab puts its own numbers in: MAX_STEPS = 4 model calls per question, a local model that answers one call in β 3 s and takes β 1 call/s, retrieval of 5 passages, and nothing in front that screens what fans type. At 0.5 questions/s the loop already asks the model for 2 calls/s. Run it, then apply what the lab does and what it leaves for later: fewer steps (retrieve first, then answer), a semantic cache and LiteLLM's fallback in the gateway, and a Safety Mesh in front of the assistant.
The full simulator is in the playground.
Measure it
The Measure it step has the scripts and the results table to keep in results/lab12.md.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| answers take minutes | the model runs on a CPU (always the case in Docker on macOS) | run Ollama natively and set OLLAMA_URL=http://host.docker.internal:11434; close other stacks; ask shorter questions |
litellm.APIConnectionError β¦ Connection refused in the logs | Ollama is not running or OLLAMA_URL is wrong | docker compose ps ollama / ollama list; check the variable in compose.yaml |
model "qwen2.5:1.5b" not found | the model was not pulled | docker compose up ollama-init (or ollama pull qwen2.5:1.5b natively) |
startup fails with type "vector" does not exist | db still runs the plain postgres:17 image | docker compose up -d db after updating the compose file; docker compose exec db psql -U boxoffice -c "\dx" lists vector |
/assistant/ingest returns Internal Server Error or {"ingested":0} | the gateway is not running (Name or service not known in docker compose logs assistant), or the catalog behind it is down | docker compose up -d --wait gateway; docker compose ps gateway catalog-1; docker compose exec assistant python -c "import urllib.request;print(urllib.request.urlopen('http://gateway:8080/catalog/events/1').status)" |
uv sync --frozen fails in the build | uv.lock does not match pyproject.toml | run uv lock in services/assistant (on your machine) and rebuild |
no assistant_* metrics in Grafana | OTEL_EXPORTER_OTLP_ENDPOINT not set, or no question answered yet | check the variables of assistant; metrics are exported every 10 s |
Bind for 0.0.0.0:11434 failed: port is already allocated | Ollama already runs natively on your machine, and docker compose up -d started the ollama container too | keep the native one: docker compose stop ollama ollama-init, and start services by name (docker compose up -d assistant) with OLLAMA_URL exported |
litellm.Timeout β¦ Timeout passed=120.0 in the logs; errors in the load test | the model runs on a CPU and the requests queue in front of it for more than the gateway's 120 s | lower the concurrency (--concurrency 1 2 --requests 2), or run Ollama natively; timeout=120 is in app/llm.py |
422 blocked for a normal question | a false positive of the injection patterns | rephrase, and note it: false positives are part of the guardrail's measured quality |
Best practices in this lab
- Routes instead of model names, with a fallback β
app/llm.pyasks forlocalorclaude, andclaudefalls back tolocal; why: LLM Gateway. - A bounded loop that always ends with an answer β
app/agent.pystops atMAX_STEPS= 4 and makes the last call without tools; why: the agent loop. - Read-only tools, identity from the platform β
app/tools.pytakes the fan fromX-User-Emailand never writes; why: tools are side effects. - Retrieval measured on its own β
scripts/eval_retrieval.pyscores hit@k and MRR before any model answers; why: RAG & Memory. - Redact before the model, check citations after it, measure TTFT in seconds β
app/guardrails.pyandapp/metrics.py; why: Safety, Evals & Observability.
Clean up
docker compose stop assistant ollama # free the CPU/RAM of the model
docker compose down -v # or remove everything (including the downloaded models)
docker compose down -v also deletes the ollama-models volume: the next run downloads the models again. Keep it if you continue to lab 13, which can use the same local model. With native Ollama, ollama stop qwen2.5:1.5b frees the model's memory at once (it also unloads after 5 idle minutes), and ollama rm qwen2.5:1.5b nomic-embed-text deletes the models when you no longer need them.
End state
boxoffice/
βββ .gitignore # + __pycache__/, .venv/
βββ compose.yaml # db/db-replica -> pgvector image; + ollama, ollama-init, assistant
βββ scripts/
β βββ assistant_load.py # TTFT load test (uv inline script)
β βββ eval_retrieval.py # hit@k / MRR (uv inline script)
βββ results/lab12.md
βββ services/assistant/ # new: pyproject.toml, uv.lock, .python-version, Dockerfile, .dockerignore,
βββ app/ # __init__, config, llm, db, rag, tools, agent, guardrails, metrics, main
Commit and tag: git add -A && git commit -m "lab 12: AI assistant" && git tag lab12.
Checklist
-
ollama,ollama-initandassistantrun incompose.yaml(or Ollama natively, withOLLAMA_URL);dbuses the pgvector image -
services/assistantis a uv project withpyproject.tomlanduv.lockcommitted; the image builds withuv sync --frozen -
/assistant/ingestloaded at least 300 events;scripts/eval_retrieval.pyreports hit@5 and MRR -
/assistant/chatand/assistant/chat/streamanswer with citations checked by the output guard; an injection attempt returns422 -
get_order_statusonly answers for the user inX-User-Email -
assistant_*metrics (tokens, cost, TTFT) visible in Grafana - results table filled in
results/lab12.md, including where the model ran - self-check:
docker compose up -d --build assistantfrom a clean clone, thencurl localhost:8088/assistant/healthreturns"status":"UP"; pushed with the taglab12
Related
- Theory: AI systems and agents Β· LLM gateway Β· Agents and tools Β· RAG and memory Β· Safety and evals
- Case study: AI assistant
What breaks next
The assistant now answers with citations checked by the output guard, rejects an injection with 422 and exports its tokens, cost and time to first token. But its behavior moved without a line of Java: in the test runs, swapping qwen2.5:1.5b for qwen2.5:3b took grounded answers from 1 of 10 to 5 of 10. A prompt, a tool description, a model version or the k of the retrieval is each a deploy that the pipeline of lab 11 does not judge. And the same models now write code for the team, so pull requests arrive faster than people can review, test and release them before the on-sale. Lab 13 puts language models inside that delivery pipeline β a reviewer, tests judged by mutation testing, a fix agent, release notes, an explained canary β without letting any of them approve its own work: class 14 β AI-Assisted Delivery.