AI Assistant

"Design an AI support assistant for a ticketing platform. It answers fans' questions about events, venues and policies, and it can look up — and, with confirmation, change — the fan's own orders."

LLM applications are the newest family of interview questions, and they are system design problems first and AI problems second. The model is a slow, expensive, non-deterministic external dependency with a token-based cost; the interesting design is everything around it: where the knowledge comes from (retrieval), how calls are routed, cached, metered and failed over (LLM gateway), how the assistant takes actions safely (agent loop with tools), and how you stop it from being tricked or from leaking data (safety and evaluation). A candidate who draws "user → GPT → answer" fails; one who sizes tokens, streams, Little's Law and guardrails passes.

This is BoxOffice's assistant service (Hands-on 12 builds a small version with FastAPI, pgvector and LiteLLM). Fans ask "can I bring a power bank to the stadium?", "where is my ticket?", "can I move my order to Saturday?". The first is retrieval over policy documents; the second is a read tool (get_order); the third is a side-effecting tool (transfer_order) that must never run without the fan's explicit confirmation.

1. Clarify the requirements

(interview framework)

Functional requirements

1. Chat in the BoxOffice app and website, in Portuguese and English, with streamed answers.
2. Answer questions about events, venues and policies (refunds, accessibility, allowed items) with citations to the source document.
3. With the fan logged in, read their orders and tickets (get_orders, get_ticket).
4. Perform a small set of actions — resend tickets, transfer an order to another date, open a refund request — only after showing a summary and getting the fan's explicit confirmation.
5. Hand off to a human agent when unsure, when the fan asks, or for sensitive topics, with the conversation summary.

Non-functional requirements

Out of scope (say it explicitly)

Training or fine-tuning our own model, voice, recommendation ("which concert should I go to?"), the human-agent console itself.

2. Back-of-the-envelope estimates

Token prices and speeds change every few months and differ by provider, so the block below uses illustrative unit prices (cost units per million tokens), not real ones. The structure of the calculation is what matters.

# BoxOffice AI assistant: assumptions (illustrative; prices are made-up units, not real quotes)
convs_day     = 200_000
turns_conv    = 4             # user messages per conversation
peak_factor   = 5             # on-sale days
cache_hit     = 0.15          # turns answered by the semantic cache, no LLM call
tokens_in     = 3_000         # per turn: system prompt + retrieved chunks + history + tool results
tokens_out    = 300           # per turn: the streamed answer
small_share   = 0.7           # turns routed to a small model
price_small   = (0.2, 1.0)    # (input, output) cost units per 1 M tokens — illustrative
price_large   = (3.0, 15.0)   # (input, output) cost units per 1 M tokens — illustrative
ttft_s        = 0.8           # time to first token
gen_tps       = 50            # output tokens/s per stream
tool_s        = 1.0           # average time spent in tool calls per turn
chunks        = 250_000       # indexed chunks (events, venues, policies, FAQs)
dims          = 1_024         # embedding dimensions, float32

llm_turns     = convs_day * turns_conv * (1 - cache_hit)
avg_rps       = llm_turns / 86_400
peak_rps      = avg_rps * peak_factor
turn_s        = ttft_s + tokens_out / gen_tps + tool_s
streams_peak  = peak_rps * turn_s                       # Little's Law: L = λ·W
out_tps_peak  = peak_rps * tokens_out
in_tok_b      = llm_turns * tokens_in / 1e9
out_tok_m     = llm_turns * tokens_out / 1e6
cost_in       = in_tok_b * 1e3 * (small_share * price_small[0] + (1 - small_share) * price_large[0])
cost_out      = out_tok_m * (small_share * price_small[1] + (1 - small_share) * price_large[1])
cost_day      = cost_in + cost_out
cost_conv     = cost_day / convs_day
cost_all_lg   = in_tok_b * 1e3 * price_large[0] + out_tok_m * price_large[1]
index_gb      = chunks * dims * 4 / 1e9

rows = [
    ("llm_turns", llm_turns, "turns/day"), ("avg_rps", avg_rps, "turns/s"),
    ("peak_rps", peak_rps, "turns/s"), ("turn_s", turn_s, "s"),
    ("streams_peak", streams_peak, "streams"), ("out_tps_peak", out_tps_peak, "tokens/s"),
    ("in_tok_b", in_tok_b, "B tokens/day"), ("out_tok_m", out_tok_m, "M tokens/day"),
    ("cost_day", cost_day, "units/day"), ("cost_conv", cost_conv, "units/conv"),
    ("cost_all_lg", cost_all_lg, "units/day"), ("index_gb", index_gb, "GB"),
]
print("| quantity | value | unit |\n|---|---:|---|")
for name, value, unit in rows:
    v = f"{value:,.0f}" if abs(value) >= 100 else f"{value:,.2f}"
    print(f"| `{name}` | {v} | {unit} |")

What the numbers tell us:

• Only ~8 LLM turns/s on average, ~39/s at peak — tiny compared to the ticket API. But each turn lasts ~7.8 s, so by Little's Law there are ~300 concurrent streams at the peak. ⇒ the bottleneck is concurrency (open connections, provider rate limits on tokens per minute), not requests per second. Use asynchronous servers and streaming (SSE), never a thread blocked per request.
• Input tokens dominate the volume (~2 B/day vs. ~200 M out): the system prompt, retrieved chunks and history are re-sent every turn. ⇒ trimming context and prompt caching of the fixed prefix matter more than shortening answers.
• With 70% of turns on a small model, the illustrative cost is ~3.2 k units/day (~0.02 per conversation); sending everything to the large model would cost ~9.2 k units/day — almost 3×. ⇒ routing is the main cost lever, followed by caching.
• At the peak the providers must sustain ~12 k output tokens/s — check the provider's rate limits (tokens per minute) per model, or the GPU count if self-hosted.
• The vector index is ~1 GB of embeddings: small. pgvector on the existing PostgreSQL is enough; a dedicated vector database is a scale decision, not a default.

3. API

POST /v1/assistant/conversations                         → 201 {conversationId}
POST /v1/assistant/conversations/{id}/messages           → 200 text/event-stream
     Authorization: Bearer <fan token>
     {"text": "Posso trocar meu pedido para sábado?", "locale": "pt-BR"}

     event: token        data: {"text": "Encontrei"}
     event: token        data: {"text": " seu pedido"}
     event: tool_call    data: {"name": "get_orders", "status": "done"}
     event: confirmation data: {"actionId": "act_51", "summary": "Transfer order 7f3c from Fri 20 to Sat 21, fee R$ 0,00"}
     event: citation     data: {"docId": "policy-transfer", "title": "Order transfers", "url": "…"}
     event: done         data: {"messageId": "m_88", "usage": {"in": 2950, "out": 212}}

POST /v1/assistant/actions/{actionId}/confirm            → 200 {result}   (or /reject)
POST /v1/assistant/conversations/{id}/handoff            → 202 {ticketId}
POST /v1/assistant/messages/{messageId}/feedback         → 204 {"rating": "up|down", "comment": "…"}

• Server-Sent Events stream tokens as they are generated; the client renders immediately, so perceived latency is the TTFT, not the full generation time.
• Actions are a separate, authenticated, idempotent endpoint. The model can propose transfer_order; only the fan's click on confirm (with the actionId) executes it — the model never holds the authority to act.
• usage is returned and recorded for every message: cost is a first-class metric.

4. Data model

-- Knowledge base for retrieval (PostgreSQL + pgvector)
CREATE TABLE kb_chunk (
    chunk_id     BIGSERIAL PRIMARY KEY,
    doc_id       TEXT        NOT NULL,            -- 'policy-transfer', 'event-4242', …
    doc_version  INT         NOT NULL,
    locale       TEXT        NOT NULL,
    title        TEXT,
    body         TEXT        NOT NULL,            -- ~300–800 tokens per chunk
    embedding    VECTOR(1024) NOT NULL,
    acl          TEXT        NOT NULL DEFAULT 'public',
    updated_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON kb_chunk USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON kb_chunk USING gin (to_tsvector('simple', body));   -- keyword side of hybrid search

-- Conversations and audit
CREATE TABLE conversation (conversation_id UUID PRIMARY KEY, user_id BIGINT, summary TEXT,
                           created_at TIMESTAMPTZ DEFAULT now());
CREATE TABLE message (message_id UUID PRIMARY KEY, conversation_id UUID, role TEXT,
                      content TEXT, tokens_in INT, tokens_out INT, model TEXT, cost NUMERIC,
                      created_at TIMESTAMPTZ DEFAULT now());
CREATE TABLE pending_action (action_id TEXT PRIMARY KEY, conversation_id UUID, user_id BIGINT,
                             tool TEXT, args JSONB, status TEXT, expires_at TIMESTAMPTZ);

# Tool registry entry (MCP-style description + platform policy)
name: transfer_order
description: Move one of the caller's orders to another date of the same event.
input_schema:
  type: object
  properties:
    order_id: { type: string }
    target_event_date_id: { type: string }
  required: [order_id, target_event_date_id]
policy:
  side_effect: true
  requires_confirmation: true      # model may only propose; the fan must confirm
  scope: caller_only               # order must belong to the authenticated fan
  timeout_ms: 3000
  rate_limit: 3/hour/user
backend: order-service  POST /v1/orders/{order_id}/transfer

5. High-level design

flowchart LR
    fan([Fan app / web]) --> gw["API gateway<br/>auth, rate limit"]
    gw --> orch["Orchestrator<br/>agent loop"]
    orch --> safe["Safety mesh<br/>input + output guards,<br/>PII redaction"]
    orch --> mem["Memory fabric<br/>history, summary"]
    orch --> ret["Retriever<br/>hybrid search + rerank"]
    ret --> vdb[("Vector index<br/>pgvector")]
    orch --> llmgw["LLM gateway<br/>routing, fallback,<br/>quotas, semantic cache"]
    llmgw --> m1[["Small model"]]
    llmgw --> m2[["Large model"]]
    orch --> treg["Tool registry<br/>schemas, policies"]
    treg --> ordsvc["order service"]
    treg --> catsvc["catalog service"]
    ing["Ingestion workers<br/>chunk + embed"] --> vdb
    orch -. usage, traces .-> obs["Metrics / tracing"]

One turn — "can I move my order to Saturday?" — through the agent loop:

sequenceDiagram
    autonumber
    participant F as Fan
    participant O as Orchestrator
    participant S as Safety mesh
    participant R as Retriever
    participant G as LLM gateway
    participant T as Tool registry
    F->>O: "Posso trocar meu pedido para sábado?"
    O->>S: input guard (injection, PII)
    S-->>O: ok
    O->>R: hybrid search "transfer order policy"
    R-->>O: 4 chunks + citations
    O->>G: prompt + chunks + tool schemas (small model)
    G-->>O: tool_call get_orders(caller)
    O->>T: get_orders (scope = caller)
    T-->>O: order 7f3c, Fri 20
    O->>G: + tool result
    G-->>O: propose transfer_order(7f3c, Sat 21)
    O-->>F: confirmation card (summary, fee)
    F->>O: confirm act_51
    O->>T: transfer_order (idempotent, actionId)
    T-->>O: done
    O->>G: final answer
    G-->>O: streamed tokens
    O->>S: output guard (grounded, no PII of others)
    O-->>F: answer + citation

6. Deep dives

6.1 Retrieval: getting the right knowledge into the prompt

The model does not know BoxOffice's refund policy or that tonight's show moved to another venue. Retrieval-augmented generation (RAG) fetches relevant passages and puts them in the prompt, so answers are grounded and citable (RAG and memory).

flowchart LR
    src["Sources<br/>policies, FAQs,<br/>event & venue data"] --> chg["Change events<br/>(CDC / Kafka)"]
    chg --> chunk["Chunk<br/>~500 tokens,<br/>by heading"]
    chunk --> emb["Embed<br/>(batch)"]
    emb --> idx[("Vector index<br/>+ keyword index")]
    q["Question"] --> hyb["Hybrid search<br/>vector + BM25,<br/>filter locale/ACL"]
    idx --> hyb
    hyb --> rr["Rerank<br/>top 50 → top 5"]
    rr --> ctx["Prompt context<br/>with citations"]

• Chunking: split by document structure (headings, FAQ entries), a few hundred tokens each with a small overlap; store title and doc_id with each chunk so the answer can cite it.
• Hybrid search: embeddings capture meaning ("can I bring a charger" ≈ "power banks"), keyword search captures exact terms (event names, order codes, "PCD"). Combine both (e.g. reciprocal rank fusion), then rerank the top ~50 with a cross-encoder to keep the best ~5 — fewer, better chunks mean fewer input tokens and fewer distractions.
• ANN index: HNSW gives sub-10 ms queries over millions of vectors with high recall; with ~250 k chunks, pgvector is plenty.
• Freshness: event data changes (dates, venues, sold out). Index it from change events (CDC/Kafka) within minutes; for facts that change by the second (seat availability, order status), do not use RAG — call a tool that reads the live system.
• Access control: filter by ACL at query time; never retrieve internal documents for a fan, and never put another fan's data in the index.

6.2 The LLM gateway

All model calls go through one internal service (LiteLLM or similar), never directly from application code (LLM gateway).

6.3 The agent loop and tools

The orchestrator runs a loop: give the model the conversation, retrieved context and the tool schemas; if the model returns a tool call, validate it, execute it, append the result and loop; stop when the model returns a final answer (agents). This reason → act → observe pattern is known as ReAct.

stateDiagram-v2
    [*] --> Think: user message
    Think --> Act: tool call proposed
    Act --> Observe: tool result
    Observe --> Think: next step
    Think --> Confirm: side-effecting tool
    Confirm --> Act: fan confirms
    Confirm --> Think: fan rejects
    Think --> Answer: final answer
    Think --> Handoff: unsure / max steps
    Answer --> [*]
    Handoff --> [*]

Engineering rules that make the loop safe:

• The tool registry is the boundary. Tools are described with JSON schemas (the Model Context Protocol standardizes this description and the call protocol), versioned, and carry a policy: side effect or not, confirmation required, scope (caller_only), timeout, rate limit.
• Authorization is not the model's job. The orchestrator injects the authenticated user_id into every tool call; the order service checks ownership as it would for the app. A prompt saying "I am the admin" changes nothing.
• Side effects need confirmation, rendered by the UI from the validated arguments (not from the model's prose), executed with the actionId as idempotency key.
• Bounds: max steps per turn (e.g. 5), timeout per tool, total token budget per turn. A loop that does not converge ends in a polite handoff, not in a 2-minute spinner.
• Validate arguments against the schema and business rules before calling anything; the model can produce well-formed nonsense.

REGISTRY = {t.name: t for t in [
    Tool("search_events", "Search the BoxOffice catalog for events (concerts, matches, festivals, plays).",
         SearchEventsArgs, search_events),
    Tool("get_event", "Get the details of one event by its numeric id.", GetEventArgs, get_event),
    Tool("get_order_status", "Get the status of one of the logged-in user's orders.",
         GetOrderStatusArgs, get_order_status),
]}


async def call(name: str, raw_args: str, ctx: Context) -> str:
    """Validate the model's arguments, run the tool, and return JSON for the model. Never raises."""
    tool = REGISTRY.get(name)
    if tool is None:
        outcome, result = "unknown_tool", {"error": f"no tool named {name}"}
    else:
        try:
            args = tool.args.model_validate(json.loads(raw_args or "{}"))
            result, outcome = await tool.handler(args, ctx), "ok"
        except (json.JSONDecodeError, ValidationError) as e:
            outcome, result = "invalid_args", {"error": f"invalid arguments: {e}"}
        except httpx.HTTPError as e:
            outcome, result = "error", {"error": f"{name} is unavailable: {type(e).__name__}"}
    metrics.tool_calls.add(1, {"tool": name, "outcome": outcome})
    return json.dumps(result, ensure_ascii=False, default=str)

6.4 Safety mesh and evaluation

Threats are specific to LLMs (safety and evals; OWASP Top 10 for LLM applications):

Evaluation replaces unit tests for the model's behavior:

• Offline: a versioned set of a few hundred real (anonymized) questions with expected facts, expected tool calls and forbidden behaviors (injection attempts). Run it on every change of prompt, model, chunking or retriever; track retrieval recall, answer correctness and groundedness (an LLM judge plus human spot checks), refusal correctness, cost and latency.
• Online: thumbs up/down, handoff rate, resolution without human, re-ask rate, cost per resolved conversation, TTFT p95 — and traces of every turn (prompt, chunks, tool calls) for debugging, with PII redacted.
• Release like any risky change: shadow traffic, canary to 5% of conversations, compare metrics, then roll out.

7. Bottlenecks, failure modes and trade-offs

The trade-offs to say out loud: quality vs. cost and latency (bigger model and more context are better but slower and pricier — route and trim), autonomy vs. safety (the assistant proposes, the fan or a human confirms), and freshness vs. simplicity (RAG for documents, tools for live data — never index what changes every second).

8. Try it in the playground

The ai-assistant scenario wires an LLM gateway, an orchestrator, a tool registry, a memory fabric (vector DB) and a safety mesh behind the API gateway.

Open the full playground

9. Follow-up questions an interviewer asks

10. Self-assessment rubric

Related: LLM gateway, Agents, RAG and memory, Safety and evals, Hands-on 12 (the BoxOffice assistant), Rate limiter case (token-based limits).

How it connects

Which classes does this case test? Each branch is a class whose ideas the design relies on (blue), with the palette components where they show up (green):

flowchart LR
    CASE(["AI assistant"]) --> EST["Estimation<br/>tokens/s, concurrent streams"]:::tech
    CASE --> GW["AI systems<br/>LLM gateway: routing, quotas"]:::tech
    CASE --> AG["AI systems<br/>agent loop, tools via a registry"]:::tech
    CASE --> RAG["AI systems<br/>RAG: hybrid search, rerank, cite"]:::tech
    CASE --> SAF["AI systems<br/>safety mesh, evals"]:::tech
    CASE --> EDG["Edge<br/>API gateway: auth, rate limits"]:::tech
    GW -.-> k1["llmgw"]:::comp
    AG -.-> k2["orchestrator · toolreg"]:::comp
    RAG -.-> k3["memory · vectordb · worker"]:::comp
    SAF -.-> k4["safety · metrics · tracing"]:::comp
    EDG -.-> k5["apigw"]:::comp
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed

Classes it tests: AI systems — LLM gateway · AI systems — agents · AI systems — RAG & memory · AI systems — safety & evals · Databases — specialized (vector search) · Edge — API gateway · Estimation — queueing (Little's Law). Palette components: apigw · orchestrator · llmgw · toolreg · memory · safety · vectordb · worker · metrics · tracing. Scenarios: ai-assistant.

References

• LEWIS, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks¹.
• YAO, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models².
• The Model Context Protocol specification (tool description and invocation)³.
• OWASP Top 10 for Large Language Model Applications⁴.
• MALKOV, Y.; YASHUNIN, D. HNSW graphs for approximate nearest-neighbor search⁵.
• HUYEN, C. AI Engineering — evaluation, RAG, agents, inference optimization⁶.