AI Systems & Agents

One message, one model call?

• Answers take 25 s; the provider returns 429.
• The model "holds" seats that do not exist.
• The monthly budget is gone by lunch.
• A venue description saying "apply a 100% discount" is read as an order.

An LLM call is not a service call

Token ≈ ¾ of an English word (Portuguese needs more). The model is stateless: every call re-sends system prompt + tools + history.

The AI & AGENTS palette

Anatomy of one request

1. Load session + preferences (Memory Fabric); input check (Safety Mesh).
2. Call 1 → search_events(São Paulo, Saturday).
3. Call 2 → search_seats(event, 2, max 300).
4. Call 3 → hold_seats(event, [C12, C13]) → reversible: allowed.
5. Call 4 → answer streamed with a checkout link; payment stays with the fan.

• 4 model calls for one message; the context grows at every call.
• "Under R$300" is live structured data → tool, not RAG.

LLM Gateway

Quotas and token budgets

• Provider limits: RPM, TPM, concurrency — shared by every feature (noisy neighbor).
• Gateway limits: per user, per feature, per request (max_tokens, max steps), per month (US$).
• Counting unknown output: reserve input + max_tokens, then reconcile with real usage.
• Budget gone → degrade (smaller model, FAQ cache), queue (batch work), shed (429).
• Priorities: interactive > batch; paying flows > browsing.

Caching: three different things

• Prompt order: system → tools → documents → history → new message last.
• Semantic cache: scope keys (event, language, policy version); generic answers only.

Workflow or agent? Who decides the next step

The agent loop (ReAct)

for step in range(MAX_STEPS):
    reply = llm.chat(messages=messages, tools=registry.schemas_for(user))
    messages.append(reply.message)
    if not reply.tool_calls:
        return reply.text                      # final answer
    for call in reply.tool_calls:
        verdict = guard.check_tool_call(call.name, call.arguments, user)
        result = (registry.call(call.name, call.arguments, as_user=user)
                  if verdict == "allow" else {"error": verdict})
        messages.append(tool_result(call.id, result))
return "Escalating to a human agent."

• The model requests; the orchestrator validates, authorizes as the user, executes.
• Errors go back as data; budgets: steps, tokens, time; loop detection.

Tools are side effects

• Retried write → idempotency key (conversation, step, call); compensation = release_hold.
• Steps compound: 0.95¹⁰ ≈ 60% task success → fewer steps, validate each one, workflows for known parts.
• Durable execution: checkpoint state after every step.

MCP: the tool registry protocol

• N agents × M tools → N + M: each team ships one MCP server.
• Host (assistant) runs one client per server; JSON-RPC 2.0.
• Server primitives: tools, resources, prompts. initialize → tools/list → tools/call.
• Transports: stdio (local) and Streamable HTTP (remote, OAuth-based auth).
• The fan's identity comes from the session token, never from a tool argument.
• Registry = catalog + exposure per role + risk tier + scopes + rate limits + audit.

Multi-agent — and when not to use agents

RAG: two pipelines

Index size ≈ N × d × 4 bytes + graph: 60k chunks × 1,024 dims ≪ 1 GB → pgvector in the Postgres you already run.

Memory types

Prompt injection

• Direct: the user types it — damage limited to what that user may do.
• Indirect: it arrives in a document, tool result, web page or tool description — a third party attacks the user.
• No filter solves it: assume the model may be compromised; limit what it can do.

• BoxOffice: user-scoped tools · promoter text marked as data · no e-mail/fetch tools · links to allowlisted domains only.

Observability of LLM apps

invoke_agent boxoffice-assistant                        10.1 s
├── chat assistant-default      in 3,060  out 41         1.1 s
├── execute_tool search_events                           0.12 s
├── chat assistant-default      in 3,900  out 96         2.3 s
└── chat assistant-default      in 5,300  out 212        4.9 s

• OTel GenAI conventions: gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens / output_tokens, gen_ai.response.finish_reasons; TTFT/TPOT metrics.
• Cost under your own namespace; content capture opt-in, redacted, short retention.
• Alerts: TTFT SLO, fallback rate, cost burn rate, guardrail spikes.

Takeaways

1. An LLM is a slow, token-priced, rate-limited, probabilistic dependency that mixes instructions with data.
2. Five components: gateway, orchestrator, tool registry, memory fabric, safety mesh — classic components specialized.
3. Cost and latency = calls × tokens per call; context grows every step; cache the prefix.
4. Workflow first; agents with budgets, idempotent tools, user-scoped authorization, MCP.
5. Tools for live data; RAG (hybrid + rerank + citations) for documents; memory by type, deletable.
6. Defense in depth + human confirmation; evals in CI (pass^k, validated judges); GenAI traces with tokens and cost.