LLM Gateway

Six months after the assistant launched, BoxOffice has five features that call models: the support assistant, a review summarizer, an event-description translator, a fraud-signal classifier and the nightly job that embeds the catalog. Each team picked its own provider, wrote its own retry loop and pasted an API key into its own configuration. Then three things happened in the same week: a provider had a two-hour outage and three features went down with it; the translator, stuck in a retry loop, burned a month of budget in one night; and nobody could answer the CFO's question, "how much does the assistant cost per conversation?".

The fix is the same one the course applied to HTTP traffic in Traffic & Edge: put one gateway in front of every model, and move the cross-cutting concerns into it.

flowchart LR
    subgraph Before["Before: every service talks to every provider"]
        a1["assistant"] --> p1["provider A"]
        a2["summarizer"] --> p1
        a2 --> p2["provider B"]
        a3["translator"] --> p2
        a4["embed job"] --> p3["self-hosted"]
    end
    subgraph After["After: one LLM gateway"]
        b1["assistant"] --> gw["LLM Gateway"]
        b2["summarizer"] --> gw
        b3["translator"] --> gw
        b4["embed job"] --> gw
        gw --> q1["provider A"]
        gw --> q2["provider B"]
        gw --> q3["self-hosted"]
    end

What the gateway does

An LLM gateway is a reverse proxy specialized for model APIs. Clients send requests in one format to a model alias (assistant-default, not a vendor model name); the gateway decides which real deployment serves it, and records what happened.

The gateway is on the critical path of every AI feature, so it must itself be stateless, horizontally scaled and highly available — the quota counters and caches live in Redis, exactly like a rate-limiting API gateway (Edge).

Routing

The same prompt can be served by models that differ by an order of magnitude in price and speed. Routing chooses, per request, the cheapest deployment that is good enough.

flowchart TD
    req["request: alias assistant-default"] --> pii{"contains personal data?"}
    pii -- yes --> region["only approved-region deployments"]
    pii -- no --> all["all deployments"]
    region --> task{"task class"}
    all --> task
    task -- "FAQ / small talk" --> small["small model"]
    task -- "booking with tools" --> large["large model"]
    small --> check{"answer valid?"}
    check -- no --> large
    check -- yes --> done(["respond"])
    large --> done

Aliases decouple applications from model names. When a new model version appears, the platform team points assistant-default at it for 5% of traffic (a canary, as in Cloud-native delivery), compares online metrics and offline evals, then rolls forward or back — with no change in the assistant's code. Pin model versions behind the alias: a provider silently updating "latest" is an untested deploy.

Fallbacks and retries

A model call has no side effects: retrying it is always safe (tool calls are a different story — see Agents & Tools). But not every error deserves a retry, and some deserve a different model.

A simplified, provider-agnostic sketch of the fallback chain, combining the retry and circuit-breaker patterns of Reliability:

import random
import time

CHAIN = {  # model alias -> ordered list of deployments
    "assistant-default": ["large@provider-a", "large@provider-b", "medium@self-hosted"],
    "assistant-cheap": ["small@self-hosted", "small@provider-a"],
}
RETRYABLE = {429, 500, 502, 503, 504}


def complete(alias, request, clients, breakers, ttft_deadline_s=3.0):
    """clients and breakers are this sketch's own interfaces, not a real SDK."""
    last_error = None
    for deployment in CHAIN[alias]:
        if breakers[deployment].is_open():
            continue  # known to be failing: skip it without waiting
        for attempt in range(2):
            try:
                reply = clients[deployment].chat(request, first_token_timeout=ttft_deadline_s)
                breakers[deployment].record_success()
                return reply
            except ProviderError as error:
                breakers[deployment].record_failure()
                last_error = error
                if error.status not in RETRYABLE:
                    raise  # 400, context too long, refusal: another try will not help
                if error.status == 429 and not error.retry_after:
                    break  # quota gone here: go straight to the next deployment
                wait = error.retry_after or 0.2 * 2**attempt
                time.sleep(wait + random.uniform(0, wait))  # backoff with jitter
    raise AllDeploymentsFailed(alias) from last_error

Two traps specific to models:

• A fallback model is a different program. Prompts tuned for one model, its tool-calling format and its refusal style may not transfer. Run the eval suite against every model in the chain, not only the primary — otherwise the fallback silently lowers quality exactly during an incident.
• Retries multiply token spend. The retry storm of the reliability class now has a price tag: cap attempts, use a retry budget (e.g. retries ≤ 10% of calls), and count retried tokens against the caller's quota.

"""The LLM gateway: one place that knows models, routes, fallbacks, timeouts and costs."""
import litellm
from litellm import Router

from .config import settings

litellm.drop_params = True  # drop parameters a provider does not support instead of failing

model_list = [{
    "model_name": "local",
    "litellm_params": {"model": settings.local_model, "api_base": settings.ollama_url},
}]
fallbacks = []
if settings.claude_model:  # optional: only when ANTHROPIC_API_KEY is set
    model_list.append({"model_name": "claude", "litellm_params": {"model": settings.claude_model}})
    fallbacks = [{"claude": ["local"]}]  # provider down or out of quota: answer with the local model

router = Router(model_list=model_list, fallbacks=fallbacks, num_retries=1, timeout=120)


def real_model(route: str) -> str:
    """The provider model behind a route name (for metrics and costs)."""
    return settings.claude_model if route == "claude" and settings.claude_model else settings.local_model


async def embed(texts: list[str]) -> list[list[float]]:
    response = await litellm.aembedding(model=settings.embed_model, input=texts, api_base=settings.ollama_url)
    return [item["embedding"] for item in response.data]

Quotas and token budgets

Provider limits are expressed in requests per minute (RPM), tokens per minute (TPM) — sometimes input and output separately — and concurrent requests, per account and per region. A single feature with a bug can exhaust the account's TPM and take every other feature down with it: the noisy neighbor problem, now with an invoice. OWASP lists it as unbounded consumption³; some call it denial of wallet.

The gateway enforces its own, finer limits:

Counting tokens before you know them. A token bucket counts tokens instead of requests, but output tokens are only known after the call. The usual approach is reserve, then reconcile: before the call, count the input tokens with the model's tokenizer and reserve input + max_tokens from the bucket; after the call, return the unused part. Keep the counters in Redis with atomic scripts, as in the rate-limiter design of Edge.

What happens when a budget runs out is a product decision, decided in advance:

1. Degrade: route to a smaller model or answer from the cache/FAQ.
2. Queue: background work (summaries, embeddings) waits for the next window — the async queue pattern.
3. Shed: return 429 with Retry-After and a friendly message; interactive traffic has priority over batch, and paying flows over browsing.

Caching: three different things

"Cache the LLM" means three different mechanisms with very different risks.

Prompt caching

Processing the input (the prefill, below) produces internal key/value tensors for every token. When the next request starts with the same tokens, the server can reuse them instead of recomputing. Most major providers and open-source servers (e.g. vLLM's automatic prefix caching) support this; providers bill cached input tokens at a fraction of the normal price, require a minimum prefix length, and keep entries for minutes of inactivity. Details — automatic or explicit cache markers, discount, lifetime — differ by provider: read their documentation.

Because the match is on an exact prefix, the order of the prompt matters:

flowchart LR
    s["1. system prompt<br/>(static)"] --> t["2. tool definitions<br/>(static)"] --> k["3. retrieved documents<br/>(per conversation)"] --> h["4. conversation history<br/>(grows)"] --> u["5. new user message<br/>(changes every call)"]

Semantic caching

A semantic cache embeds the user question (see RAG & Memory) and looks for a previous question whose embedding is close enough. "What time do the gates open at Arena Paulista?" and "when do doors open at Arena Paulista?" should hit. The danger is the reverse case: "can I get a refund for the Saturday show?" and "can I get a refund for the Sunday show?" are almost identical as vectors and may need different answers.

Rules that keep a semantic cache safe:

• Scope the key: tenant, language, event or venue ID, and policy version are part of the key, not only the vector.
• Only for generic answers: never cache answers that used tools, personal data or the user's permissions.
• Tune the threshold with data: label a sample of (question, cached question) pairs as same/different and pick the similarity threshold for the precision you need; a hit ratio gained with wrong answers is a bug.
• Short TTL and invalidation when the source document changes (the same problem as in Caching).

Inference performance: TTFT, TPOT and throughput

Generating an answer has two phases with opposite profiles:

• Prefill — the model processes all input tokens in parallel. It is compute-bound and ends when the first output token appears: it sets the time to first token (TTFT). TTFT grows with the uncached prompt length and with queueing at the server.
• Decode — the model generates output tokens one at a time, each step reading all the model weights from GPU memory. It is memory-bandwidth-bound and sets the time per output token (TPOT), also called inter-token latency (ITL).

With TTFT = 0.6 s and TPOT = 20 ms (50 tokens/s), a 250-token answer takes 0.6 + 0.02 × 249 ≈ 5.6 s; a 1,000-token answer, ~20.6 s. Answer length is a latency decision.

import matplotlib.pyplot as plt
from io import StringIO

# Left: end-to-end latency = TTFT + TPOT x (output tokens - 1)
models = {"large model (TTFT 0.6 s, 50 tok/s)": (0.6, 0.020),
          "small model (TTFT 0.2 s, 150 tok/s)": (0.2, 1 / 150)}
tokens = list(range(1, 1001, 10))

# Right: one self-hosted replica decoding a batch of b sequences per step.
# Step time = fixed weight read + a small per-sequence cost (illustrative).
W_READ_MS, PER_SEQ_MS = 12.0, 0.15
batch = list(range(1, 257))
per_stream = [1000 / (W_READ_MS + PER_SEQ_MS * b) for b in batch]
aggregate = [b * s for b, s in zip(batch, per_stream)]

fig, ax = plt.subplots(1, 2, figsize=(11, 4))
for label, (ttft, tpot) in models.items():
    ax[0].plot(tokens, [ttft + tpot * (n - 1) for n in tokens], label=label)
ax[0].set_xlabel("output tokens")
ax[0].set_ylabel("end-to-end latency (s)")
ax[0].set_title("Latency grows with the answer length")
ax[0].legend()
ax[0].grid(alpha=.3)

ax[1].plot(batch, aggregate, color="tab:green", label="replica throughput (tok/s)")
ax[1].set_xlabel("concurrent sequences in the batch")
ax[1].set_ylabel("replica throughput (tokens/s)", color="tab:green")
twin = ax[1].twinx()
twin.plot(batch, per_stream, color="tab:red", label="per-user speed (tok/s)")
twin.set_ylabel("per-user speed (tokens/s)", color="tab:red")
ax[1].axvline(64, ls="--", color="gray")
ax[1].text(66, max(aggregate) * 0.1, "KV-cache full\n(8k-token contexts)", color="gray")
ax[1].set_title("Batching: throughput vs per-user speed")
ax[1].grid(alpha=.3)
plt.tight_layout()

buffer = StringIO()
plt.savefig(buffer, format="svg", transparent=True)
print(buffer.getvalue())
plt.close()

Streaming

Since the answer arrives token by token, the gateway streams it to the client — usually with Server-Sent Events (SSE) over HTTP, as seen in Protocols. The fan starts reading after the TTFT instead of after the full latency, which is why TTFT is the interactive SLO (e.g. p95 TTFT < 1.5 s) and TPOT only needs to beat reading speed (people read roughly 4–8 words per second, so ~15–20 tokens/s already feels fluent).

Streaming has architectural consequences:

• Long-lived connections. By Little's law, 33 requests/s × 12 s per streamed answer ≈ 400 open connections per instance pool. Use asynchronous servers (FastAPI/uvicorn, Netty), and raise idle timeouts on load balancers and proxies that would cut a slow stream.
• Tool calls and JSON cannot be acted on until complete. Only the final natural-language answer benefits from streaming.
• Output guardrails and streaming conflict. Checking the whole answer before showing it removes the TTFT benefit; checking chunk by chunk may show a problematic sentence before it is caught. Common compromise: stream, check in windows of a few sentences, and retract if needed — and never stream content of high-risk actions before validation.

Batching and the KV cache

If you self-host a model (or just want to understand provider limits), one number explains most behavior: decode is limited by memory bandwidth, not by compute. Every decode step reads all the weights. An 8-billion-parameter model in 16-bit precision is ~16 GB; a GPU with ~3 TB/s of memory bandwidth needs at least ~5 ms to read it — so a single sequence cannot exceed ~200 tokens/s however powerful the GPU.

Batching amortizes that read: one pass over the weights advances b sequences at once. Replica throughput grows almost linearly with b while each user's speed degrades slowly (right plot above). Two techniques made this practical:

• Continuous (in-flight) batching⁴: instead of waiting for a whole batch to finish, the server adds and removes sequences at every token step, so a short answer does not wait for a long one.
• PagedAttention⁵: the KV cache of each sequence is stored in fixed-size pages, like virtual memory, eliminating fragmentation and allowing prefix sharing across requests (which is how self-hosted prompt caching works).

The limit is memory for the KV cache, the key/value tensors kept for every token of every active sequence:

For a Llama-3-8B-class model (32 layers, 8 KV heads, head dimension 128, 16-bit values): 2 × 32 × 8 × 128 × 2 = 128 KiB per token. An 8,000-token conversation needs ~1 GiB. On an 80 GB GPU with ~16 GB of weights, ~64 GB remain: ~64 concurrent 8k-token conversations, or ~512 of 1k tokens. Concurrency is bounded by context length × users, which is why long prompts are expensive even when they are cheap per token.

Other serving techniques you will see in 2026 (all trade something for throughput or latency):

Self-hosted or API?

A back-of-the-envelope check: a GPU that costs US$ c per hour and sustains T output tokens/s at your batch size costs \(c / (3600 \cdot T \cdot u)\) per token at utilization u. At 10% utilization it is 10× more expensive per token than at 100%. Self-hosting wins only with steady load — or when data cannot leave your network. For offline work (embedding the catalog, running evals), many providers also offer batch APIs: asynchronous, results within hours, at a discount.

Observability at the gateway

Because every call passes through it, the gateway is the natural place to emit the LLM telemetry described in Safety, Evals & Observability: one span per call with model, tokens (input, cached, output), cost, TTFT and finish reason, plus metrics aggregated per alias, feature and user tier. The dashboards answer the CFO's question directly: US$ per conversation, by feature, per day.

Products

The gateway is a pattern, not a product. Examples, to explore — check each product's current documentation for capabilities:

How it connects

flowchart LR
    SVC["assistant, summarizer,<br/>translator, embed job"] --> GW["LLM gateway<br/>routing by alias"]
    GW --> FB["fallbacks + circuit breaker<br/>→ Reliability"]
    GW --> Q["token quotas, priorities<br/>→ API gateway rate limiting"]
    GW --> PC["prompt caching<br/>(stable prefix)"]
    GW --> SC["semantic caching<br/>→ Caching invalidation"]
    GW --> ST["streaming over SSE, TTFT<br/>→ Edge protocols"]
    GW --> CAN["canary by alias<br/>→ Cloud-native delivery"]
    GW --> OBS["tokens, cost, TTFT<br/>→ Safety, Evals & Observability"]
    Q --> BG["background work waits<br/>→ Messaging queues"]

Palette components: llmgw · apigw (its classic cousin) · thirdparty (the model providers behind it) · cache (semantic cache) · queue (background work waiting for quota) · metrics and tracing (tokens, cost, TTFT). Scenarios: ai-assistant, cascading-failure (a slow provider without a breaker), rate-limiter (quotas as rate limits). Continues in: Edge — API gateway, Reliability — patterns, Caching — invalidation, Edge — protocols, Agents & Tools.