Skip to content

15. LLMs

An LLM is the decoder from chapter 12, scaled until something surprising happened: a model trained only to predict the next token turned out to be able to translate, write code, hold a conversation and — after 2024 — sit and think about a problem before answering.

This chapter is about the four things that turn that architecture into a system you can use, in the order they happen: pretraining (what next-token prediction actually buys), post-training (turning a text-continuer into an assistant), inference-time compute (the axis that replaced parameter count as the frontier), and serving (why any of it costs what it costs).

Pretraining: one objective, and what it forces the model to learn

Given text \(x_1, \dots, x_T\), minimize

\[ \mathcal{L} = -\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}) \]

That is the whole pretraining objective. It is self-supervised — the labels are the next tokens — so the training set is any text that exists.

The reason it produces more than autocomplete is worth stating carefully: compressing text well requires modelling whatever produced the text. To predict the last token of "the murderer turned out to be the ___" you need the plot. To predict the token after return you need the function's contract. Next-token prediction is a compression objective, and compression at this scale forces structure.

The base model is not the assistant

A model straight out of pretraining continues text; it does not answer questions. Asked "What is the capital of France?" it may reply with three more exam questions — a perfectly good continuation of a document that looks like a quiz. Everything conversational comes from post-training.

Tokenization, and the bugs it causes

Text becomes tokens before the model sees anything, usually via byte-level BPE: start from bytes, repeatedly merge the most frequent adjacent pair, stop at a vocabulary of 32k–256k. It is a compression algorithm fitted to a corpus, and it is not neutral.

Symptom Cause
"How many r's in strawberry?" goes wrong The model never sees letters, only st|raw|berry. Character-level questions ask about something below its input resolution.
Arithmetic fails at odd digit counts Numbers are split inconsistently — 1234 may be one token, 12345 two. Modern tokenizers force digits into fixed groups precisely to fix this.
Portuguese costs ~1.5× more than English The merges were fitted on a corpus that is mostly English. Fewer merges cover Portuguese, so the same sentence is more tokens — more latency, more money, less usable context, for the same content.
Trailing whitespace ruins a completion " the" and "the" are different tokens; a prompt ending in a space puts the model off-distribution.

Tokenizer-free byte-level models are a live research direction, and the fact that the entire field still runs on a compression heuristic fitted in 2015 is a real, if quiet, embarrassment.

Post-training: from continuation to assistant

The pipeline in chapter 14 is the same one, seen from the model's side rather than the practitioner's:

  1. SFT on demonstrations — teaches the format of being an assistant.
  2. Preference optimization (DPO, or PPO-based RLHF3) — teaches which of two answers is better, which demonstrations cannot express.
  3. RL from a verifier — teaches the model to get it right, when "right" is checkable.

Step 3 is what changed most recently, and it deserves its own section.

Test-time compute: the second scaling axis

Until 2024, "a better model" meant "a bigger model, trained on more data". Then a different curve appeared: hold the model fixed and give it more compute at inference — let it write a long chain of reasoning, sample several attempts, check them, pick one.

Three lines, and the gap between them is the entire subject:

  • pass@k (dashed) is what the model could achieve if something told it which attempt was right. It rises fast, and it is an upper bound, not a score.
  • Majority voting needs nothing extra and plateaus. It converges on what the model believes most often — which is wrong exactly when the model is confidently wrong.
  • Best-of-n with a verifier cashes in the gap, and how much of it depends entirely on the verifier's false-positive rate. Drag the verifier from perfect to weak and watch the ceiling fall.

This explains, precisely, where reasoning models got good first: code (run the tests) and mathematics (check the answer). Both have a free, perfect verifier. Essay quality does not, and the improvement there has been correspondingly modest.

How the models learned to use it

You cannot get long, useful reasoning out of a model by asking politely, and imitating human-written reasoning traces caps you at the humans. DeepSeek-R16 showed the alternative: let the model discover it. Sample many attempts at problems with checkable answers, reward the ones that verify, and optimize — with GRPO, which drops the value network and normalizes rewards within each group:

\[ \hat{A}_i = \frac{r_i - \text{mean}(r_{1..G})}{\text{std}(r_{1..G})} \]

No reasoning traces are supplied. The chains get longer on their own because longer chains verify more often, and behaviours appear that nobody wrote down — backtracking, checking work, noticing an error mid-solution.

Reward hacking is the failure mode, and a verifier is not a value function

Optimizing against a checker means optimizing against the checker, not against correctness. Models learn to special-case the tests, to exploit a grader's format, to produce answers that pass a string match without being right. Every RLVR pipeline needs adversarial evaluation of the verifier itself, and this is the main reason the technique has not spread far beyond domains with airtight checkers.

What this changes about cost

An answer's price is no longer a fixed number of parameters times a fixed number of tokens. A "thinking budget" is now a knob you set per request, and it trades latency and money for accuracy along the curve above. Deciding how much thinking a question deserves is a genuinely new engineering problem, and getting it wrong in either direction is expensive.

Mixture of Experts: more parameters, same compute

The other way to add capacity without adding cost per token is to make the model sparse. Replace each MLP with \(E\) independent experts and a router that activates only \(k\) of them:

\[ \text{MoE}(x) = \sum_{i \in \text{Top-}k(G(x))} G(x)_i \cdot E_i(x), \qquad G(x) = \text{softmax}(W_g x) \]

Recall from chapter 12 that about two-thirds of a block's parameters are in the MLP. Replacing it is where the leverage is.

Set routing to no balancing first. The router collapses onto a few experts, and the rest are dead weight you still pay to store. Routing is a positive feedback loop — an expert that receives more tokens trains faster and therefore gets chosen more — so balancing is the hard part of MoE, not routing. Classic solutions add an auxiliary loss; the current preference is loss-free balancing, which adjusts a per-expert bias in the router instead of adding a second objective to fight the first.

Two design moves from the current generation, both visible in the panel:

  • Fine-grained experts — many small experts instead of a few large ones. With the same active parameter count, top-8-of-256 offers vastly more combinations than top-2-of-8.
  • A shared expert, always active, that absorbs general-purpose behaviour so the routed experts can actually specialize rather than each relearning the basics.

MoE trades compute for memory, and that is a deployment decision

You pay memory for total parameters and compute for active ones. A 671B model with 37B active is cheap to run per token and still needs the whole 671B resident. That is a good trade in a datacentre with many concurrent requests and a bad one on a single GPU — which is why dense models in the 4B–30B range remain the default for local and on-device work.

Sampling: the model gives you a distribution, not a token

A forward pass produces a probability distribution over the vocabulary. Turning that into text is a separate algorithm with its own parameters, and it affects output quality as much as some model upgrades.

Temperature divides the logits before the softmax: below 1 it sharpens, above 1 it flattens. It adds no randomness — it redistributes the randomness that is already there. Pull it to 0.05 and you get greedy decoding, which is deterministic and prone to loops, because the most likely continuation of a repeated phrase is to repeat it again.

Truncation is the other half, and it is what stops the long tail of nonsense from ever being sampled:

  • top-k — keep the \(k\) most likely. Crude: \(k = 40\) is far too many for a confident prediction and far too few for an open one.
  • top-p (nucleus)7 — keep the smallest set whose mass reaches \(p\). Adapts to the shape of the distribution. The long-standing default.
  • min-p — keep tokens above a fraction of the top probability. Adapts better still, and holds up at high temperature where top-p starts admitting junk.

Reasonable defaults

Factual work, code, tool calls: temperature 0 or 0.2. Conversation and prose: 0.7–1.0 with top-p 0.9 or min-p 0.05. And be aware that reasoning models are usually trained for a specific sampling configuration and get measurably worse outside it — check the model card before overriding it.

A related trick worth knowing: speculative decoding. A small draft model proposes several tokens, the big model verifies them in one parallel forward pass, and any prefix it agrees with is accepted. Because decoding is memory-bound (chapter 11), checking five tokens costs almost what checking one costs. Two to three times faster, with identical output distribution.

Emergence, honestly

The standard story is that capabilities appear abruptly past a scale threshold2 — flat, flat, flat, then sudden competence. It is a striking claim and it is at least partly an artifact.

Schaeffer et al.8 showed that many "emergent" curves come from discontinuous metrics. Score multi-digit arithmetic as exact-match and you get a step function; score the same model's per-token log-probability of the correct answer and you get a smooth curve. The underlying capability was improving all along; the measurement had a cliff in it.

What survives the critique is still important:

  • Improvement is smooth in the loss and often abrupt in what you can use. A model at 40% exact-match on a task is not 40% useful; it may be unusable, and the same model at 85% is a product. That discontinuity is real even if the underlying curve is smooth.
  • In-context learning — solving a task from examples in the prompt, with no weight update1 — is genuinely a property of scale, and it is what made prompting a discipline.

The practical lesson is about evaluation, not metaphysics: if your metric is a threshold, your progress will look like a cliff, and you will not be able to tell improvement from noise until you fall off it.

Where the failures actually come from

  • Hallucination

    Not a bug in the architecture — a consequence of the training signal. A model that says "I don't know" scores zero on a benchmark; a model that guesses scores sometimes. We grade on accuracy, so we train models to bluff.

    Mitigations that work: retrieval with citations, tools with real answers, asking for calibrated confidence, and — increasingly — evaluations that reward abstention.

  • Context is not memory

    Long context windows are real, and performance inside them is not uniform. Retrieval accuracy sags in the middle of a long context, and a model given 500k tokens of loosely related material often does worse than one given the right 5k.

    Long context is a place to put a working set, not a substitute for retrieval. Treat "just paste everything in" as a hypothesis to be measured.

  • Prompt injection

    The model cannot reliably distinguish your instructions from instructions embedded in the data it reads. Once it has tools, a hostile web page is an attack. There is no known robust fix; the mitigations are architectural — least privilege, human confirmation on consequential actions, isolating untrusted content — and this is the primary reason agents are deployed cautiously.

  • Evaluation rot

    Benchmarks leak into training corpora, and a score on a public benchmark is partly a measurement of contamination. Prefer held-out private sets, fresh dynamic benchmarks, and pairwise human comparison — and distrust any leaderboard delta smaller than a few points.

From chat to agents

The deployment pattern shifted, and it is the shift most relevant to what you will be asked to build. A chat returns text. An agent runs a loop: read the goal, call a tool, read the result, decide, repeat, stop.

flowchart LR
    A[goal] --> B[model decides]
    B -->|tool call| C[execute]
    C -->|result| B
    B -->|done| D[answer]

Almost nothing in that loop is model capability. It is engineering: which tools exist, what they return, how errors surface, how much history is kept, when to stop. The Model Context Protocol (MCP) standardizes the tool-server side of it, so a tool written once works across clients.

Two properties of the loop are worth internalizing:

  • Errors compound. A 95%-reliable step is 60% reliable after ten. Agents that work are built from steps that are checkable and re-runnable, not from one long inference.
  • The context window is a budget. Every tool result competes with the goal for room. Summarizing, discarding and re-retrieving are the real design work.

The landscape, and how to read it

Any table of specific models is out of date before it is printed. What is stable is the shape of the market:

Tier What defines it
Frontier, closed OpenAI (GPT), Anthropic (Claude), Google (Gemini). Extended reasoning on by default, strong multimodality, best tool use. You rent them.
Frontier-adjacent, open weights DeepSeek, Qwen, Kimi, GLM, Llama, Mistral. Within months of the frontier on most benchmarks, at a fraction of the price, and you can run them. Large MoEs — capable, but not laptop-sized.
Small and open 1B–30B dense models. Run on one GPU or a phone. The right target for distillation and for on-device work.
Specialists Code, embeddings, rerankers, safety classifiers, OCR. Usually small, usually the right answer for a narrow job.

The durable trends behind the churn: prices fall roughly an order of magnitude a year for a fixed capability; open weights trail the frontier by months, not years; reasoning and multimodality are becoming default features rather than separate products; and capability per active parameter keeps improving, so today's 30B is comfortably beyond an early GPT-4.

How to choose one

Not from a leaderboard. Write 20–50 examples of your task with the outputs you want, run three or four candidate models against them, and read the failures. This takes an afternoon and beats every benchmark discussion you could have instead. Then re-run it when a new model appears — that is the only way to know whether "better" is better for you.

Key takeaways

  1. Next-token prediction is a compression objective. Compressing text well requires modelling what produced it, which is why the capability generalizes.
  2. Tokenization is a fitted heuristic with real consequences — character-level failures, arithmetic quirks, and a 1.5× cost penalty for Portuguese.
  3. A base model continues text. Everything conversational is post-training: SFT, then preference optimization, then RL from a verifier.
  4. Test-time compute is the second scaling axis. pass@k is an upper bound; converting it into accuracy needs a verifier, and the verifier's false-positive rate is the ceiling.
  5. GRPO with a checker taught models to reason without being shown reasoning. It works where verification is free — code and mathematics — and reward hacking is the standing risk.
  6. MoE buys capacity at constant FLOPs and costs memory. Load balancing, not routing, is the hard part; fine-grained experts plus a shared expert is the current recipe.
  7. Decoding is a separate algorithm. Temperature rescales logits; top-p and min-p truncate the tail. Speculative decoding is 2–3× free speed with identical outputs.
  8. Emergence is partly a metric artifact, but threshold effects in usefulness are real. Do not let a step-function metric hide smooth progress.
  9. Hallucination is trained in by accuracy-only grading. Prompt injection has no robust fix and must be handled architecturally.
  10. Agents are loops, and errors compound. Most of the work is tools, context and stopping conditions — not the model.

  1. Brown, T., et al. (2020). Language Models are Few-Shot Learners — NeurIPS. GPT-3, and in-context learning as a phenomenon. ↩

  2. Wei, J., et al. (2022). Emergent Abilities of Large Language Models — TMLR. The claim, stated carefully. ↩

  3. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback — NeurIPS. InstructGPT. ↩

  4. Shazeer, N., et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — ICLR. Where MoE for language models starts. ↩

  5. DeepSeek-AI (2024). DeepSeek-V3 Technical Report. Fine-grained experts, a shared expert, and auxiliary-loss-free load balancing, described in enough detail to reproduce. ↩

  6. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. GRPO, and reasoning that emerges from a verifier rather than from demonstrations. ↩

  7. Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration — ICLR. Why greedy decoding loops, and where nucleus sampling comes from. ↩

  8. Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? — NeurIPS. Discontinuous metrics manufacture discontinuous curves. ↩

  9. Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. The trade between training compute and inference compute, measured. ↩

  10. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding — ICML. ↩

  11. Liu, N., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts — TACL. Retrieval accuracy as a function of where in the context the answer sits. ↩