15. Large Language Models

An LLM is the decoder from chapter 12, scaled until something surprising happened: a model trained only to predict the next token turned out to be able to translate, write code, hold a conversation and — after 2024 — sit and think about a problem before answering.

This chapter is about the four things that turn that architecture into a system you can use, in the order they happen: pretraining (what next-token prediction actually buys), post-training (turning a text-continuer into an assistant), inference-time compute (the axis that joined parameter count as a way to buy capability), and serving (sparse experts, decoding and the KV cache: why any of it costs what it costs).

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Seven of the eight panels of this page (the BPE merges, test-time compute, MoE, sampling, the next-token code, the KV cache and speculative decoding) run inside the slides; the agent episode is only here. Open the slides in a new tab

Pretraining: one objective, and what it forces the model to learn

Given text \(x_1, \dots, x_T\) of \(T\) tokens, minimize

where \(p_\theta(x_t \mid x_{<t})\) is the probability the model, with parameters \(\theta\), gives the token that actually comes after the ones before it. That is the whole pretraining objective. It is self-supervised — the labels are the next tokens — so the training set is any text that exists. The log is natural, so the loss is in nats (\(\log_2\) would give bits; 1 nat ≈ 1.44 bits). Divide \(\mathcal{L}\) by \(T\) and exponentiate and you get the perplexity of chapter 8.4: a mean loss of 3 nats per token is a perplexity of \(e^3 \approx 20\), as unsure as a uniform choice among 20 tokens. Perplexity is per token, so it changes with the tokenizer; dividing the total loss, in bits, by the number of characters instead gives bits per character, which does not.

The reason it produces more than autocomplete is worth stating carefully: compressing text well requires modelling whatever produced the text. To predict the last token of "the murderer turned out to be the ___" you need the plot. To predict the token after return you need the function's contract. Next-token prediction is a compression objective, and compression at this scale forces structure.

How big, and on how much text

How the loss falls as you add parameters \(N\) and tokens \(D\) is the subject of the scaling laws in chapter 12: training compute is \(C \approx 6ND\), the compute-optimal recipe is roughly 20 tokens per parameter, and deployed models are trained far past it because every token they serve is paid for by \(N\). SmolLM2-135M, the model of this chapter's lab, saw 2 trillion tokens¹⁹ — about 15 000 per parameter.

The lab measures the \(N\) axis on the Pythia suite¹⁸, models of different sizes trained on the same 300B tokens in the same order: from 70M to 1B parameters, the loss on WikiText-2 (Wikipedia text, part of the training data of every one of them alike) falls from 4.09 to 2.82 nats per token, close to a straight line on log–log axes (\(L \propto N^{-0.10}\), with \(L\) the loss and \(N\) counted without the embedding tables, as Kaplan et al. did). Each doubling of \(N\) buys a smaller absolute drop than the one before, and that is the whole shape of a power law. Along \(D\) the lab finds the other half: the 70M model stops improving after a few thousand tokens per parameter, because data only helps until the model's size becomes the limit.

Tokenization, and the bugs it causes

Text becomes tokens before the model sees anything, usually via byte-level BPE¹⁶: start from bytes, repeatedly merge the most frequent adjacent pair into a new symbol, stop at a vocabulary of 32k–256k. On the toy corpus aaabdaaabac, the most frequent pair aa becomes Z (ZabdZabac), then ab (tied with Za) becomes Y (ZYdZYac), then ZY becomes X (XdXac): eleven symbols down to five, and a vocabulary that now holds X = aaab. It is a compression algorithm fitted to a corpus, and it is not neutral. Step through the three merges: the pair counts, the merge, and what it does to the text and to the vocabulary.

Byte-level models without a tokenizer are an active research direction. Until one wins, the whole field runs on a compression heuristic brought to NLP in 2015, and it is the first suspect whenever a model fails at something that looks trivial.

Post-training: from continuation to assistant

The pipeline in chapter 14 is the same one, seen from the model's side rather than the practitioner's:

1. SFT (supervised fine-tuning) on demonstrations — teaches the format of being an assistant.
2. Preference optimization — teaches which of two answers is better, which demonstrations cannot express. RLHF (reinforcement learning from human feedback³) trains a reward model on people's comparisons and then optimizes the model against it with PPO (proximal policy optimization); DPO (direct preference optimization) trains on the comparisons directly, with no reward model.
3. RL from a verifier — teaches the model to get it right, when "right" is checkable: reinforcement learning with verifiable rewards, RLVR.

The lab puts SmolLM2-135M next to its instruct version on the same question. The base model continues the text; the instruct model, which sees the question wrapped in a chat template with special tokens around each turn, answers it. Post-training also narrows the distribution: along each model's own answer to five questions, the next-token distribution has a mean entropy (\(H = -\sum p \log_2 p\), in bits) of 3.14 bits under the base model and 1.21 under the instruct one.

Step 3 is what changed most recently, and it deserves its own section.

Test-time compute: the second scaling axis

Until 2024, "a better model" meant "a bigger model, trained on more data". Then a different curve appeared: hold the model fixed and give it more compute at inference — let it write a long chain of reasoning, sample several attempts, check them, pick one. Measured at equal FLOPs, this can beat a model many times larger on problems the small one solves at least sometimes⁹.

The panel simulates a benchmark: each problem has its own chance of being solved on one try (the slider sets the average) and its own favourite wrong answers.

Three lines, and the gap between them is the entire subject:

• pass@k (dashed), the chance that at least one of \(k\) attempts is right, is what the model could achieve if something told it which attempt was right: \(1 - (1-p)^k\) on a problem it solves with probability \(p\) per attempt. It rises fast, and it is an upper bound, not a score.
• Majority voting needs nothing extra and plateaus. On each problem it converges on the answer the model gives most often — which is wrong exactly when the model is confidently wrong.
• Best-of-n with a verifier cashes in the gap, and how much of it depends on the verifier's false-positive rate. Drag the verifier from perfect to weak and watch the ceiling fall.

The ceiling has a formula. On a problem the model solves with probability \(p\), a verifier that accepts a right answer with probability \(\text{TPR}\) (its true-positive rate) and a wrong one with probability \(\text{FPR}\) (its false-positive rate) picks a right answer, as \(k\) grows, with probability

With \(p = 0.25\) and the panel's weak verifier (\(\text{TPR} = 0.80\), \(\text{FPR} = 0.35\)): \(0.20 / (0.20 + 0.26) \approx 0.43\), however many samples you draw. With \(\text{FPR} = 0\) it is 1.

This is consistent with where reasoning models got good first: code (run the tests) and mathematics (check the answer). Both have a cheap verifier with almost no false positives. Essay quality does not, and the improvement there has been more modest.

How the models learned to use it

You cannot get long, useful reasoning out of a model by asking politely, and imitating human-written reasoning traces caps you at the humans. DeepSeek-R1⁶ showed the alternative: let the model discover it. Sample many attempts at problems with checkable answers, reward the ones that verify, and optimize — with GRPO¹², which drops the value network (the second network PPO trains to predict the expected reward) and normalizes the rewards within each group of \(G\) attempts at the same problem. Attempt \(i\) gets the reward \(r_i\) (1 if it verifies, 0 if not) and the advantage \(\hat{A}_i\), how hard its tokens are pushed up or down:

In R1-Zero, trained this way straight from the base model, no reasoning traces were supplied. The chains got longer on their own because longer chains verified more often, and behaviours appeared that nobody wrote down — backtracking, checking work, noticing an error mid-solution. The released R1 added a small set of long worked examples before the RL stage, mainly to make the chains readable; the reasoning itself came from the verifier.

Mixture of Experts: more parameters, same compute

The other way to add capacity without adding cost per token is to make the model sparse⁴. Replace each MLP with a set of independent experts \(E_1, E_2, \dots\) (each one an MLP) and a router that activates only \(k\) of them for each token \(x\):

The router is one matrix, \(W_g\), that scores every expert; \(g(x)\) turns the scores into probabilities, Top-\(k\) keeps the indices of the \(k\) largest and the output is the sum of those experts' outputs, weighted by their probabilities.

Recall from chapter 12 that about two-thirds of a block's parameters are in the MLP. Replacing it is where the leverage is.

Set routing to no balancing first. The router collapses onto a few experts, and the rest are dead weight you still pay to store. Routing is a positive feedback loop — an expert that receives more tokens trains faster and therefore gets chosen more — so balancing is the hard part of MoE, not routing. Classic solutions add an auxiliary loss; the current preference is loss-free balancing, which adjusts a per-expert bias in the router instead of adding a second objective to fight the first.

Two design moves from the current generation⁵, both visible in the panel:

• Fine-grained experts — many small experts instead of a few large ones. With the same active parameter count, top-8-of-256 offers vastly more combinations than top-2-of-8.
• A shared expert, always active, that absorbs general-purpose behaviour so the routed experts can actually specialize rather than each relearning the basics.

Sampling: the model gives you a distribution, not a token

A forward pass produces a probability distribution over the vocabulary. Turning that into text is a separate algorithm with its own parameters, and it affects output quality as much as some model upgrades. The panel's two distributions are illustrations; the lab reads real ones off SmolLM2-135M.

Temperature divides the logits before the softmax: below 1 it sharpens, above 1 it flattens. It adds no randomness — it redistributes the randomness that is already there. Pull it to 0.05 and you get, in effect, greedy decoding, which is deterministic and prone to loops, because the most likely continuation of a repeated phrase is to repeat it again.

Truncation is the other half, and it is what stops the long tail of nonsense from ever being sampled:

• top-k — keep the \(k\) most likely. Crude: \(k = 40\) is far too many for a confident prediction and far too few for an open one. In the lab, the 90% nucleus after "The Eiffel Tower is located in the city of" holds 3 tokens; after "My favourite food is" it holds 1,550.
• top-p (nucleus)⁷ — keep the smallest set whose mass reaches \(p\). Adapts to the shape of the distribution. The long-standing default.
• min-p¹³ — keep tokens above a fraction of the top probability. Adapts better still, and holds up at high temperature where top-p starts admitting junk.

The lab measures the loop on a 100-token greedy continuation: 42% of its 4-grams are repeats, against 11% on average for samples at temperature 0.7 with top-p 0.9.

The same logits serve training and generation, and the panel below runs both as the lab writes them: the per-token loss and the perplexity of a four-token sequence (2.01 here, against 5 for a uniform guess over the five-token vocabulary), then warp and one multinomial draw on the last row. The last line of the code counts what the KV cache saves.

Serving: why a token costs what it costs

Generation has two phases. Prefill reads the whole prompt in one forward pass: every position is computed in parallel, and the GPU is busy doing arithmetic. Decode then produces one token per forward pass, and each pass has to read every weight of the model from memory to do very little arithmetic with it. Decoding is limited by memory bandwidth, not by FLOPs.

A worked number: an 8B model in 16 bits is 16 GB of weights. A GPU that reads 3.35 TB/s (an H100) needs about 5 ms just to stream them once, so a single conversation cannot decode much faster than about 200 tokens per second, whatever the FLOPs. Serving systems recover the waste by batching: one read of the weights serves every request in the batch.

What decoding must not do is recompute the past. The keys and values of earlier tokens never change, so they are computed once and kept: the KV cache of chapter 11. Without it, step \(t\) reprocesses all \(t\) tokens and generating \(n\) tokens costs \(O(n^2)\) token computations instead of \(O(n)\). The lab turns it off: for 400 new tokens the network processes 403 tokens with the cache and 81,400 without (202× the work), and on a CPU that took 6.7 s against 43.1 s (6.4×; fixed per-step costs keep the time ratio below the work ratio). The gap widens with every extra token. The cache is paid in memory instead — for SmolLM2-135M, a key and a value in each of 30 layers, for each of its 3 key/value heads of 64 numbers: \(2 \times 30 \times 3 \times 64 = 11\,520\) numbers per token.

The panel runs six steps of that loop on the lab's prompt, without and with the cache. With it, the first row is the prefill and every later row a decode step:

Two more tricks follow from decoding being memory-bound:

• Quantization stores the weights in 8 or 4 bits. Fewer bytes to read per token means faster decoding, usually at a small quality cost.
• Speculative decoding¹⁰. A small draft model proposes several tokens, the big model checks them all in one parallel forward pass, and they are accepted left to right, each with probability \(\min(1, p/q)\), where \(q\) is the probability the draft gave the token and \(p\) the one the big model gives it; the first rejected token is replaced by one drawn from the big model, and that rule keeps the sampled distribution exactly the big model's. Because one pass costs the same memory traffic whether it checks one token or five, the original paper reports 2–3× faster decoding, with the same output distribution.

One round of speculative decoding, step by step: what the draft proposes, what the single pass of the big model returns, and the rule that decides what is kept.

Emergence, honestly

The standard story is that capabilities appear abruptly past a scale threshold² — flat, flat, flat, then sudden competence. It is a striking claim and it is at least partly an artifact.

Schaeffer et al.⁸ showed that many "emergent" curves come from discontinuous metrics. Score multi-digit arithmetic as exact-match and you get a step function; score the same model's per-token log-probability of the correct answer and you get a smooth curve. The underlying capability was improving all along; the measurement had a cliff in it.

What survives the critique is still important:

• Improvement is smooth in the loss and often abrupt in what you can use. A model at 40% exact-match on a task is not 40% useful; it may be unusable, and the same model at 85% is a product. That discontinuity is real even if the underlying curve is smooth.
• In-context learning — solving a task from examples in the prompt, with no weight update¹ — improves strongly with scale, and it is what made prompting a discipline.

The practical lesson is about evaluation, not metaphysics: if your metric is a threshold, your progress will look like a cliff, and you will not be able to tell improvement from noise until you fall off it.

Where the failures actually come from

From chat to agents

The deployment pattern shifted, and it is the shift most relevant to what you will be asked to build. A chat returns text. An agent runs a loop: read the goal, call a tool, read the result, decide, repeat, stop.

The panel steps through one concrete episode: a question about a data file that takes three tool calls, one of which fails. Watch the context window: it only ever grows.

Almost nothing in that loop is model capability. It is engineering: which tools exist, what they return, how errors surface, how much history is kept, when to stop. The Model Context Protocol (MCP) standardizes the tool-server side of it, so a tool written once works across clients.

Two properties of the loop are worth internalizing:

• Errors compound. A 95%-reliable step is 60% reliable after ten. Agents that work are built from steps that are checkable and re-runnable, not from one long inference.
• The context window is a budget. Every tool result competes with the goal for room. Summarizing, discarding and re-retrieving are the real design work.

The landscape, and how to read it

Any table of specific models is out of date before it is printed. What is stable is the shape of the market:

The durable trends behind the churn: prices fall by an order of magnitude or more per year for a fixed capability (between 9× and 900× per year depending on the task, in one 2025 measurement¹⁵); open weights trail the frontier by months, not years; reasoning and multimodality are becoming default features rather than separate products; and capability per active parameter keeps improving, so today's ~30B open models score above the original GPT-4 on most public benchmarks.

Lab: the distribution, measured

The claims of this page about tokens, distributions and decoding are measured in a Colab notebook with small open models that download without a login: SmolLM2-135M and its instruct version, and the Pythia suite. It runs on CPU; a T4 GPU (Runtime → Change runtime type) only makes section 7 faster. Eight sections, each ending in a checkpoint with a check value:

1. tokenization: strawberry, digits, and the price of Portuguese in three tokenizers;
2. the next-token distribution, and what a trailing space does to it;
3. temperature, top-k and top-p written by hand, and the greedy loop measured;
4. perplexity and bits per character, on normal, shuffled and Portuguese text;
5. the KV cache: its size from the formula and from the tensors, and the speed-up of keeping it;
6. base against instruct on the same question, and what the chat template adds;
7. a scaling law measured on Pythia, along \(N\) and along \(D\);
8. an open task: decoding settings against repetition and likelihood, in a table you fill in.

The fine-tuning side (LoRA, SFT on your own data) belongs to the course's LLM exercise; this lab does not train anything.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. Next-token prediction is a compression objective. Compressing text well requires modelling what produced it, which is why the capability generalizes.
2. Tokenization is a fitted heuristic with real consequences — character-level failures, arithmetic quirks, and Portuguese costing 1.8× the tokens of English in an English-centred tokenizer.
3. A base model continues text. Everything conversational is post-training: SFT, then preference optimization, then RL from a verifier.
4. Test-time compute is the second scaling axis. pass@k is an upper bound; converting it into accuracy needs a verifier, and the verifier's false-positive rate sets the ceiling.
5. GRPO with a checker taught models to reason without being shown reasoning. It works where verification is free — code and mathematics — and reward hacking is the standing risk.
6. MoE buys capacity at constant FLOPs and costs memory. Load balancing, not routing, is the hard part; fine-grained experts plus a shared expert is the current recipe.
7. Decoding is a separate algorithm. Temperature rescales logits; top-p and min-p truncate the tail. Greedy decoding loops.
8. Decoding a token is memory-bound. The KV cache turns \(O(n^2)\) into \(O(n)\) and is paid in memory; batching, quantization and speculative decoding (2–3×, same output distribution) all attack the bytes read per token.
9. Emergence is partly a metric artifact, but threshold effects in usefulness are real. Do not let a step-function metric hide smooth progress.
10. Hallucination is trained in by accuracy-only grading. Prompt injection has no robust fix and must be handled architecturally.
11. Agents are loops, and errors compound. Most of the work is tools, context and stopping conditions — not the model.