Overview

The previous chapter built one mechanism. A Transformer is what you get when you decide that mechanism is enough — that you can throw away recurrence and convolution and keep only attention, plus the smallest possible amount of scaffolding around it¹.

The scaffolding turns out to be short: a per-position MLP, a residual connection, a normalization. That block, repeated, is the entire architecture — of GPT, of Llama, of ViT, of Whisper, of the diffusion backbones in chapter 22. Almost nothing else in modern AI has this property of being one design used everywhere.

This chapter has two jobs. The first is to show what the block does. The second is more useful and less often done: to show which parts of the 2017 design are still there and which were quietly replaced, because the diagram in the original paper is not the block you would write today.

The seven steps, and where the running example is

The toy model of the running example has a vocabulary of 12 tokens, \(d = 4\), two heads and one block; GPT-2 small has 50,257 tokens, \(d = 768\), twelve heads and twelve blocks (the table compares them). The attention chapter ended with the attention output \(a\) of the five tokens; this one adds it to the input, finishes the block, and turns the result into " roof".

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Four simulators of these pages run inside the slides. Open the slides in a new tab

From text to the next word: the full path

Before opening the block, it is worth seeing where it sits. A language model takes "the cat climbed the" and returns a probability for every vocabulary token being the next one:

(The ids are GPT-2's; the probabilities are illustrative.) Three things to notice:

• The shape does not change inside the stack. The embedding turns the \(n\) tokens into an \(n \times d\) matrix, and every block takes \(n \times d\) and returns \(n \times d\). That is what lets you stack 32 or 126 identical blocks. Only the last layer goes back to the vocabulary size \(V\).
• Every position makes a prediction. Row \(i\) of the output is the distribution of token \(i+1\). During generation only the last row matters; during training all of them count — the causal mask from the previous chapter makes sure none of them sees the answer.
• Generating is repeating. Sample a token from the distribution (how is its own decision), append it to the input, and run again. The KV cache is what keeps that "run again" from recomputing everything.

See it run: Transformer Explainer

Everything so far can be watched on a real model. Transformer Explainer², from the Polo Club at Georgia Tech, runs GPT-2 small³ in the browser — 124M parameters, 12 blocks of 12 heads, \(d = 768\), a vocabulary of 50,257 tokens — and draws every stage of the path above for the text you type, from the tokens to the probability of the next one. Its expanded views go down to the matrix multiplications, and every number on screen comes from the model, not from an illustration.

Twenty minutes, in this order. Each step says what to click, what to look for and which part of this chapter (or the previous one) it shows.

1. Tokens and embedding — the full path; the embedding layer. Keep the example prompt and expand the Embedding column. Count the tokens: they are not the words. Find each token's ID, its 768-number vector and the positional vector added to it. In GPT-2 that position is a learned table, one vector per position up to 1024 — neither the sinusoid of 2017 nor RoPE. Then type a sentence in Portuguese and count again: more tokens per word, because the vocabulary was built on English text.
2. Q, K and V — inside a block; Query, Key, Value. Expand the step that turns each token's vector into three. Check the sizes: Q, K and V have 768 numbers each, split into 12 heads of 64. Nothing has crossed between tokens yet — that only happens in the next step.
3. Attention and the mask — masks. Expand the attention matrix and follow its three steps: \(QK^\top\), the division by \(\sqrt{64} = 8\) with the causal mask, and the softmax. Check that everything above the diagonal is empty and that each row adds up to 1: the first token can only look at itself, the last one at everybody. Then flip through the heads and the blocks and find two patterns: a head where each token looks at the previous token, and heads that pour most of the weight on the first token. The lab measures both, and a third kind — induction heads⁴, which copy a pattern seen earlier in the text.
4. MLP — inside a block and where the parameters are. Expand it: 768 → 3072 → 768 with a GELU, applied to each token separately (no line crosses between tokens). In GPT-2 small that is 4.7M of the block's 7.1M parameters — the two-thirds of the note on parameters, with MHA.
5. Residual and normalization — pre-LN. Hover the dashed lines (the residual) and the dots before attention and before the MLP (LayerNorm). The norm is inside the branch: GPT-2, in 2019, was already pre-LN. Dropout is drawn too; at inference it does nothing.
6. Logits — the full path. After the 12th block only the last token's vector goes on: through a final LayerNorm, then multiplied by the embedding table, transposed (GPT-2 reuses it as the output layer), it gives 50,257 logits, one per vocabulary token. Training would use every row; generation only needs the last one.
7. Temperature and sampling — choosing the next token. Expand the probabilities and move the temperature from 0.2 to 2: watch the top bar fall and the tail rise. Switch top-k to top-p and press Generate several times at each setting. At \(T = 0.2\) the same token comes back almost every time; at \(T = 2\) with a large \(k\) odd tokens start to win.

What it does not show is as useful as what it shows. GPT-2 is a 2019 block: pre-LN already, but LayerNorm, a GELU MLP at \(4d\), full multi-head attention, learned absolute positions and biases everywhere — five of the six rows of the table of what changed are still on their 2017 side. Nothing is trained in the page (the weights are fixed), and an attention pattern is a picture of what one head does on one sentence, not an explanation of the model.

Lab: the full path, measured on GPT-2

Every claim of this chapter is measured in a Colab notebook on GPT-2 small — the same model that runs in Transformer Explainer. It runs in the browser, on CPU; there is nothing to install. It complements the Transformers exercise (to be published later), where you build the pieces from scratch: here you open a trained model. Nine sections, each ending in a checkpoint with a check value:

1. text → tokens, and why Portuguese costs twice as many as English;
2. where the 124M parameters live: two-thirds of each block is MLP, and the output layer is the embedding table;
3. the full path in shapes, and every position's prediction read off the logits;
4. block 0 written by hand with matrix products, matching the library to \(10^{-5}\);
5. what real heads do: the causal mask, a previous-token head, the first-token sink and induction heads copying a sequence the model has never seen;
6. attention removed from every block: two different sentences get the identical prediction;
7. the residual stream read at every depth, where the answer builds up block by block;
8. temperature, top-k and top-p on GPT-2's logits, generation, and the KV cache timed;
9. your turn: choose the decoding for three prompts of your own.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. A language model is embedding → \(L\) identical blocks → projection to the vocabulary. Inside the stack the shape is always \(n \times d\); every position predicts the next token.
2. The output layer turns the last vector into one score per vocabulary token (the logits) and the softmax turns the scores into probabilities. Every position predicts its next token, so training scores every row at once; generation reads only the last one.
3. A block is attention to talk (the only exchange of information between positions), an MLP to think (each token alone), and residuals that add corrections instead of replacing the vector.
4. Pre-LN replaced post-LN, and that is the single most consequential change since 2017 — it restores the identity path (\(x_L = x_0 + \sum f_\ell\)) and removes the need for warmup.
5. Think in terms of a residual stream that every block reads from and adds to, not a pipeline that transforms.
6. RMSNorm, SwiGLU, RoPE, GQA, no biases — five defaults you should assume unless told otherwise. Between two-thirds and three-quarters of a block's parameters are in the MLP.
7. Decoder-only won because every token is a training signal, one objective covers every task, and only causal attention supports a KV cache. Encoders survive as embedding and reranking models.
8. The next token is chosen, not computed: temperature reshapes the distribution without changing its order, top-k and top-p cut the tail, and greedy decoding loops. The chosen id is appended, the model runs again (the KV cache keeps the old rows), and the ids go back to text at the end.
9. Scaling laws are the Transformer's real product. With \(C \approx 6ND\), Kaplan said "go bigger", Chinchilla corrected it to "20 tokens per parameter", and practice went past both: overtrain a smaller model, because inference is paid forever.
10. Data is finite and inference-time compute is a second scaling axis. Both are where the frontier moved.