12. Transformers

The previous chapter built one mechanism. A Transformer is what you get when you decide that mechanism is enough — that you can throw away recurrence and convolution and keep only attention, plus the smallest possible amount of scaffolding around it¹.

The scaffolding turns out to be short: a per-position MLP, a residual connection, a normalization. That block, repeated, is the entire architecture — of GPT, of Llama, of ViT, of Whisper, of the diffusion backbones in chapter 22. Almost nothing else in modern AI has this property of being one design used everywhere.

This chapter has two jobs. The first is to show what the block does. The second is more useful and less often done: to show which parts of the 2017 design are still there and which were quietly replaced, because the diagram in the original paper is not the block you would write today.

From text to the next word: the full path

Before opening the block, it is worth seeing where it sits. A language model takes "the cat climbed the" and returns a probability for every vocabulary token being the next one:

flowchart TB
  T["'the cat climbed the'<br/>→ indices [12, 873, 4051, 88]"] --> E["embedding<br/>(n × d)"]
  E --> B1["block 1"] --> B2["block 2"] --> BD["…"] --> BL["block L"]
  BL --> N["final norm"]
  N --> U["unembedding: d → V<br/>(n × V)"]
  U --> P["softmax at the last position<br/>roof 0.41 · wall 0.18 · sofa 0.09 · …"]

(The indices and probabilities are illustrative.) Three things to notice:

• The shape does not change inside the stack. The embedding turns the \(n\) tokens into an \(n \times d\) matrix, and every block takes \(n \times d\) and returns \(n \times d\). That is what lets you stack 32 or 126 identical blocks. Only the last layer goes back to the vocabulary size \(V\).
• Every position makes a prediction. Row \(i\) of the output is the distribution of token \(i+1\). During generation only the last row matters; during training all of them count — the causal mask from the previous chapter makes sure none of them sees the answer.
• Generating is repeating. Sample a token from the distribution, append it to the input, and run again. The KV cache is what keeps that "run again" from recomputing everything.

Inside a block: talk, then think

A block has two sublayers, and each has a well-defined job:

• Attention is the only part where tokens exchange information. It is a meeting: each token asks, the others answer, and it leaves with a summary of what it heard.
• The MLP (or feed-forward) processes each token alone, with the same weights at every position. It is the individual work after the meeting: take what arrived and turn it into something useful. It is also where most of the model's stored "knowledge" lives.

Neither of them replaces the token's vector. Each computes a correction that the residual connection adds to it:

flowchart TB
  X["x  (n × d)"] --> N1["Norm"] --> A["attention<br/>(tokens talk)"] --> P1(("+"))
  X -- "residual" --> P1
  P1 --> N2["Norm"] --> F["MLP<br/>(each token alone)"] --> P2(("+"))
  P1 -- "residual" --> P2
  P2 --> Y["y  (n × d)"]

The script below pushes a 5-token sentence through a complete block, small enough to read (\(d = 8\)), prints the shape at every step, and then counts the parameters of the same block at the width of a real model (its labels are in Portuguese; the shapes and numbers are what matter):

"""Um bloco pré-LN inteiro em numpy: as formas em cada etapa e onde os parâmetros moram.

Primeiro uma frase de 5 tokens atravessa um bloco pequeno (d = 8) e cada etapa imprime sua forma —
o fluxo residual entra (n, d) e sai (n, d). Depois a mesma conta de parâmetros para d = 4096, a
largura de um modelo classe 8B, com GQA e SwiGLU.
"""
import numpy as np

rng = np.random.default_rng(0)

def rmsnorm(x, eps=1e-6):
    return x / np.sqrt((x ** 2).mean(-1, keepdims=True) + eps)

def softmax(s):
    e = np.exp(s - s.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)

def silu(x):
    return x / (1 + np.exp(-x))

def swiglu_width(d):
    return round(8 * d / 3 / 256) * 256 if d >= 256 else 8 * d // 3

def params(d, h, h_kv):
    dh = d // h
    attn = d * d + 2 * d * h_kv * dh + d * d            # W_Q, W_K e W_V (GQA), W_O
    ffn = 3 * d * swiglu_width(d)                       # W_1, W_3, W_2
    return attn, ffn, 2 * d                             # duas RMSNorm, um ganho cada

n, d, h, h_kv = 5, 8, 2, 1
dh, f = d // h, swiglu_width(d)
Wq, Wk, Wv, Wo = (rng.normal(size=s) / np.sqrt(d) for s in [(d, d), (d, h_kv * dh), (d, h_kv * dh), (d, d)])
W1, W3, W2 = rng.normal(size=(d, f)) / np.sqrt(d), rng.normal(size=(d, f)) / np.sqrt(d), rng.normal(size=(f, d)) / np.sqrt(f)

def show(nome, t):
    print(f"  {nome:<34}{str(t.shape):>10}")

x = rng.normal(size=(n, d))
print(f"n = {n} tokens, d = {d}, {h} heads de query, {h_kv} head de K/V")
show("x (fluxo residual, entrada)", x)

u = rmsnorm(x)
q = (u @ Wq).reshape(n, h, dh).transpose(1, 0, 2)
k = (u @ Wk).reshape(n, h_kv, dh).transpose(1, 0, 2).repeat(h // h_kv, axis=0)   # GQA: K/V repetidos
v = (u @ Wv).reshape(n, h_kv, dh).transpose(1, 0, 2).repeat(h // h_kv, axis=0)
show("Q, por head", q)
show("K, V (1 head, repetido)", k)
causal = np.triu(np.full((n, n), -np.inf), 1)
A = softmax(q @ k.transpose(0, 2, 1) / np.sqrt(dh) + causal)
show("A = softmax(QKᵀ/√d_h + M)", A)
o = (A @ v).transpose(1, 0, 2).reshape(n, d) @ Wo
x = x + o
show("x + atenção", x)

u = rmsnorm(x)
g = silu(u @ W1) * (u @ W3)
show("SwiGLU, camada oculta", g)
x = x + g @ W2
show("x + MLP (saída do bloco)", x)

print("\nd = 4096, 32 heads de query, 8 de K/V:")
attn, ffn, norm = params(4096, 32, 8)
total = attn + ffn + norm
for nome, p in [("atenção", attn), ("MLP (SwiGLU)", ffn), ("normalização", norm)]:
    print(f"  {nome:<14}{p / 1e6:8.1f} M  {100 * p / total:4.0f}%")
print(f"  {'bloco':<14}{total / 1e6:8.1f} M")
print(f"  × 32 blocos  {32 * total / 1e9:8.2f} B")

n = 5 tokens, d = 8, 2 heads de query, 1 head de K/V
  x (fluxo residual, entrada)           (5, 8)
  Q, por head                        (2, 5, 4)
  K, V (1 head, repetido)            (2, 5, 4)
  A = softmax(QKᵀ/√d_h + M)          (2, 5, 5)
  x + atenção                           (5, 8)
  SwiGLU, camada oculta                (5, 21)
  x + MLP (saída do bloco)              (5, 8)

d = 4096, 32 heads de query, 8 de K/V:
  atenção           41.9 M    24%
  MLP (SwiGLU)     135.3 M    76%
  normalização       0.0 M     0%
  bloco            177.2 M
  × 32 blocos      5.67 B

The residual stream enters \(5 \times 8\) and leaves \(5 \times 8\); only the internal steps change shape. The attention matrix is \(5 \times 5\) per head (every token against every token), and the MLP's hidden layer is wider than \(d\). At real width, 32 blocks come to 5.7B parameters; with the embedding table and the output layer (\(2 \times 128\text{k} \times 4096 \approx 1\text{B}\)) that gets close to 7B. Llama 3 8B uses an even wider MLP (14336 instead of 11008), which makes up the 8B.

The block, and the six things that changed

The block the script above assembled is the 2026 one. The 2017 block differed in six places:

Configure it. The default is what a model released this year looks like; switch every control to its first option and you get the 2017 paper, exactly.

Two of those changes are worth more than the rest.

Pre-LN: where the normalization goes

The 2017 block normalizes after the residual addition:

which means the residual path passes through a normalizer at every layer. The gradient no longer has the clean identity route that chapter 9 identified as the whole point of a residual connection, and the consequence is concrete: post-LN Transformers do not train without a learning-rate warmup, and get harder to train the deeper they get⁵.

Moving the normalization inside the branch fixes it:

Now the residual stream is untouched from the loss down to layer 1. You can see it in one line: unrolling the recurrence, the output of a pre-LN stack of \(L\) blocks is the input plus the sum of every correction,

and that \(I\) is the express lane: the gradient reaches the first layer intact, at least through that term, no matter how many blocks sit in between. In post-LN every block ends in a \(\text{Norm}\) and the sum never appears on its own — there is no clean \(I\). This single rearrangement is why 100-layer stacks are unremarkable, and it is the reason the warmup schedules in old tutorials look like superstition — they were compensating for a design that has been abandoned.

Normalization and the feed-forward layer

RMSNorm⁶ drops the mean subtraction from LayerNorm and keeps only the rescaling:

An example with \(x = [3,\ 4]\) shows the difference. LayerNorm subtracts the mean (3.5) and divides by the standard deviation (0.5): out comes \([-1,\ 1]\). RMSNorm only divides by the root mean square, \(\sqrt{(9+16)/2} \approx 3.54\): out comes \([0.85,\ 1.13]\). Both put the vector on a standard scale, which is what the next layer needs; centring turned out to be dispensable.

One reduction instead of two, no bias, and no measurable quality cost. Universal since Llama.

The MLP changed too. The original is two matrices with a ReLU between them and a 4× hidden width. The modern one is SwiGLU⁷ — three matrices, where one branch gates the other:

Read the formula as two branches side by side: \(xW_3\) proposes some content, and \(\text{Swish}(xW_1)\) decides, dimension by dimension, how much of it gets through — near zero closes, larger values open or amplify. It is a dimmer controlled by the input itself, the same principle as the LSTM's gates.

The hidden width drops to \(\tfrac{8}{3}d\) so the parameter count stays put, and the multiplicative gate buys a consistent, if modest, quality win. It has survived every ablation since 2020.

Encoder, decoder, and which one survived

The 2017 model was an encoder–decoder for translation: an encoder that sees the whole source bidirectionally, a decoder that generates causally and reads the encoder through cross-attention. Three families grew out of it, and their fortunes diverged sharply.

Why did decoder-only win a competition it looks like it should lose? It sees less context per token, after all. Three reasons, and none of them is about elegance:

1. Every token is a training signal. With the causal mask, one forward pass over a sequence of length \(n\) produces \(n\) predictions. BERT masks 15% of tokens and learns from those, so it extracts roughly a sixth of the signal per unit of compute.
2. One objective covers every task. "Predict the next token" subsumes classification, extraction, translation and dialogue once the model is good enough to be instructed — no task-specific head, no separate fine-tune.
3. The KV cache only works causally. A bidirectional model must re-encode everything when the input changes; a causal one appends. That is the difference between an interactive chat and a batch job.

Scaling: three different questions, three different answers

The Transformer's real property is not any single design choice; it is that loss falls predictably as you add parameters, data and compute, over many orders of magnitude, with no sign of a wall³. That reliability is what justified spending a billion dollars on a training run.

But "scale it up" hides a question — scale what? The budget is compute \(C\), measured in FLOPs, and it is split between the model size \(N\) (parameters) and the amount of data \(D\) (training tokens):

The 6 comes from counting: each parameter does about 2 operations per token on the way forward (one multiply, one add) and 4 on the way back. With \(C\) fixed, doubling the model forces you to halve the data. "Scale what?" is the question of how to split \(C\) between \(N\) and \(D\) — and the answer changed twice.

2020, Kaplan et al.³: for a fixed compute budget, make the model bigger. GPT-3 followed the advice: 175B parameters on 300B tokens, under 2 tokens per parameter.

2022, Chinchilla⁴: that was wrong, and the error was in the learning-rate schedule of the original experiments. Redone properly, the optimum is to scale parameters and data together — roughly 20 tokens per parameter. Chinchilla, at 70B parameters and 1.4T tokens, beat the 280B Gopher trained on the same compute. Overnight, everybody's models got smaller and their datasets got bigger.

Now: nobody trains at the Chinchilla optimum either, and the panel shows why. Tick price inference too. Chinchilla minimizes the cost of one training run; a deployed model is paid for by every token it ever serves, and serving cost scales with \(N\), not with \(D\). So the right move is to overtrain a smaller model — Llama-3-8B saw 15T tokens, about 1900 per parameter, nearly 100× past Chinchilla. It is compute-inefficient to train and much cheaper to own.

Put all three on the same ruler with \(C \approx 6ND\):

Chinchilla and Llama 3 8B cost almost the same to train. The second is 9× smaller, so every token it generates in deployment costs about 9× less — and that is the cost paid billions of times.

Two more corrections belong here:

• Data is finite. High-quality text is a limited resource, and repeating it has sharply diminishing returns after a few epochs⁹. This is a real constraint on the \(D\) axis and a large part of why synthetic data and curation are now serious research areas.
• Training compute is no longer the only axis. A model can also be given more compute at inference — sampling many chains of thought, searching, verifying. That curve scales too, and it is the subject of chapter 15.

A minimal modern block

import torch, torch.nn as nn, torch.nn.functional as F

class Block(nn.Module):
    """One decoder block, 2026 conventions: pre-norm, RMSNorm, GQA, SwiGLU, no biases."""
    def __init__(self, d, n_heads, n_kv_heads):
        super().__init__()
        self.h, self.kv, self.dh = n_heads, n_kv_heads, d // n_heads
        self.n1, self.n2 = nn.RMSNorm(d), nn.RMSNorm(d)
        self.wq = nn.Linear(d, d, bias=False)
        self.wk = nn.Linear(d, n_kv_heads * self.dh, bias=False)   # GQA: fewer K/V heads
        self.wv = nn.Linear(d, n_kv_heads * self.dh, bias=False)
        self.wo = nn.Linear(d, d, bias=False)
        h = round(8 * d / 3 / 256) * 256                          # SwiGLU keeps params level
        self.w1, self.w3 = nn.Linear(d, h, bias=False), nn.Linear(d, h, bias=False)
        self.w2 = nn.Linear(h, d, bias=False)

    def attn(self, x, freqs):
        B, T, _ = x.shape
        q = self.wq(x).view(B, T, self.h, self.dh).transpose(1, 2)
        k = self.wk(x).view(B, T, self.kv, self.dh).transpose(1, 2)
        v = self.wv(x).view(B, T, self.kv, self.dh).transpose(1, 2)
        q, k = apply_rope(q, freqs), apply_rope(k, freqs)           # position lives here, not at the input
        o = F.scaled_dot_product_attention(                         # FlashAttention when available
            q, k, v, is_causal=True, enable_gqa=True)
        return self.wo(o.transpose(1, 2).reshape(B, T, -1))

    def forward(self, x, freqs):
        x = x + self.attn(self.n1(x), freqs)                        # pre-norm: the stream stays clean
        h = self.n2(x)
        return x + self.w2(F.silu(self.w1(h)) * self.w3(h))         # SwiGLU

Compare it to the 2017 diagram and the diff is exactly the six controls in the panel above — check it line by line against the table at the start of the chapter. The mechanism did not change; the engineering around it did. apply_rope is the rotation from the previous chapter, left out for brevity.

Key takeaways

1. A language model is embedding → \(L\) identical blocks → projection to the vocabulary. Inside the stack the shape is always \(n \times d\); every position predicts the next token.
2. A block is attention to talk (the only exchange of information between positions), an MLP to think (each token alone), and residuals that add corrections instead of replacing the vector.
3. Pre-LN replaced post-LN, and that is the single most consequential change since 2017 — it restores the identity path (\(x_L = x_0 + \sum f_\ell\)) and removes the need for warmup.
4. Think in terms of a residual stream that every block reads from and adds to, not a pipeline that transforms.
5. RMSNorm, SwiGLU, RoPE, GQA, no biases — five defaults you should assume unless told otherwise. Between two-thirds and three-quarters of a block's parameters are in the MLP.
6. Decoder-only won because every token is a training signal, one objective covers every task, and only causal attention supports a KV cache. Encoders survive as embedding and reranking models.
7. Scaling laws are the Transformer's real product. With \(C \approx 6ND\), Kaplan said "go bigger", Chinchilla corrected it to "20 tokens per parameter", and practice went past both: overtrain a smaller model, because inference is paid forever.
8. Data is finite and inference-time compute is a second scaling axis. Both are where the frontier moved.