What Changed Since 2017

Beyond the running example. The toy block of the previous pages is GPT-2's design from 2019: LayerNorm before each sublayer, a GELU MLP, full multi-head attention, positions learned at the input. This page is about which parts of the 2017 block are still in the one you would write today, which were replaced, and which family of models — encoder, decoder or both — won. ← Generation · next: Scaling →

The block, and the six things that changed

The block the script of the block page assembles is the 2026 one. The 2017 block differed in six places:

Configure it. The default is what a model released this year looks like; switch every control to its first option, tick the biases, and you get the 2017 paper, exactly.

Two of those changes are worth more than the rest.

Pre-LN: where the normalization goes

The 2017 block normalizes after the residual addition:

which means the residual path passes through a normalizer at every layer. The gradient no longer has the clean identity route that chapter 9 identified as the whole point of a residual connection, and the consequence is concrete: post-LN Transformers do not train without a learning-rate warmup, and get harder to train the deeper they get².

Moving the normalization inside the branch fixes it:

Now the residual stream is untouched from the loss down to layer 1. You can see it in one line: unrolling the recurrence, the output of a pre-LN stack of \(L\) blocks is the input plus the sum of every correction,

and that \(I\) is the express lane: the gradient reaches the first layer intact, at least through that term, no matter how many blocks sit in between. In post-LN every block ends in a \(\text{Norm}\) and the sum never appears on its own — there is no clean \(I\). This single rearrangement is why 100-layer stacks are unremarkable, and it is the reason the warmup schedules in old tutorials look like superstition — they were compensating for a design that has been abandoned.

Normalization and the feed-forward layer

RMSNorm³ drops the mean subtraction from LayerNorm and keeps only the rescaling:

An example with \(x = [3,\ 4]\) shows the difference. LayerNorm subtracts the mean (3.5) and divides by the standard deviation (0.5): out comes \([-1,\ 1]\). RMSNorm only divides by the root mean square, \(\sqrt{(9+16)/2} \approx 3.54\): out comes \([0.85,\ 1.13]\). Both put the vector on a standard scale, which is what the next layer needs; centring turned out to be dispensable.

One reduction instead of two, no bias, and no measurable quality cost. Universal since Llama.

The MLP changed too. The original is two matrices with a ReLU between them and a 4× hidden width. The modern one is SwiGLU⁴ — three matrices, where one branch gates the other:

Read the formula as two branches side by side: \(xW_3\) proposes some content, and \(\text{Swish}(xW_1)\) decides, dimension by dimension, how much of it gets through — near zero closes, larger values open or amplify. It is a dimmer controlled by the input itself, the same principle as the LSTM's gates.

The hidden width drops to \(\tfrac{8}{3}d\) so the parameter count stays put — three matrices of \(d \times \tfrac{8}{3}d\) are \(8d^2\), the same as two of \(d \times 4d\) — and the multiplicative gate buys a consistent, if modest, quality win. That is the whole case for it, and it has been enough for every major open model since Llama.

Encoder, decoder, and which one survived

The 2017 model was an encoder–decoder for translation: an encoder that sees the whole source bidirectionally, a decoder that generates causally and reads the encoder through cross-attention. Three families grew out of it, and their fortunes diverged sharply.

Why did decoder-only win a competition it looks like it should lose? It sees less context per token, after all. Three reasons, and none of them is about elegance:

1. Every token is a training signal. With the causal mask, one forward pass over a sequence of length \(n\) produces \(n\) predictions. BERT masks 15% of tokens and learns from those, so it extracts roughly a seventh of the signal (one token in 6.7) per unit of compute.
2. One objective covers every task. "Predict the next token" subsumes classification, extraction, translation and dialogue once the model is good enough to be instructed — no task-specific head, no separate fine-tune.
3. The KV cache only works causally. A bidirectional model must re-encode everything when the input changes; a causal one appends. That is the difference between an interactive chat and a batch job.

A minimal modern block

import torch, torch.nn as nn, torch.nn.functional as F

class Block(nn.Module):
    """One decoder block, 2026 conventions: pre-norm, RMSNorm, GQA, SwiGLU, no biases."""
    def __init__(self, d, n_heads, n_kv_heads):
        super().__init__()
        self.h, self.kv, self.dh = n_heads, n_kv_heads, d // n_heads
        self.n1, self.n2 = nn.RMSNorm(d), nn.RMSNorm(d)
        self.wq = nn.Linear(d, d, bias=False)
        self.wk = nn.Linear(d, n_kv_heads * self.dh, bias=False)   # GQA: fewer K/V heads
        self.wv = nn.Linear(d, n_kv_heads * self.dh, bias=False)
        self.wo = nn.Linear(d, d, bias=False)
        h = round(8 * d / 3 / 256) * 256                          # SwiGLU keeps params level
        self.w1, self.w3 = nn.Linear(d, h, bias=False), nn.Linear(d, h, bias=False)
        self.w2 = nn.Linear(h, d, bias=False)

    def attn(self, x, freqs):
        B, T, _ = x.shape
        q = self.wq(x).view(B, T, self.h, self.dh).transpose(1, 2)
        k = self.wk(x).view(B, T, self.kv, self.dh).transpose(1, 2)
        v = self.wv(x).view(B, T, self.kv, self.dh).transpose(1, 2)
        q, k = apply_rope(q, freqs), apply_rope(k, freqs)           # position lives here, not at the input
        o = F.scaled_dot_product_attention(                         # FlashAttention when available
            q, k, v, is_causal=True, enable_gqa=True)
        return self.wo(o.transpose(1, 2).reshape(B, T, -1))

    def forward(self, x, freqs):
        x = x + self.attn(self.n1(x), freqs)                        # pre-norm: the stream stays clean
        h = self.n2(x)
        return x + self.w2(F.silu(self.w1(h)) * self.w3(h))         # SwiGLU

Compare it to the 2017 diagram and the diff is exactly the six controls in the panel above — check it line by line against the table at the top of this page. The mechanism did not change; the engineering around it did. apply_rope is the rotation from the previous chapter, left out for brevity.

← Generation · next: Scaling →