Artificial Neural Networks and Deep Learning · Insper

Transformers

From text to the next token — and what changed since 2017

What does a language model compute?

Input: "Data visualization empowers users to". What comes out of GPT-2?

  • Not a word: a probability for each of the 50,257 tokens of its vocabulary.
  • Measured in the lab: visualize 0.158, then create, see, make… — 50,257 numbers that add up to 1.
  • To write text: pick one, append it to the input, run again.

The full path

text
"…users to"
→
tokens
n indices
→
embedding
n × d
→
L blocks
n × d → n × d
→
norm +
unembedding
n × V
→
softmax
last row
V probabilities

GPT-2 small: tokens in the prompt, numbers per token, blocks, tokens in the vocabulary.

unembedding: the output layer, d → V, one score (a logit) per vocabulary token

  • The shape does not change inside the stack — that is what lets you stack 12 or 126 identical blocks.
  • Every position predicts: row is the distribution of token .
  • Generating is repeating: sample, append, run again. The KV cache (the K and V of the positions already seen, kept) keeps "again" from recomputing everything.

Tokens are not words

"Data visualization empowers users to" → 6 tokens:

Data · visualization · em · powers · users · to

  • The space is part of the token: " the", "the" and " The" are 3 different tokens.
  • 50,257 pieces, built on English web text.

Same sentence in Portuguese: how many tokens per word?

  • English: 1.08 – 1.15 tokens per word (tokens ÷ words, on the lab's three sentences).
  • Portuguese: 1.93 – 2.00 — gato = g + ato, cachorro = 4 pieces.
  • Twice the compute and half the context, for the same sentence.

Every position predicts

Logits of the prompt: (1, 6, 50257) = (sentences, positions, vocabulary) — one distribution per position. GPT-2, measured:

after… top guess true next p(true)
Data . visualization 11.91
Data visualization \n em 11.86
Data visualization em powers powers 0.423 0.86
…empowers you users 0.109 2.22
…empowers users to to 0.847 0.17
…users to visualize ? — —
  • Training scores every row at once: the loss is the mean of over the 5 rows with an answer: 5.40 nats (nat: the unit of the natural log).
  • Generation only needs the last row.

Inside a block: talk, then think

x: the residual stream, n × d (one vector per token) · Norm: LayerNorm in GPT-2 · h: the stream after attention

A pre-LN Transformer block: the residual stream carries x along the top; attention, where the tokens exchange information, and then the MLP, which works on each token alone, each read a normalized copy and add a correction back to the stream. x n × d + h + y n × d residual stream: x passes untouched h passes untouched Norm attention tokens talk the last token reads the earlier ones (causal mask) Norm MLP each token alone the same MLP on each token, separately

Neither sublayer replaces the vector: each adds a correction. The MLP holds most of the stored knowledge. in, out — stacking is just repeating.

Exercise · follow the tensor

Block 0 of GPT-2 small, the 6-token prompt. Fill it in before the next slide.

step shape
, residual stream in ?
Q, K, V per head ?
?
+ attention ?
MLP hidden layer ?
, residual stream out ?

tokens, numbers per token

12 heads, each with

: the causal mask, above the diagonal

GPT-2's MLP:

Answer as (heads ×) tokens × numbers.

Follow the tensor · answer

6×768Q, K, V12 heads of 6412×6×64QKᵀ/√64 + Msoftmax12×6×6A·V, concatWO, + x6×768MLP768 → 30726×30723072 → 768+ h6×768

step shape where it comes from
6 × 768 tokens, numbers each
Q, K, V per head 12 × 6 × 64 is 6 × 768, cut into 12 heads of 768/12 = 64
12 × 6 × 6 per head, = (6 × 64)(64 × 6): every token against every token
+ attention 6 × 768 is 12 × 6 × 64; heads side by side, 12 · 64 = 768; , then add
MLP hidden layer 6 × 3072 = 4 · 768 = 3072, each token alone
6 × 768 3072 → 768, then add : the same shape that came in

One block by hand = the library

Block 0 of GPT-2 with nothing but @, a mask, a softmax, LayerNorm and GELU — the model's own weights:

compared with the library max difference
block 0 output, 6 × 768
attention weights, 12 × 6 × 6
logits after all 12 blocks + final norm + wteᵀ

wte: the token embedding table, 50,257 × 768; transposed, it is the output layer (768 → 50,257).

Float32 rounding, nothing else: the two equations of "Inside a block" are the model.

Simulator · one block, line by line

x · residual stream in

The running example: from y to “␣roof”

The toy (d = 4, one block, V = 12), last token of "the cat climbed the": .

  1. Final norm: , →
  2. Unembedding (): column of ␣roof = →
  3. Softmax: → ␣
token
␣roof 2.491 0.395
␣wall 1.463 0.141
␣sofa 0.999 0.089
␣floor 0.970 0.086
␣cat 0.634 0.062
7 others 0.227

Every row predicts its next token: p(target) = 0.500, 0.444, 0.274, 0.311, 0.395 → mean −ln p = 0.979 nats (uniform: ln 12 = 2.485). Without attention the last row says ␣cat 0.455, ␣roof 0.143.

Where the parameters are

GPT-2 small, one block, measured (; every linear has weights and biases):

part count parameters
attention: (768 → 3 · 768) and (768 → 768) 2,362,368
MLP: 768 → 3072 → 768 4,722,432
two LayerNorms (gain and bias) 3,072
the block the sum 7,087,872 · MLP 66.6%
  • MHA (multi-head attention: each head has its own K and V): MLP = two-thirds. GQA (grouped-query attention: 32 query heads share 8 K/V heads), 8B class, : MLP M, attention 42M → three-quarters of 177M.
  • Whole GPT-2: 124.4M. The embedding table, M, is 31%, and the output layer is that table.
  • MoE (mixture of experts: many MLPs, and a router sends each token to a few of them) replaces the MLP, because that is where the parameters are.

Check yourself: remove attention

Zero the output of every attention sublayer, keep the MLPs. What is the prediction based on now?

  • Two 10-token sentences, both ending in the:
…cat… climbed onto the …cash… walked to the
with attention roof 0.083 · bed 0.059 bank 0.092 · front 0.028
attention removed same 0.047 · first 0.019 same 0.047 · first 0.019
  • Without attention the two distributions are identical (difference 0): the last position sees only the and its index (9). Mean loss on the lab's 72-token paragraph: 3.96 → 5.71 nats.

What real heads do

144 heads (12 blocks × 12), scored on three jobs — GPT-2 small, measured:

job best head (block.head) weight
look at the previous token 4.11 1.00
look at the first token ("sink") 5.1 0.99 — 64 of 144 heads put > half there
induction: "last time I saw this, what came next?" 5.5 0.95

4.11 = block 4, head 11, counting from 0 · weight = the mean attention the head puts on the target (0 to 1)

  • A random 50-token sequence, repeated: loss 11.95 nats on the first copy (a random guess would give ), 0.21 on the second, top-1 100%.
  • The model copies a sequence it has never seen — and those heads are how.

Go and try · Transformer Explainer

GPT-2 small running in your browser, every stage drawn for the text you type — poloclub.github.io/transformer-explainer (Cho et al., 2024).

  1. Embedding: count the tokens of the example, then of a Portuguese sentence. The position vector is learned — neither sinusoid nor RoPE.
  2. Attention: expand the matrix. Everything above the diagonal is empty. Flip heads and blocks: find a previous-token head and heads that load the first token.
  3. Temperature: move it from 0.2 to 2 and press Generate a few times at each. Watch the top bar and the tail.

The first visit downloads about 660 MB of weights; the chapter has a step-by-step guide to all seven stages.

The six things that changed since 2017

piece 2017 GPT-2 (2019) 2026
norm placement after the sum (post-LN) before the sublayer (pre-LN) pre-LN
normalizer LayerNorm LayerNorm RMSNorm
MLP ReLU, 2 matrices, GELU, 2 matrices, SwiGLU, 3 matrices,
attention MHA MHA GQA — a 4× smaller KV cache (8 K/V heads for 32 Q heads)
position sinusoidal, at the input learned, at the input RoPE, on Q and K
bias everywhere everywhere none

The mechanism did not change; the engineering around it did. GPT-2 — the model in the Explainer and in the lab — already had the first row. RoPE: rotary position embedding — Q and K are rotated by an angle proportional to the position, so their score depends on the distance between the tokens.

Simulator · configure the block

Pre-LN: where the norm goes

  • Post-LN puts a normalizer on the residual path at every layer: the gradient loses its identity route — no training without warmup (a learning rate that starts small), harder with depth.
  • Pre-LN unrolls into the input plus the sum of every correction (: what block adds; : the identity):

  • That is why 100-layer stacks are unremarkable.

The residual stream, read at every depth

Apply the final norm and the output layer to the stream after each block ("logit lens"). Final answer: visualize. Rank 1 = the most likely of 50,257.

read after block embedding 3 6 8 9 10 11 12
rank of visualize 8532 381 62 12 2 1 1 1
its probability 0.000 0.000 0.002 0.012 0.285 0.919 0.941 0.158
  • The answer is accumulated in one stream, block by block — not computed at the end.
  • Block 12 lowers the confidence to the 0.158 of the first slide: the last block spreads mass over the alternatives.

RMSNorm and SwiGLU

RMSNorm keeps the rescaling, drops the centring. :

  • LayerNorm: mean 3.5, std 0.5 →
  • RMSNorm: only the root mean square →

One reduction instead of two, no measurable quality cost.

SwiGLU — one branch gates the other:

Swish(u) = u·σ(u), σ: the sigmoid · ⊙: elementwise product

  • Three matrices of : , the same as two of ().
  • A small, consistent gain — enough for every major open model since Llama.

Encoder, decoder: which one survived

family attention trained by in 2026
encoder-only — BERT bidirectional masked tokens embeddings, retrieval, rerankers; cannot generate
encoder–decoder — T5, BART both + cross sequence to sequence where the two sides differ: speech (Whisper), text into images
decoder-only — GPT, Llama, Claude causal next token won — every frontier language model

Decoder-only sees less context per token. Why did it win?

Why decoder-only won

  1. Every token is a training signal. One pass over tokens gives predictions; BERT learns from the ~15% it masks.
  2. One objective covers every task. Next-token prediction subsumes classification, extraction, translation, dialogue — once the model can be instructed.
  3. The KV cache only works causally. A bidirectional model re-encodes everything when the input grows; a causal one appends.

BERT trains on [MASK], a token that never appears at inference. Next-token prediction has no such gap: training is deployment.

Choosing the next token: temperature

zi: the logit of token i (the model's output before the softmax) · T: the temperature

GPT-2, after …empowers users to:

p( visualize) tokens for 90%
0.2 0.976 1
0.5 0.656 6
0.8 0.294 97
1.0 0.158 294
2.0 0.013 5,956

tokens for 90%: how many of the most likely add up to 0.9

  • sharpens, flattens; the order never changes — greedy decoding (always the top token) is the same at every .
  • : greedy. : uniform over 50,257.
  • Logits , : on top. doubles them to : .

Top-k, top-p — and what comes out

  • Top-k keeps the most likely; top-p keeps the smallest set that adds up to — its size follows the model's confidence:
prompt most likely (top-1) top-p 0.9 keeps
…44th President of the United States 0.998 1 token
Once upon a time, there was a man 0.044 3,651 tokens
  • Greedy: "visualize and visualize data in a way that is easy to understand." — then it loops.
  • , no filter: "along button pilot discriminationave…" — the tail wins.

Generating the running example

␣roof ␣wall ␣sofa
0.5 0.761 0.097 0.039
1 0.395 0.141 0.089
2 0.208 0.124 0.098
  • top-k 2: ␣roof 0.737, ␣wall 0.263
  • top-p 0.9 at : 8 of 12 stay (a flat distribution)
  • + top-p 0.9: 6 stay; running sum 0.626, 0.770, 0.845… → gives ␣roof, gives ␣sofa
step input ids next
1 [0, 2, 4, 5, 1] ␣roof 0.395
2 … + 6 . 0.538
3 … + 10 <|endoftext|> 0.401
  • Rows 1–5 never change (causal mask): the KV cache keeps their K and V; each step computes one row.
  • Decode: the ␣cat ␣climb ed ␣the ␣roof . → "the cat climbed the roof."

Simulator · choose, append, run again

Scaling: three questions, three answers

The budget is compute , in FLOPs (floating-point operations), split between parameters and training tokens :

  • 6 = about 2 operations per parameter per token forward (multiply, add) + 4 backward.
  • With fixed, doubling the model halves the data.

Chinchilla: 70B () parameters, 1.4T () tokens. What was ?

  • FLOPs.

Simulator · split the compute

Kaplan, Chinchilla, and practice

model tokens / parameter, (FLOPs)
GPT-3 (2020) — go bigger 175B 0.3T 1.7
Chinchilla (2022) — together 70B 1.4T 20
Llama 3 8B (2024) — overtrain 8B 15T 1875 ≈ 1900
  • Chinchilla and Llama 3 8B cost almost the same to train.
  • The second is 9× smaller (70B / 8B ≈ 9), so every token it serves costs about 9× less — and that cost is paid billions of times.

Two more corrections

  • Data is finite. High-quality text is limited, and repeating it has sharply diminishing returns after a few epochs: the axis has a wall.
  • Training compute is no longer the only axis. More compute at inference — many chains of thought, search, verification — scales too (the LLMs class).

The Transformer's real product is not one design choice: it is that loss falls predictably with parameters, data and compute.

Lab · the full path, measured on GPT-2

Colab notebook, ~2 h, GPT-2 small on CPU. Each section ends in a checkpoint with a check value.

  1. text → tokens, in two languages
  2. where the 124M parameters live
  3. the full path, in shapes
  4. one block by hand = the library
  5. what real heads do
  1. remove attention, keep the rest
  2. the residual stream at every depth
  3. temperature, top-k, top-p, KV cache
  4. your turn: choose the decoding

open in Colab →

Takeaways

  1. A language model is embedding → identical blocks → projection to the vocabulary; all the way, and every position predicts.
  2. A block: attention to talk, MLP to think, residuals that add corrections.
  3. Pre-LN restores the identity path; think in terms of a residual stream.
  4. RMSNorm, SwiGLU, RoPE, GQA, no biases; two-thirds to three-quarters of a block is MLP.
  5. Decoder-only won: every token a signal, one objective, a KV cache.
  6. The next token is chosen: temperature reshapes, top-k / top-p cut the tail.
  7. : Kaplan, Chinchilla, then overtrain small, because inference is paid forever.

Outline: (1) the full path, text to next token, on a real GPT-2; (2) inside a block: talk, then think; (3) Transformer Explainer; (4) the six things that changed since 2017; (5) which family won; (6) choosing the next token; (7) scaling. Closes with the Colab lab.

Let the class answer "the next word" first, then correct: a distribution. Choosing is a separate step, with its own knobs, at the end of the class.

In the lab (section 4) the bloco_a_mao function prints exactly these six shapes.

The lab's block_by_hand with one head, 3 tokens, d = 4 and hand-picked weights. Step with Next and read the caption: the formula with the numbers of token 3 (A row 3 = 0.107, 0.787, 0.107; y31 = 1.651). Then the cases: "row 2 × 10" (u does not change, only the residual carries the size), "only the last token changes" (rows 1 and 2 do not move: causal mask), "attention removed" (h = x, each row alone).

The pages of the chapter walk the toy from text to here, every number checkable. GPT-2 does the same three lines with d = 768 and V = 50,257, and its W_U is wte transposed; the toy has its own W_U.

Ten minutes in class, or as homework before the lab. The guide on the page ties each stage to a section of the chapter. GPT-2 is a 2019 block: next slide shows which parts are already modern.

Start from the default (2026). Switch every control to its first option and tick the biases: the 2017 paper. Watch the residual line turn grey with post-LN, and the parameter split: two-thirds MLP with MHA, three-quarters with GQA.

Greedy writes ␣roof every time. Sampling sometimes writes ␣sofa: a good sentence the model also considered. The end-of-text token is how the model stops itself.

Default: T = 0.7, top-p 0.9, u = 0.5 → ␣roof; the last line runs again and predicts "." (0.538). Then: greedy (u does not matter), hot T = 2 (the tail grows), top-k 2; move u to 0.8 → ␣sofa.

Find the minimum of the curve at 1e23: about 20 tokens per parameter. Then tick "price inference too" and drag the tokens to serve from 1e10 to 1e15: the cheapest model shrinks, and is trained on far more tokens than Chinchilla would say.