Artificial Neural Networks and Deep Learning · Insper

Large Language Models

The distribution, measured

Start with a question

A model is trained only to predict the next token of web text.
Why can it then translate, write code and hold a conversation?

  • Compressing text well requires modelling whatever produced the text.
  • To predict the last word of a detective novel you need the plot.
  • To predict the line after def you need the function's contract.
  • Next-token prediction is a compression objective, and compression at this scale forces structure.

One objective

xt: the token at position t · x<t: the tokens before it · pθ: the probability the model gives · T: tokens in the text · natural log, so the loss comes out in nats

  • Self-supervised: the label of each position is the next token of the same text — any text is training data.
  • Divide by and exponentiate: the perplexity . A mean loss of 3 nats per token → , as unsure as a uniform pick among 20 tokens.

The base model is not the assistant. It continues documents. Asked a question, it may write three more questions — a good continuation of a quiz.

How big, and on how much text

— training FLOPs from parameters and tokens : 2 per parameter per token forward, 4 backward (the Transformers chapter).

model tokens / parameter =
GPT-3 (2020) 175B 0.3T 0.3T / 175B = 1.7
Chinchilla (2022) 70B 1.4T 1.4T / 70B = 20 — the compute-optimal rule
Llama 3 8B (2024) 8B 15T 15T / 8B ≈ 1,900
SmolLM2-135M (2025) — the lab's model 135M 2T 2T / 135M ≈ 15,000

M = 10⁶ · B = 10⁹ · T = 10¹²

Deployed models are trained far past Chinchilla: every token they serve costs ≈ FLOPs, paid forever.

A scaling law, measured in the lab

Pythia: models of every size trained on the same 300B tokens, in the same order. Loss on WikiText-2:

model non-embedding loss (nats/token) drop
pythia-70m 18.9M 4.091
pythia-160m 85.1M (×4.5) 3.500 0.59
pythia-410m 302.3M (×3.6) 3.006 0.49
pythia-1b 805.7M (×2.7) 2.820 0.19
  • A straight-line fit of against : slope , i.e. . Each step buys a smaller drop than the last: the shape of a power law.
  • Along : pythia-70m stops improving after 67.1B / 18.9M ≈ 3,500 tokens per parameter (loss 4.06); with the full 300B, 4.09. Data helps until size is the limit.

Non-embedding N (as in Kaplan et al.): embedding tables are lookups, not compute.

Tokenization: BPE in three merges

BPE (byte-pair encoding): start from bytes; merge the most frequent adjacent pair into a new symbol; repeat until the vocabulary has 32k–256k entries.

Byte-pair encoding trained on the toy corpus aaabdaaabac. At each step the pairs of neighbouring symbols are counted, the most frequent pair becomes a new symbol of the vocabulary and every occurrence of it in the text is merged, left to right. Three merges, aa to Z, ab to Y and ZY to X, take the text from 11 symbols to 5, and X stands for aaab. VOCABULARY LENGTH OF THE TEXT PAIR COUNTS

What the model actually reads

How many r's in strawberry?

  • SmolLM2: strawberry is one token; strawberry is st raw berry.
  • The letters are not in the input.

How does 12345 reach the model?

  • GPT-2: 1234 → 12 34, 12345 → 123 45 — the same digit in different pieces.
  • SmolLM2, Llama 2, Qwen: one digit per token.

Notation: the space at the start of strawberry is part of the token. It is a different token, with a different number, from strawberry without the space.

The price of Portuguese

The same paragraph, in English and in Portuguese. How many times more tokens does Portuguese cost?

tokenizer vocabulary EN PT PT / EN
GPT-2 50,257 93 168 1.81
SmolLM2 49,152 93 167 1.80
Qwen2.5 151,665 92 119 1.29
  • Merges fitted on mostly English text: more latency, more money, less context per idea.
  • A multilingual tokenizer cuts the tax; it does not remove it.

One trailing space

"The capital of France is" against "The capital of France is␣". What happens to the probability of Paris?

  • Without the space: P( Paris) = 0.0938. With it: P(Paris) = 0.000022 — ratio 0.0938 / 0.0000218 ≈ 4,303× smaller.
  • In training text a space is glued to the next word. A bare space at the end is off-distribution, and the model bets on digits.
  • And even without the space the base model prefers the (0.262): it continues text, it does not answer.

Post-training: from continuation to assistant

  1. SFT (supervised fine-tuning) on demonstrations — teaches the format of being an assistant.
  2. Preference optimization (RLHF — RL from human feedback — with PPO, or DPO) — which of two answers is better, which demonstrations cannot say.
  3. RL from a verifier (RLVR, RL with verifiable rewards) — getting it right, when "right" can be checked.

RLHF trains a reward model on people's comparisons and optimizes against it with PPO (proximal policy optimization) · DPO (direct preference optimization) trains on the comparisons directly, with no reward model

The pipeline is the one from the Transfer Learning chapter, seen from the model's side. Step 3 is what changed most recently.

Base against instruct, measured

The instruct model reads the question inside a chat template (<|im_start|>, <|im_end|>: special tokens that open and close each turn):

<|im_start|>system
You are a helpful AI assistant ...<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
  • Base: The capital of France is Paris., then the question again, then the answer again — an FAQ page that never stops.
  • Instruct: The capital of France is Paris. and the end-of-turn token.
  • Entropy of the next token, averaged along the answer, five questions: 3.14 bits (base, like a pick among tokens) → 1.21 bits (instruct, ).
  • Post-training teaches a format and narrows the distribution.

Test-time compute: the second scaling axis

Until 2024, a better model meant a bigger model. Then: hold the model fixed and spend more compute at inference.

  • Write a long chain of reasoning.
  • Sample several attempts; check them; pick one.
  • At equal FLOPs, this can beat a model many times larger on problems the small one solves at least sometimes (Snell et al., 2025).

A model solves a problem 25% of the time. With 16 attempts, how often is some attempt right?

That frequency has a name: pass@k, the chance that at least one of k attempts is right (here k = 16).

Simulator · pass@k, voting and verifiers

The verifier sets the ceiling

Without a verifier, the bound is pass@k , the chance that some of the attempts is right, where is the chance that one attempt is right: .

With a verifier that accepts a right answer with probability and a wrong one with , the chance of picking a right one, as , is

  • , weak verifier (, ): 0.43 — however many samples you draw.
  • : 1. That is code (run the tests) and maths (check the answer) — where reasoning models improved first.

How the models learned to reason

GRPO (DeepSeekMath, 2024): sample a group of attempts, reward the ones that verify, normalize within the group — no value network.

ri: the verifier's score for attempt i · Âi: the advantage, how hard to push attempt i up or down. E.g. G = 4, r = (1, 0, 0, 1) → mean 0.5, std 0.5 → Â = (+1, −1, −1, +1).

  • R1-Zero: no reasoning traces supplied. Chains got longer because longer chains verified more often; backtracking and self-checking appeared.
  • The released R1 added a few worked examples first — for readability.

Reward hacking: optimizing against the checker, not against correctness. The verifier needs adversarial evaluation of its own.

Mixture of Experts: more parameters, same compute

  • Replace each MLP with many experts and a router that activates of them per token .
  • : expert (an MLP) · : the router's weights · : the probability it gives each expert · Top-: the indices of the largest.
  • The MLP holds about two-thirds of a block's parameters ( of , the Transformers chapter): that is where the leverage is.

DeepSeek-V3: 671B parameters, 37B active per token. What do you pay for in memory? And in compute?

Simulator · routing and load

Balancing is the hard part

  • Routing is a positive feedback loop: an expert that gets more tokens trains faster and is chosen more. Without a balancing term, the router collapses.
  • Fix: an auxiliary loss — or, in DeepSeek-V3, a per-expert bias adjusted until loads even out, with no second objective.
  • Fine-grained experts: top-8 of 256 gives combinations (the ways to choose 8 of 256); top-2 of 8 gives .
  • A shared expert, always on, holds the common knowledge so the routed ones can specialize.

You pay memory for total parameters and compute for active ones: a good trade in a datacentre, a bad one on a laptop.

The model gives you a distribution

Is top-k with a good rule for every prompt?

prompt (SmolLM2-135M) top token 90% nucleus top 40 cover
"The Eiffel Tower is located in the city of" Paris 0.881 3 tokens 97%
"My favourite food is" a 0.045 1,550 tokens 33%

top-k: keeps the k most likely · 90% nucleus (top-p, p = 0.9): the smallest set of tokens whose probability sums to 0.9 · min-p: the tokens above a fraction of the top one's probability

  • keeps 40 − 3 = 37 junk tokens in the first case and throws away most good answers in the second (the 40 cover only 33% of the mass).
  • top-p and min-p adapt to the shape of the distribution; top-k does not.

Simulator · temperature and truncation

Simulator · the next token, line by line

logits · row t = after token t (edit them)

Greedy loops, measured

Greedy decoding always picks the most likely token. What could go wrong over 100 tokens?

  • "The best way to learn deep learning is" → "…You need to understand the problem domain and the problem you are trying to solve." three times in a row
  • Repeated 4-grams (the fraction of 4-token sequences that had already appeared earlier in the text): 42% greedy against 11% sampled (temperature τ = 0.7, top-p = 0.9; mean of 5 samples).
  • Once a phrase appears, repeating it is the most likely continuation: the loop feeds itself.
  • Defaults: facts, code and tools at τ = 0–0.2; prose at 0.7–1.0 with top-p 0.9 or min-p 0.05.

Serving: decoding is memory-bound

  • Prefill reads the whole prompt in one parallel pass: arithmetic-bound.
  • Decode makes one token per pass, and each pass streams every weight from memory.

An 8B model in 16 bits on a GPU that reads 3.35 TB/s (an H100). Fastest possible single-stream decoding?

  • Weights: 8·10⁹ × 2 bytes = 16 GB, read in full for every token.
  • 16 GB / 3.35 TB/s ≈ 4.8 ms per token → at most 1 / 4.8 ms ≈ 200 tokens/s, whatever the FLOPs.
  • Batching shares one read of the weights among many requests.

The KV cache: how much it holds

Keys and values of past tokens never change: compute once, keep. Numbers per token:

2: one key and one value · L: layers · hkv: K/V heads (with GQA, fewer than the query heads) · dhead: the size of each head

  • SmolLM2-135M (; 9 query heads share K/V heads; ): numbers × 4 bytes (float32) = 46,080 bytes per token. A thousand tokens: 43.9 MiB.
  • Llama 3 8B (, , , bf16): bytes = 128 KiB per token → 16 GiB at a 128k context: the size of the weights themselves, per conversation.

Diagram · prefill, decode and the cache

Generating 6 tokens after the 4-token prompt Once upon a time, one forward pass per step. Each row is a step: its input is the prompt and the tokens sampled so far, and it ends by sampling the next token, which joins the next row's input. Without a cache every step recomputes the whole sequence, 39 positions in all; with a KV cache step 1 is the prefill of the 4 prompt tokens and every later step computes only the newest token, 9 positions in all, reading the earlier keys and values from the cache. The 6 passes stay sequential in both cases. INPUT OF EACH FORWARD PASS SAMPLED computed read from the cache not generated yet THIS EXAMPLE · m = 4, n = 6 forward passes6 positions computed cache after step 6 POSITIONS IN THE LAB new tokens nwithoutwith

The KV cache: what it buys

Prompt "Once upon a time": tokens. For new tokens, with the cache: ; without: — against . The generated tokens are identical.

new tokens tokens processed (with / without) time with cache without speed-up
50 53 / 1,425 (27×) 0.8 s 1.8 s 2.3×
100 103 / 5,350 (52×) 1.6 s 4.7 s 3.0×
200 203 / 20,700 (102×) 3.2 s 13.2 s 4.1×
400 403 / 81,400 (202×) 6.7 s 43.1 s 6.4×

Two more ways to read fewer bytes

Quantization

  • Weights in 8 or 4 bits instead of 16.
  • Fewer bytes per token → faster decoding: the 8B in 4 bits is 8·10⁹ × 0.5 byte = 4 GB, and the ceiling rises from ~200 to ~800 tokens/s.
  • Usually a small quality cost.

Speculative decoding

  • A small draft model proposes 5 tokens.
  • The big model checks all 5 in one pass — same memory traffic as checking one.
  • Keep them left to right while they pass; the first rejected one is replaced by the big model: the same output distribution (next slide).
  • 2–3× faster (Leviathan et al., 2023).

Diagram · one round of speculative decoding

One round of speculative decoding after the context The capital of France is. A small draft model proposes five tokens with probabilities q; the big model scores all of them in one forward pass and returns its own probabilities p. Each draft token is kept with probability min(1, p over q), left to right: Paris, the full stop and It are kept, has is rejected and replaced by is, drawn from the big model, and the last draft is discarded. One pass of the big model yields four tokens. CONTEXT The capital of France is draft model (small) · q target model (big) · p keep with prob. min(1, p/q) u, a uniform draw kept OUTPUT

Emergence, honestly

Score 3-digit addition by exact match across model sizes. Why might the curve look like a cliff?

  • Exact match is a threshold: all digits right or zero. Per-token log-probability of the right answer improves smoothly (Schaeffer et al., 2023).
  • What survives: improvement is smooth in the loss and abrupt in usefulness — 40% exact match may be unusable, 85% is a product.
  • If your metric is a threshold, your progress will look like a cliff.

Where the failures come from

failure where it comes from
Hallucination Accuracy-only grading rewards guessing over "I don't know" (Kalai et al., 2025)
Context is not memory Retrieval sags in the middle of long contexts (Liu et al., 2024)
Prompt injection Instructions in the data look like instructions; no robust fix — least privilege, human confirmation
Evaluation rot Public benchmarks leak into training data; prefer private, fresh and pairwise evaluations

From chat to agents

An agent runs a loop: read the goal, call a tool, read the result, decide, repeat, stop.

Each step of an agent is 95% reliable. How reliable is a task of ten steps?

  • 0.60. Errors compound.
  • Agents that work are built from checkable, re-runnable steps, not one long inference.
  • The context window is a budget: every tool result competes with the goal for room.

Lab · the distribution, measured

Colab notebook, ~2 h, CPU (a T4 only speeds up section 7). Each section ends in a checkpoint with a check value.

  1. tokenization and the price of Portuguese
  2. the next-token distribution, and one space
  3. temperature, top-k, top-p by hand; the greedy loop
  4. perplexity and bits per character
  1. the KV cache: size and speed-up
  2. base against instruct
  3. a scaling law on Pythia, along and
  4. decoding settings on the record: repetition against loss

open in Colab →

Takeaways

  1. Next-token prediction is compression; compressing well requires modelling the source.
  2. The loss is a power law in and ; deployed models are trained far past Chinchilla.
  3. Tokenization is fitted, not neutral: letters, digits, and Portuguese at 1.8× the tokens.
  4. A base model continues; post-training adds the format and narrows the distribution.
  5. pass@k is an upper bound; the verifier's false-positive rate sets the ceiling.
  6. MoE: memory for total parameters, compute for active ones; balancing is the hard part.
  7. Decoding is a separate algorithm, and a memory-bound one: KV cache, batching, quantization, speculation.

Outline: (1) pretraining — the objective, scaling, tokenization; (2) post-training — base against instruct; (3) test-time compute and verifiers; (4) MoE; (5) sampling and serving; (6) emergence, failures, agents. Closes with the Colab lab, whose numbers appear throughout.

Five steps: count the pairs (aa 4, overlapping inside aaa); aa → Z, 11 → 9 symbols (the merge does not overlap); recount, Za and ab tie at 2, the example takes ab → Y, 7 symbols; ZY → X = aaab, XdXac, 5 symbols; stop, every pair once. Hover a pair to see its occurrences. The vocabulary gained X = aaab: a compression algorithm fitted to a corpus, and therefore not neutral.

Answer: 1 − 0.75^16 ≈ 99% for a single problem; on the panel's benchmark, with uneven difficulty, about 80%. Then drag the verifier to weak and watch the green curve stop; majority voting plateaus near 40%.

Answer to the question: memory for all 671B, compute for 37B. Then switch routing to "no balancing": the router collapses and dozens of experts are dead weight that is still stored.

Temperature divides the logits: it never removes a token. Push it to 2 with no truncation and watch "banana" get mass; then add min-p 0.1 and see it disappear while the confident prompt barely moves.

The same logits, used twice. Lines 2–3 are training: −log of the token that came, mean, exp — perplexity 2.01, against 5 for a uniform guess; no sampling setting changes it. Lines 4–10 are the lab's warp on the last row and one draw: with τ = 0.7 and top-p 0.9 only "cat" and "ball" survive. Then the cases: greedy (always "cat", whatever u: the loop of the next slide), hot (u = 0.25 picks "chased", p = 0.057: the tail), top-k 2. The last line previews the KV cache slides: 403 against 81,400.

Start without a cache: every step recomputes the whole sequence, 4 + 5 + ... + 9 = 39 positions for 6 new tokens. Click "With a KV cache": step 1 is the prefill (the 4 prompt tokens in one parallel pass), steps 2-6 decode one token each, reading the cached keys and values: 9 positions. Still 6 sequential passes. The table is the lab's count at 50 to 400 new tokens. Hover a step for its numbers.

Seconds measured on a CPU in the lab; your numbers will differ, the shape will not.

Six steps: the big model alone makes 1 token per pass; the draft writes 5 with probabilities q; one big pass gives p at 6 positions; keep each with probability min(1, p/q): Paris (p >= q), "." (0.80, u = 0.41), "It" (0.90, u = 0.12); "has" 0.24 with u = 0.66 is rejected and replaced by "is", drawn from max(0, p - q); the last draft is discarded. 4 tokens for one big pass, and the output distribution is exactly the big model's. Numbers illustrative.