Skip to content

Scaling

The path from text to the next token in seven steps: tokens, embeddings, query/key/value, mask and heads, the block, the output and generation. ATTENTION TRANSFORMERS Tokenstext → ids Embeddingsids → X Q · K · Vone head Masktwo heads BlockLN · MLP Outputlogits → p Generationpick text → → next token

Beyond the running example. The toy has 256 weights and was never trained; GPT-2 small has 124 million and saw about 40 GB of text. This page is about what happens when the same design is made bigger — and bigger in what. ← What changed since 2017 · Overview

Scaling: three different questions, three different answers

The Transformer's real property is not any single design choice; it is that loss falls predictably as you add parameters, data and compute, over many orders of magnitude, with no sign of a wall1. That reliability is what justified spending a billion dollars on a training run.

But "scale it up" hides a question — scale what? The budget is compute \(C\), measured in FLOPs, and it is split between the model size \(N\) (parameters) and the amount of data \(D\) (training tokens):

\[ C \approx 6\,N D \]

The 6 comes from counting: each parameter does about 2 operations per token on the way forward (one multiply, one add) and 4 on the way back. With \(C\) fixed, doubling the model forces you to halve the data. "Scale what?" is the question of how to split \(C\) between \(N\) and \(D\) — and the answer changed twice.

2020, Kaplan et al.1: for a fixed compute budget, make the model bigger. GPT-3 followed the advice: 175B parameters on 300B tokens, under 2 tokens per parameter.

2022, Chinchilla2: that was wrong, and Hoffmann et al. traced most of the error to the learning-rate schedule of the original experiments, which was not matched to each run's length. Redone properly, the optimum is to scale parameters and data together — roughly 20 tokens per parameter. Chinchilla, at 70B parameters and 1.4T tokens, beat the 280B Gopher trained on the same compute. Overnight, everybody's models got smaller and their datasets got bigger.

The panel does not use the numbers the paper printed

Chinchilla reaches its conclusion three ways, and the third — the parametric fit the panel plots — was published with coefficients that imply about 70 tokens per parameter, not 20. They disagree with the paper's own other two methods and with the model it actually trained. A 2024 replication traced this to a badly converged optimizer and refitted it4; the panel uses the refitted coefficients, which land back in the low tens. Expect the ratio to drift a little with the budget anyway, because the two exponents are not equal — "20 tokens per parameter" is a rule of thumb, not a constant of nature.

Now: nobody trains at the Chinchilla optimum either, and the panel shows why. Tick price inference too. Chinchilla minimizes the cost of one training run; a deployed model is paid for by every token it ever serves, and serving cost scales with \(N\), not with \(D\). So the right move is to overtrain a smaller model — Llama-3-8B saw 15T tokens, about 1900 per parameter, nearly 100× past Chinchilla. It is compute-inefficient to train and much cheaper to own.

Put all three on the same ruler with \(C \approx 6ND\):

Model \(N\) \(D\) tokens per parameter \(C\) (FLOPs)
GPT-3 (2020) 175B 0.3T 1.7 \(3.2 \times 10^{23}\)
Chinchilla (2022) 70B 1.4T 20 \(5.9 \times 10^{23}\)
Llama 3 8B (2024) 8B 15T ~1900 \(7.2 \times 10^{23}\)

Chinchilla and Llama 3 8B cost almost the same to train. The second is 9× smaller, so every token it generates in deployment costs about 9× less — and that is the cost paid billions of times.

Two more corrections belong here:

  • Data is finite. High-quality text is a limited resource, and repeating it has sharply diminishing returns after a few epochs3. This is a real constraint on the \(D\) axis and a large part of why synthetic data and curation are now serious research areas.
  • Training compute is no longer the only axis. A model can also be given more compute at inference — sampling many chains of thought, searching, verifying. That curve scales too, and it is the subject of chapter 15.

← What changed since 2017 · Overview

  1. Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. The power laws — and the parameter-heavy conclusion that Chinchilla overturned. ↩↩

  2. Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models — NeurIPS. Chinchilla; the form of the parametric loss plotted in the panel above comes from here (its coefficients, from the replication below). ↩

  3. Muennighoff, N., et al. (2023). Scaling Data-Constrained Language Models — NeurIPS. What repeating your data actually buys, and where it stops buying anything. ↩

  4. Besiroglu, T., Erdil, E., Barnett, M., & You, J. (2024). Chinchilla Scaling: A replication attempt. Reconstructs the data behind Hoffmann et al.'s third approach, shows the published coefficients do not fit it, and refits them. ↩