Generation

Step 7 of 7. In: the 12 logits of the last row (step 6). Out: one token, appended to the input; then the whole path runs again, until the model writes <|endoftext|>. At the end the ids go back to text: "the cat climbed the roof." ← Output · next: What changed since 2017 →

Choosing the next token: temperature, top-k, top-p

The model's work ends at the logits \(z\), one per vocabulary token. Turning them into text is a separate decision with its own knobs, and it is the part a user of a model actually touches:

• Temperature \(T\) divides the logits before the softmax. With logits \([2, 1, 0]\), \(T = 1\) gives \([0.67, 0.24, 0.09]\) and \(T = 0.5\) gives \([0.87, 0.12, 0.02]\): a lower temperature widens the gaps. \(T \to 0\) is greedy decoding (always the top token); \(T \to \infty\) is the uniform distribution.
• Top-k¹ keeps the \(k\) most likely tokens, renormalizes and samples among them.
• Top-p² (nucleus) keeps the smallest set whose probabilities add up to \(p\). Unlike top-k, its size follows the model's confidence: on GPT-2, after "…the 44th President of the United" top-p 0.9 keeps 1 token (States, 0.998); after "Once upon a time, there was a" it keeps 3651.

Greedy decoding is not the safe default it looks like: it tends to loop, repeating a phrase that was likely once and is therefore likely again². At the other end, a high temperature without a filter hands the long tail of 50,257 tokens enough mass to win — GPT-2 at \(T = 2\) produces noise. Production settings sit in between (a temperature below 1 and a top-p or top-k cut), and the lab measures all of them on the same prompt.

On the running example

The logits of the last row, \(z\), are on the output page. Dividing them by \(T\) before the softmax:

• Greedy (\(T \to 0\)) writes ␣roof, every time.
• Top-k = 2 keeps ␣roof and ␣wall and divides by what they hold: \(0.395 / (0.395 + 0.141) = 0.737\) and \(0.263\).
• Top-p = 0.9 at \(T = 1\): the running sums are 0.395, 0.537, 0.626, 0.712, 0.774, 0.832, 0.881, 0.931, so the smallest set that reaches 0.9 has 8 of the 12 tokens — the toy's distribution is flat, and even ␣the and ed stay in.
• \(T = 0.7\) with top-p 0.9, a common production setting (the LLMs lab samples with it): 6 tokens stay, renormalized to ␣roof 0.626, ␣wall 0.144, ␣sofa 0.074, ␣floor 0.071, ␣cat 0.044, ␣dog 0.040. To draw one, take a uniform random number \(u\) and walk the running sum (0.626, 0.770, 0.845, 0.916, 0.960, 1): \(u = 0.5\) falls in ␣roof, \(u = 0.8\) in ␣sofa — "the cat climbed the sofa", a perfectly good sentence the model also considered.

Append, and run again

Say the draw gave ␣roof, id 6. The input becomes [0, 2, 4, 5, 1, 6] and the whole path runs again, now with six tokens. The causal mask makes rows 1 to 5 come out exactly as before — the old tokens never look at the new one — so a model does not recompute them: it keeps their keys and values (the KV cache) and computes only the new row:

• \(x_6 = E[6] + P_6 = [0, -0.5, 1, -0.5] + [0.5, 0, 0, -0.5] = [0.5, -0.5, 1, -1]\), and \(u_6 = [0.632, -0.632, 1.265, -1.265]\);
• its query, \(q_6 = [u_4, u_2 \mid u_4, u_1] = [-1.265, -0.632 \mid -1.265, 0.632]\), against the six keys (five from the cache, one new): head 1 puts 0.40 on ␣roof itself and 0.28 on ␣cat, head 2 spreads over the, ed and ␣the;
• through the block and the output layer, the new last row predicts . with 0.538 (then ␣climb 0.168).

Append . (id 10) and run once more: <|endoftext|> comes first, with 0.401. That is the model's way of saying the text is finished, and generation stops there — or at a maximum length, whichever comes first.

Each step costs one row of work instead of a whole pass: in the toy the cache holds 8 numbers per token (2 for K and V × 1 layer × 2 heads × 2), in GPT-2 small 18,432.

Back to text

Detokenizing is a lookup and a concatenation: each id becomes its piece, the pieces are glued, the special token is dropped.

[0, 2, 4, 5, 1, 6, 10] → the ␣cat ␣climb ed ␣the ␣roof . → "the cat climbed the roof."

In GPT-2 the pieces are bytes, and a character outside ASCII can be split between two tokens ("ã" is two bytes, and a token may hold only the first); the decoder joins the bytes first and turns them into characters only when a character is complete — which is why a chat interface sometimes holds a token back before showing it.

The panel runs the whole step, line by line: temperature, top-k, top-p, the draw, the append and the next run, with the new row's prediction. Try the cases, then move \(u\) and watch which token wins.

← Output · next: What changed since 2017 →