23. Autoregressive Generation

Diffusion models have dominated image generation since 2021. There is a second way to do it, older than diffusion and very much alive: turn the image into a sequence of discrete tokens and generate it the way a language model generates text, one token at a time, each conditioned on the ones before.

It is not the way to get the best image per unit of compute — the previous chapter is. It came back for a different reason: it is the one formulation in which a picture and a sentence are the same kind of object, so one model, one sequence and one loss can read, reason and draw. OpenAI describes GPT-4o's image generation (March 2025) as "an autoregressive model natively embedded within ChatGPT"⁷; open multimodal models such as Chameleon⁴, Emu3⁹ and Janus-Pro¹⁰ generate images as tokens; and outside images, the same recipe — a learned tokenizer, then next-token prediction — is the standard design for speech and audio and a common one for robot actions.

This chapter builds it from the bottom: the tokenizer, the autoregressive loop and what it costs, the three ways to make it faster or better (parallel rounds, coarse-to-fine, continuous tokens), and the designs that put image generation inside a language model.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. The simulators of this page run inside the slides. Open the slides in a new tab

The Problem: How to Tokenize an Image?

Text is naturally discrete (words, subwords). An image is a grid of pixels, each three intensities in \([0, 255]\): discrete in principle, but with \(256^3 \approx 16.8\) million colours per pixel and 65 536 pixels in a \(256 \times 256\) image, far too many and too fine to predict one by one. To use autoregressive generation, we need a visual vocabulary, and a decoder that turns a grid of vocabulary entries back into pixels.

The classic solution is VQ-GAN¹, a VQ-VAE (chapter 17) trained with perceptual and adversarial losses so that its reconstructions are sharp. A CNN encoder turns the image into a grid of vectors; each vector is replaced by the nearest of \(K\) learned codebook vectors; the image becomes a grid of integer indices; a CNN decoder maps the codebook vectors back to pixels.

The arithmetic is worth doing once. A \(256 \times 256\) image with a downsampling factor of 16 becomes a \(16 \times 16\) grid: \(N = 256\) tokens. With \(K = 16\,384\) codes, each token is \(\log_2 K = 14\) bits, so the whole image is described by at most \(256 \times 14 = 3\,584\) bits — about 450 bytes for an image that takes 196 608 bytes raw. Everything else, the decoder invents. That ceiling is the fidelity limit of every model built on top of the tokenizer.

Training a quantizer, and how it fails

The nearest-neighbour lookup has no gradient, so VQ training needs three pieces. With \(x\) the image, \(z_e\) the encoder output, \(e\) the selected code, \(D\) the decoder, \(\mathrm{sg}[\cdot]\) the stop-gradient (the identity in the forward pass, a constant in the backward pass) and \(\beta\) the weight of the commitment term (0.25 in the lab):

The last expression is the straight-through estimator: in the forward pass \(z_q = e\), in the backward pass the decoder's gradient is copied to the encoder as if quantization were the identity. Step through one training step: the forward pass, the wall the gradient meets at the argmin, the copy that gets past it, and which codes the two other terms reach.

The weak point is the middle term. Only the code that was selected gets a gradient. A code that starts far from every encoder output is never selected, never moves and is never used: a dead code. In practice large codebooks are mostly dead. VQGAN-LC reports that the original VQGAN, with 16 384 codes, used 3.4% of them on ImageNet, and a variant with factorized codes 11.2%¹¹. Train one below and watch it happen.

With a wide random initialisation, a 32-code book ends at 21 used codes and a 64-code book at 28: the bigger the book, the larger the dead fraction. Initialising the codes from data, or restarting dead codes on random encoder outputs, brings usage to (nearly) 100%. The field has used several fixes:

• Better VQ. Initialise codes from data, restart dead codes, keep the codes in a low-dimensional, normalised space. LlamaGen³ uses 8-dimensional codes and reports 97% usage of 16 384 entries.
• No learned codebook at all. FSQ¹² (finite scalar quantization) projects the latent to a handful of dimensions, bounds each with \(\tanh\) and rounds it to a few levels: levels \((8, 6, 5)\) give \(8 \cdot 6 \cdot 5 = 240\) codes, every one of them a point of a fixed grid. There is no codebook loss, no commitment loss, and nothing that can die. LFQ¹³ (MAGVIT-v2) is the binary case — each of 18 dimensions is just its sign — which gives a \(2^{18} = 262\,144\)-word vocabulary.
• No quantizer at all. Keep continuous tokens and change what predicts them — the MAR section below.

The lab trains a VQ tokenizer and an FSQ tokenizer of the same size on MNIST and measures exactly this: the VQ codebook used 82 of its 256 codes (32%), the FSQ grid all 240 of its codes (100%), and FSQ reconstructed slightly better (MSE 0.0034 against 0.0047) with 3 numbers per position instead of 16.

Autoregressive Token Generation

With a trained tokenizer, any image is a sequence of \(N\) integer indices — in raster order, row by row. We generate this sequence exactly like an LLM generates text:

where \(c\) is the condition (a class, or a text prompt). Training is ordinary next-token cross-entropy with a causal mask, all positions in parallel, with the true previous tokens as inputs (teacher forcing). Sampling draws one token, appends it and runs the model again.

Three things carry over directly from language models:

• Sampling knobs. Temperature and top-\(k\) trade diversity for fidelity, exactly as for text.
• Classifier-free guidance works here too. Run the model with and without the condition and extrapolate the logits (the scores before the softmax), \(\ell = \ell_\varnothing + w\,(\ell_c - \ell_\varnothing)\), with \(\ell_c\) the logits given \(c\), \(\ell_\varnothing\) the logits without it and \(w > 1\) the guidance scale. LlamaGen and VAR both rely on it for their best numbers.
• The cost model is the LLM's. One forward pass per token, and the passes are sequential: token \(i\) needs the sampled value of token \(i-1\). The KV cache keeps each pass to one new token, but it cannot remove the sequence. A \(16 \times 16\) grid is 256 sequential passes; 1 024 for Chameleon's 512 × 512 images; 4 096 for a \(64 \times 64\) grid.

The panel counts what one sample of 8 tokens costs, pass by pass, without and with the cache:

The length of that sequence is the reason this works on tokens and not on pixels. ImageGPT¹⁶ (2020) did run a GPT on raw pixels, at \(32 \times 32\) and \(64 \times 64\) with a reduced 9-bit palette: 1 024 to 4 096 steps for a thumbnail. A tokenizer that downsamples by 16 divides the sequence by 256.

MaskGIT: Parallel Generation via Masking

Raster order is slow and also arbitrary: token 200 is not "after" token 199 in any sense the image cares about. MaskGIT² drops the order and decodes in a few parallel rounds, with a bidirectional Transformer trained like BERT:

1. Start with all tokens masked [MASK].
2. At each round, predict all masked tokens at once (bidirectional attention).
3. Sample them, and keep only the most confident ones; re-mask the rest.
4. Repeat. The number of tokens still masked after round \(t\) of \(T\) follows a cosine schedule, \(\lfloor N \cos(\tfrac{\pi}{2}\,t/T) \rfloor\): few tokens committed early, when the model knows least, many at the end.

The paper uses \(T = 8\) rounds for \(16 \times 16 = 256\) tokens and 12 rounds for \(32 \times 32 = 1\,024\), and reports decoding 30–64× faster than autoregressive decoding.

Why not a single round? Because in one pass every masked token is sampled from its own distribution given the visible tokens, independently of the others. With nothing visible, two neighbouring tokens can each be likely and still be wrong together. Each round conditions on what the previous rounds committed, which is what makes the joint sample coherent. The lab measures it on MNIST: with one round, 69% of the generated digits are recognised as the digit asked for and their Fréchet distance to real digits is 138; with eight rounds, 90% and 69. Eight rounds are 6× fewer forward passes than the 49 of the autoregressive model, which scores 84% and a Fréchet distance of 31: MaskGIT's digits are cleaner but more alike, because keeping only the most confident tokens favours typical ones.

Muse¹⁷ (2023) took the same decoder to text-to-image at 512 × 512 and reported it several times faster than the diffusion models of its day. Calling MaskGIT a discrete diffusion model — the forward process masks instead of adding noise — is fair, and it is one of the places where the families blur.

Next-scale prediction: VAR

VAR⁶ keeps the autoregression but changes what one step produces: not a token, but a whole token map at the next resolution. For a \(256 \times 256\) image it generates ten maps, of side 1, 2, 3, 4, 5, 6, 8, 10, 13 and 16. Each map is predicted in one forward pass, conditioned on all the coarser ones (they are residuals; the final map is their upsampled sum), with tokens inside a map produced in parallel. Step through the ten passes:

That is an order the data actually has — the coarse layout is decided before the detail — and it needs \(O(\log n)\) steps instead of \(O(n^2)\) for an \(n \times n\) grid: 10 forward passes instead of 256. VAR reported ImageNet 256 × 256 FID (the Fréchet distance between generated and real images in the features of an Inception network; lower is better) of 1.73 against 2.27 for DiT-XL/2 (and against 18.65 for the raster-order baseline it started from, with about 20× faster inference), and wrote that it "for the first time, makes GPT-like AR models surpass diffusion transformers"; it received the NeurIPS 2024 best paper award. Infinity¹⁸ (2025) took next-scale prediction to text-to-image at 1 024 × 1 024 with a bitwise tokenizer, generating an image in about 0.8 s.

Continuous tokens: MAR drops the quantizer

The codebook is where autoregressive image models lose fidelity: everything between codebook entries is unrepresentable. MAR⁵ removes it. Tokens stay continuous (the latents of a VAE, as in Stable Diffusion), and the categorical softmax is replaced by a small diffusion model per token: the Transformer outputs a conditioning vector \(z_i\) for position \(i\), and a small MLP (21 M parameters by default, about 5% of MAR-L, the 400 M-parameter model), trained with the usual denoising loss, samples the continuous token \(x_i\) from \(p(x_i \mid z_i)\) in 100 denoising steps.

Here \(x_{i,t}\) is \(x_i\) with noise \(\epsilon\) added up to diffusion time \(t\) (in this formula \(t\) is a noise level, not a token), and \(\epsilon_\theta\), the MLP, tries to recover \(\epsilon\). These letters are MAR's: \(x_i\) is one continuous token, not the image \(x\) of the tokenizer, and \(z_i\) is the Transformer's output, not the encoder output \(z_e\).

MAR also generates in a random order and in parallel groups, MaskGIT-style, with bidirectional attention, and reached FID 1.55 on ImageNet 256 × 256. Fluid¹⁹ scaled the recipe to 10.5 B parameters for text-to-image and found the two halves of the lesson separately: continuous tokens beat discrete ones, and random order beats raster order. NextStep-1²⁰ (2025) is a 14 B autoregressive text-to-image model built the same way, with a flow-matching head per token.

The axis is not what people think it is

"Autoregressive versus diffusion" sounds like a choice of model family. It is really two independent choices:

Once you see the two axes, the taxonomy stops being tribal. The same idea now runs in the other direction: diffusion language models generate text by iterative unmasking over the whole sequence instead of left to right — LLaDA²¹ (8 B, 2025) is competitive with an autoregressive LLaMA 3 8B; Google's experimental Gemini Diffusion (May 2025) and Inception's commercial Mercury are fast examples. They trade the KV cache for parallel decoding.

Any-to-Any: images inside a language model

The step that brought autoregression back is removing the distinction between text and image tokens. Chameleon⁴ (Meta, 2024) interleaves them in one sequence, from the first pretraining step: a 512 × 512 image is 1 024 tokens from an 8 192-entry codebook, added to the text vocabulary, and one Transformer is trained with one next-token loss.

[TEXT: "a photo of"] [IMG_TOK_3742] [IMG_TOK_891] ... [IMG_TOK_5531] [TEXT: "a cat"]

That is one design. Since 2024 there have been at least four, and they differ on exactly the two axes above: what an image position holds and how it is predicted. Step through them:

Two corrections to things that are often repeated. First, the architecture of the frontier closed systems is not public: "GPT-4o is an autoregressive model with a diffusion decoder" is a plausible reading of a whiteboard image in OpenAI's launch post, not a documented fact, and nothing official describes how Gemini draws. Second, the open models that lead in image quality are mostly not the discrete-token ones: Janus-Pro and Emu3 show that a single next-token model can draw, but the designs that put a diffusion or flow loss inside the model (Transfusion, BAGEL, HunyuanImage 3.0) or behind it (MetaQuery, BLIP3-o) are where most 2025 effort went.

AR vs. Diffusion in 2026

REPA and RAE are diffusion transformers that lean on a pretrained image encoder (DINOv2), one by aligning the model's features with it, the other by diffusing in its latent space (chapter 22). On class-conditional ImageNet the two families have traded the lead since 2024 and sit within a few tenths of FID of each other; neither is "better" in general. For standalone text-to-image the strongest open models are diffusion or flow transformers (chapter 22). Where autoregression is the natural choice is the reason it came back: generation that has to live in the same context as text — conversations that edit an image, interleaved documents, reasoning before drawing.

Where this is used today

The recipe — learn a tokenizer, then predict the next token — is not only an image technique. It is the standard design in several places, and every one of them now has a diffusion or flow variant competing with it, which is the two-axes picture again:

Where each one is used

Implementation: tokenizer + autoregressive Transformer

The lab builds all of this on MNIST. The core is small:

import torch
import torch.nn as nn
import torch.nn.functional as F

# 1. Vector quantizer, with the two losses that train the codebook and the encoder
class VectorQuantizer(nn.Module):
    def __init__(self, n_codes, d_code, beta=0.25):
        super().__init__()
        self.codebook = nn.Embedding(n_codes, d_code)
        self.beta = beta

    def forward(self, z):                          # z: (B, H, W, d_code), encoder output
        flat = z.reshape(-1, z.shape[-1])
        idx = torch.cdist(flat, self.codebook.weight).argmin(-1)   # nearest code
        q = self.codebook(idx).view_as(z)
        loss = F.mse_loss(q, z.detach()) + self.beta * F.mse_loss(z, q.detach())
        q = z + (q - z).detach()                   # straight-through estimator
        return q, idx.view(z.shape[:-1]), loss

# 1'. FSQ: no codebook, no loss. levels=(8, 6, 5) gives 240 codes on a fixed grid
def fsq(z, levels=(8, 6, 5)):                      # z: (..., len(levels))
    L = torch.tensor(levels, dtype=z.dtype, device=z.device)
    half = (L - 1) / 2
    offset = (L % 2 == 0) * 0.5                    # even levels: shift the grid by half a step
    z = torch.tanh(z + torch.atanh(offset / half)) * half - offset
    return z + (z.round() - z).detach()            # round, with a straight-through gradient

# 2. Autoregressive generation with a GPT-like model over the token ids
class ImageGPT(nn.Module):
    def __init__(self, n_codes, n_classes, seq_len, d_model=192, n_heads=4, n_layers=4):
        super().__init__()
        self.tok_emb = nn.Embedding(n_codes, d_model)
        self.cls_emb = nn.Embedding(n_classes, d_model)       # the condition is token 0
        self.pos_emb = nn.Parameter(torch.zeros(1, seq_len + 1, d_model))
        layer = nn.TransformerEncoderLayer(d_model, n_heads, 4 * d_model,
                                           batch_first=True, norm_first=True)
        self.blocks = nn.TransformerEncoder(layer, n_layers)
        self.head = nn.Linear(d_model, n_codes)

    def forward(self, y, tokens):                  # tokens: (B, T), T <= seq_len
        x = torch.cat([self.cls_emb(y)[:, None], self.tok_emb(tokens)], 1)
        T = x.shape[1]
        x = x + self.pos_emb[:, :T]
        causal = nn.Transformer.generate_square_subsequent_mask(T, device=x.device)
        return self.head(self.blocks(x, mask=causal, is_causal=True))   # (B, T, n_codes)

    @torch.no_grad()
    def generate(self, y, n_tokens, temperature=1.0, top_k=None):
        tokens = torch.zeros(len(y), 0, dtype=torch.long, device=y.device)
        for _ in range(n_tokens):                  # one forward pass per token
            logits = self(y, tokens)[:, -1] / temperature
            if top_k:
                logits[logits < logits.topk(top_k)[0][:, -1:]] = -float('inf')
            nxt = torch.multinomial(logits.softmax(-1), 1)
            tokens = torch.cat([tokens, nxt], 1)
        return tokens                              # decode with the tokenizer's decoder

Training the Transformer is F.cross_entropy(model(y, s[:, :-1]).flatten(0, 1), s.flatten()): the output at position \(i\) (which has seen the condition and \(t_1, \ldots, t_{i-1}\)) is scored against \(t_i\).

The same code, run on a \(2 \times 2\) image with 2-D codes and \(K = 4\) so that every number can be checked by hand: the quantizer (nearest code, the two auxiliary losses, straight-through), then one step of the autoregressive model, with its training loss and a draw. In the default case code 2 wins no position and gets a zero gradient, and the loss for \(t_4\) is 0.515, against \(\log 4 = 1.386\) for a blind guess. Change a number, the temperature or the case and watch which line moves:

Lab: tokens, measured

The chapter's claims are measured in a Colab notebook on MNIST (70 000 handwritten digits, 28 × 28): a tokenizer, an autoregressive Transformer and a MaskGIT, all trained from scratch. It needs a GPU: in Colab, Runtime → Change runtime type → T4 GPU (the free tier is enough; the reference run of the whole notebook took 10.5 minutes on an Apple laptop GPU). Eight sections, each ending in a checkpoint with a check value:

1. a judge for generated digits: a small CNN classifier and a Fréchet distance in its feature space;
2. a VQ tokenizer: reconstruction, and how many of its 256 codes it actually uses;
3. an FSQ tokenizer of the same size: the same measurements, with no codebook;
4. the image as a sequence: 49 tokens, the bits they can carry, and their frequency;
5. an autoregressive Transformer: bits per image, and which positions are hard;
6. sampling it: 49 sequential passes, digit accuracy, Fréchet distance and time;
7. a MaskGIT on the same tokens: quality against the number of rounds, and the speed-up;
8. a table to fill in: temperature, tokenizer and decoder against quality and cost.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. Autoregressive image generation needs a tokenizer: the image becomes \(N\) integers from a vocabulary of \(K\), at most \(N \log_2 K\) bits. The tokenizer is both the enabling trick and the fidelity ceiling.
2. A VQ codebook collapses: unused codes get no gradient and die. Initialisation from data, dead code restarts and low-dimensional codes help; FSQ and LFQ replace the codebook with a fixed grid and nothing can die.
3. Cost is one sequential forward pass per token, with or without a KV cache. That is why it runs on latent tokens (256 – 4 096), not pixels.
4. Raster order is arbitrary. MaskGIT decodes in 8–12 parallel rounds; VAR predicts the next scale in ~10 steps. Parallel rounds need several rounds because tokens sampled in the same pass are independent of each other.
5. MAR shows the quantizer is optional: continuous tokens with a small diffusion head per token.
6. The real axes are generation order and per-element distribution, not "autoregressive versus diffusion". On ImageNet the two families are at rough parity; for text-to-image quality, diffusion and flow transformers lead.
7. The reason the approach came back is unification. Multimodal models now generate images in four ways — discrete next token, discrete masked, a diffusion loss inside the LLM, or an LLM driving a separate diffusion decoder — and the frontier closed systems do not say which.
8. Beyond images, tokenizer + next-token prediction is the standard design for speech and audio and a common one for video world models and robot actions.