Artificial Neural Networks and Deep Learning · Insper

Autoregressive Image Generation

An image as a sentence of tokens

Two ways to make an image

Diffusion / flow

  • all positions, every step
  • continuous latents
  • the best image per unit of compute (previous chapter)

Autoregressive over tokens

  • image → grid of integers
  • one token at a time, like an LLM writes text
  • older than diffusion

Diffusion makes better pictures for the money. Why did OpenAI call GPT-4o's image generation "an autoregressive model natively embedded within ChatGPT"?

Where this is used today

Domain Learned tokens + next-token prediction
Images in multimodal LLMs Chameleon, Emu3 / Emu3.5, Janus-Pro; GPT-4o says "autoregressive"
Speech and audio codec tokens (SoundStream, EnCodec, Mimi) + an LM: VALL-E, MusicGen, Moshi
Video and world models VideoPoet; NVIDIA Cosmos ships AR and diffusion; Genie 3 is frame-by-frame
Robot policies RT-2, OpenVLA: 256 bins per action dimension; FAST: DCT + BPE

Every row also has a diffusion or flow competitor. Keep that in mind: it is the two axes at the end.

Step 1: a visual vocabulary

  • VQ-GAN: a VQ-VAE trained with perceptual and adversarial losses
  • encoder → grid of vectors → nearest of codebook vectors → grid of integers
  • decoder: codebook vectors → pixels

A 256 × 256 image, downsampling 16, . How many tokens, and how many bits can they carry?

downsampling 16: each token summarizes a 16 × 16 block of pixels · K = number of codebook vectors

  • per side: tokens, bits each
  • at most 3 584 bits ≈ 450 bytes, for bytes of pixels
  • everything else, the decoder invents: the fidelity ceiling

Simulator · tokenizing an image

Training a quantizer

x = image · ze = encoder output · e = the code nearest to ze · D = decoder · sg[·] = stop-gradient: a constant on the way back · β = commitment weight (0.25 in the lab)

  • argmin has no gradient: the straight-through estimator copies the decoder's gradient to the encoder ( forward, backward)

Which codes does the middle term move?

  • Only the ones that were chosen. A code that is never nearest never moves: a dead code.

Diagram · one quantizer step, forward and back

One training step of a VQ tokenizer. Forward: the image x goes through the encoder to a vector z_e, which is replaced by the nearest of K codebook vectors (the code e); its number k is the token; the decoder D rebuilds the image from it. Backward: the gradient of the reconstruction stops at the argmin, the straight-through estimator copies it from z_q to z_e, the codebook term moves only the chosen code and the commitment term pulls z_e toward it. The codes that were not chosen get no gradient. ONE TRAINING STEP straight-through: the gradient is copied ximage EncoderCNN zed numbers Nearest codeindex k = token zq = edecoder input Decoder DCNN D(zq)rebuilt CODEBOOKK VECTORS,NUMBERED 0 … K − 1 0 1 2 3 4 6 … ek = 5 0000000← gradient 0: these codes were not chosen β ‖ze − sg[e]‖²commitment: moves ze ‖sg[ze] − e‖²moves the code e ‖x − D(zq)‖²reconstruction values, forward (upper arrows) gradient, backward (lower arrows) sg[·] = stop-gradient: a constant on the way back β = commitment weight, 0.25 in the lab

Simulator · a codebook in 2-D

Fixing collapse

  • Better VQ: initialise from data, restart dead codes, low-dimensional normalised codes
    (LlamaGen: 8-d codes, 97% of 16 384 used)
  • FSQ: 3–6 numbers, each -bounded and rounded to a few levels; → codes on a fixed grid
  • LFQ (MAGVIT-v2): 18 signs → codes
  • No quantizer: continuous tokens (MAR, later)

FSQ has no codebook, no codebook loss, no commitment loss: nothing can die.

Lab · VQ against FSQ on MNIST

28 × 28 digits, downsampling 4 → grid , 49 tokens

tokenizer codes used on the test set perplexity reconstruction MSE
VQ, 16-d codes 256 82 of 256 (32%) 44 0.0047
FSQ, levels (8, 6, 5) 240 of 240 (100%) 77 0.0034

used = codes that appear among the 10 000 × 49 = 490 000 test tokens · perplexity = 2entropy of the code usage: how many equally used codes the usage is worth

  • same reconstruction quality by eye, with 3 numbers per position (FSQ) instead of 16 (VQ)
  • effective vocabulary 1.8× larger:

Step 2: generate the tokens like text

ti = the i-th token (an integer from the codebook) · N = number of tokens · c = the condition (a class or a prompt)

Training uses a causal mask and runs all positions at once. Can sampling do the same?

  • No: token needs the sampled value of token .
  • Training: parallel (teacher forcing). Sampling: sequential forward passes.
  • Temperature, top- and classifier-free guidance on the logits carry over from LLMs.

teacher forcing: the inputs are the true previous tokens · classifier-free guidance: ℓ = ℓ∅ + w (ℓc − ℓ∅), the logits with c pushed away from those without it, w > 1

Simulator · tokenize, then predict, line by line

z · encoder output
codebook (K = 4)
model logits for t4

Simulator · raster order

What it costs

image grid = side / f tokens = sequential passes
256², f = 16 256/16 = 16 → 16 × 16 256
512², f = 16 (Chameleon) 512/16 = 32 → 32 × 32 1 024
1024², f = 16 1024/16 = 64 → 64 × 64 4 096
ImageGPT: raw pixels 64², f = 1 64 × 64 4 096, for a thumbnail

f = the tokenizer's downsampling factor: each token summarizes an f × f block of pixels

  • the KV cache makes each pass process one new token; it does not remove the sequence
  • the tokenizer is what makes it affordable: downsampling each side by 16 divides the sequence by

Diagram · what the KV cache saves, and what it does not

Sampling 8 tokens after a condition c, one forward pass per token. Each row is a pass; its input is c and the tokens sampled so far, and it ends by sampling the next token, which becomes part of the next row's input. Without a cache every pass recomputes the whole prefix, 36 positions in all; with a KV cache each pass computes only the newest position, 8 in all, but still attends to every earlier key, and the 8 passes stay sequential. INPUT OF EACH FORWARD PASS SAMPLED computed read from the cache not sampled yet THIS SAMPLE · 8 TOKENS sequential passes8 positions computed keys read by attention THE SAME COUNT AT SCALE tokens Npassescomputed

The likelihood is exact, but of the tokens

Lab, MNIST, 49 FSQ tokens per digit:

model bits per token × 49 tokens = bits per image
ceiling: every code equally likely 387.4
token frequencies only, no context 6.26 (the entropy) 307
autoregressive Transformer (862 832 params) 3.07 150.5

bits = −log₂ p(token): what the model pays, on average, per token of the test digits

  • the stroke region costs up to 5.17 bits per token; the last corner, 0.31
  • exact by the chain rule, but of the tokens: many images share them

MaskGIT: a few parallel rounds

  1. start with every token [MASK]
  2. a bidirectional Transformer predicts all masked tokens at once
  3. sample them, keep the most confident, re-mask the rest
  4. after round of , are still masked — with , : after round 1, only 5 kept

8 rounds for 256 tokens, 12 for 1 024. The paper reports 30–64× faster decoding than autoregression.

Simulator · the cosine schedule

Why not one round?

One pass predicts every token. Why not sample all 49 and stop?

  • Tokens sampled in the same pass are independent of each other, given what is visible.
  • Lab, 1 000 class-conditional digits. Real digits: accuracy 98.1%, Fréchet distance 25.2 (the floor).
decoder passes accuracy Fréchet distance ↓
autoregressive 49 0.840 31.2
MaskGIT, 1 round 1 0.689 138.2
MaskGIT, 4 rounds 4 0.882 64.3
MaskGIT, 8 rounds 8 0.899 69.3

accuracy = fraction the judge (a small CNN) classifies as the requested digit · Fréchet distance between the judge's features for generated and real digits: it penalizes bad samples and samples too alike

VAR: predict the next scale

  • not one token per step: one whole token map, at the next resolution
  • 256²: maps of side 1, 2, 3, 4, 5, 6, 8, 10, 13, 16 → ten sides, 10 passes instead of
  • each map conditioned on all coarser ones; tokens inside a map in parallel
  • an order the data has: layout before detail

ImageNet 256²: FID 1.73 against 2.27 for DiT-XL/2 (18.65 for its raster baseline, ~20× slower). NeurIPS 2024 best paper. Infinity (2025): the same idea, text-to-image at 1 024².

FID = Fréchet distance between generated and real images in the features of an Inception network: lower is better

Diagram · ten maps, ten passes

Next-scale prediction in VAR for a 256 by 256 image: ten token maps of side 1, 2, 3, 4, 5, 6, 8, 10, 13 and 16, 680 tokens in all, each map produced by one forward pass conditioned on all the coarser ones. The maps are residuals: upsampled to 16 by 16 and added, they give the feature map the decoder reads. Below, the sequential passes: 256 for raster order, 10 for VAR. NEXT SCALE · ONE TOKEN MAP PER FORWARD PASS Σ upsample to 16 × 16, add → decoder next token · 256 passes next scale · 10 passes ← one pass per map, of 1 to 256 tokens

MAR: drop the quantizer

The codebook caps fidelity. How do you sample a continuous token autoregressively?

  • Not a Gaussian (one blurry mean for a multimodal distribution).
  • A small diffusion model per token: the Transformer outputs , an MLP (~21 M, about 5% of the model) denoises from it.

  • MAR: random order, parallel groups, FID 1.55. Fluid, NextStep-1: the same at text-to-image scale.

zi = the Transformer's output for position i · xi = the continuous token at position i · xi,t = xi noised to level t (here t is diffusion time, not a token) · ε = the sampled noise · εθ = the MLP, which tries to guess it

The axis is not what people think it is

categorical (codebook) continuous (diffusion / flow head)
one token at a time VQ-GAN, LlamaGen, Chameleon, Emu3 NextStep-1
parallel rounds MaskGIT, Muse, Show-o MAR, Fluid
coarse to fine VAR, Infinity —
all positions, every step discrete diffusion Stable Diffusion, FLUX, DiT
  • order and per-token distribution are independent choices
  • the other direction: diffusion language models (LLaDA, Gemini Diffusion, Mercury)

Images inside a language model

[TEXT: "a photo of"] [IMG_3742] [IMG_891] ... [IMG_5531] [TEXT: "a cat"]
  • Chameleon (2024): 512² image = 1 024 tokens from 8 192 codes, added to the text vocabulary
  • one Transformer, one next-token loss, from the first pretraining step

Is that the only way to put image generation inside an LLM?

Diagram · four designs, one sequence

Who does what

model image tokens generates images by
Chameleon, Emu3, Janus-Pro VQ next token
Emu3.5 (2025) IBQ, 131 072 codes next token, + parallel mode (~20×)
Transfusion, BAGEL VAE latents diffusion / flow loss inside the LLM
HunyuanImage 3.0 (2025) VAE latents Transfusion-style, 80 B MoE
MetaQuery, BLIP3-o — LLM conditions a separate diffusion model

Janus-Pro: SigLIP to understand, VQ to generate. Understanding wants semantics; generation wants pixels back.

What is not public

GPT-4o / GPT Image: "an autoregressive model natively embedded within ChatGPT". Nothing official beyond that. "AR + diffusion decoder" is a reading of a whiteboard picture in the launch post.

Gemini image models: billed as 1 290 output tokens per image. Architecture not described.

  • every design on the previous slide is consistent with what is public
  • say "not disclosed" instead of repeating a guess

AR vs diffusion in 2026

AR / masked over tokens diffusion / flow
ImageNet 256² FID VAR 1.73, MAR 1.55, RAR 1.48 DiT 2.27; REPA / RAE ≈ 1.1–1.4
text-to-image, open competitive leads (SD3, FLUX, Qwen-Image)
text + image, one model natural needs a 2nd loss or a 2nd model
sampling passes, 8–12 rounds, ~10 scales 1–50 passes over all positions
likelihood exact (tokens) bound / ODE

FID: lower is better · REPA, RAE: diffusion transformers that lean on a pretrained image encoder (DINOv2), chapter 22

Lab · tokens, measured

Colab notebook, ~2 h of work, T4 GPU (free tier, ~10 min of compute), MNIST, everything trained from scratch. Each section ends in a checkpoint.

  1. a judge: CNN accuracy + Fréchet distance
  2. a VQ tokenizer and its dead codes
  3. FSQ: no codebook
  4. the image as 49 tokens
  1. an autoregressive Transformer: bits per image
  2. sampling: 49 passes
  3. MaskGIT: rounds against quality
  4. quality against cost: the table is yours

open in Colab →

Takeaways

  1. A tokenizer turns an image into integers: at most bits, the fidelity ceiling.
  2. VQ codebooks collapse; FSQ / LFQ replace the codebook with a fixed grid.
  3. One sequential pass per token; the KV cache does not remove the sequence.
  4. MaskGIT: 8–12 rounds, but not 1. VAR: ~10 scales. MAR: continuous tokens.
  5. The real axes: order × per-token distribution.
  6. It came back for unification; on quality, diffusion still leads text-to-image.

Outline: (1) why this came back, and where it is used; (2) the tokenizer and codebook collapse; (3) autoregression and what it costs; (4) MaskGIT, VAR, MAR; (5) the two axes; (6) images inside a language model, and what is not public; (7) AR vs diffusion in 2026. Closes with the Colab lab.

Answer: one sequence, one loss, one model for text and images. Reading, reasoning and drawing in the same context.

Step through: image → encoder → 16×16×256 latent → nearest code → 16×16 integers (256 tokens, K = 16,384) → decoder.

Eight steps: encode; the nearest code replaces z_e (its number k is the token); decode and score; backward, the argmin is a wall; straight-through copies the gradient to z_e; the two other terms (the code moves toward z_e, z_e toward the code); who gets nothing: every code that was not chosen gets 0, the dead codes of the next slide.

Wide random init, K = 32: train 30 steps, it ends at 21/32 used. K = 64: 28/64. Then "copies of data points" (63/64) or tick "restart dead codes" (64/64): (nearly) 100%. The original VQGAN used 3.4% of its 16 384 codes on ImageNet.

The lab's code on a 2 × 2 image, K = 4 codes in 2-D: every number can be checked by hand. Step line by line in "the example on the page". Lines 2-6 are the tokenizer: dist, argmin → tokens [1, 3, 0, 3]; code 2 wins nothing and gets a zero gradient (line 5); the two auxiliary terms are the same number in the forward pass (0.0475), they differ only in where the gradient goes. Lines 7-10 are one step of the AR model: loss = −log 0.598 = 0.515 (blind guess log 4 = 1.386). On the last line move τ: 0.1 → always code 3, 5 → almost uniform. Cases: "a code far from the data" → a dead code, aux doubles; "a point between two codes" → a tie, argmin takes the first, the target t4 changes and the loss jumps to 3.0; "the model bets on another token" → loss 2.28 and the draw gives 1.

16×16 = 256 tokens, one per step. Switch to random order: still one at a time, same 256 passes.

Start without a cache: pass i recomputes the whole prefix, 36 positions for 8 tokens; the dashed arrows are the sequential dependency. Click "With a KV cache": 8 positions, but still 8 passes, and the new position still reads i keys (36 in all). The table: 49 tokens (the lab) 1 225 against 49; 256 tokens 32 896 against 256. Hover a pass for its numbers.

5, 20, 44, 75, 114, 159, 207, 256 kept after rounds 1–8: few early, many late. Scattered, not raster.

with 8 rounds it is recognised slightly more often (0.899 against 0.840), but its FD is more than twice the autoregressive model's (69.3 against 31.2): its digits are cleaner and more alike.

Step through the ten passes: each map comes out of one forward pass, conditioned on all coarser ones (the orange bracket); 1, 5, 14, 30, 55, 91, 155, 255, 424, 680 tokens so far. The bars: 256 passes for raster order, 10 for VAR. Last step: the maps are residuals, upsampled to 16 × 16 and added.

Click through the tabs: what an image position holds, how it is predicted, the attention mask, what decodes it.

Roughly at parity on ImageNet; diffusion leads standalone text-to-image; AR wins where generation shares a context with text.