Skip to content

16. Generative Models

Every generative model answers one question: how do you turn a distribution you can sample from into one you cannot? You have a random number generator. You want a photograph. The gap between those two things is the whole field.

Nobody attacks it directly. \(p(x)\) over a million pixels is not something you can write down, and the normalizing constant alone is intractable. So every family in this module makes the same move — decompose the hard sampling problem into a sequence of easy ones — and they differ only in how they cut it. That choice determines everything downstream: training stability, sample quality, whether you get a likelihood, and how many forward passes an image costs.

This chapter is the map. The seven that follow are the territory.

Four ways to cut the problem

Family The decomposition Trained by Sampling cost Gives a likelihood?
Autoregressive Chain rule: \(p(x) = \prod_i p(x_i \mid x_{<i})\) Maximum likelihood, directly One pass per dimension Exact
Latent variable (VAE) \(p(x) = \int p(x\mid z)\,p(z)\,dz\) — a simple \(z\), a learned decoder A lower bound on the likelihood (ELBO) One pass A bound
Adversarial (GAN) No decomposition. Learn a generator, and a critic that judges it A minimax game One pass None
Diffusion / flow Time: a path from noise to data, learned as a local correction Regression onto a known target Tens to hundreds of passes Yes, via the ODE

Two of those rows are already deprecated as headline methods, and two run the world. The rest of this chapter is about why.

The same target, three samplers

Below, the same data distribution — eight Gaussians on a ring — is generated three ways. Every quantity in the panel is exact: the score, the velocity field and the discrete marginals are all available in closed form for this target, so nothing is trained and nothing is approximated. What you are watching is the decomposition, isolated from any question of whether a network fitted it well.

Three lessons, and they generalize far past this toy:

  1. The autoregressive cost is the number of dimensions. Two here, and free. One per pixel or per token for real data — which is why autoregressive image models work on 1024 latent tokens and not on a million pixels (chapter 23).
  2. The diffusion cost is the number of steps, and it is a dial. Drag it to 1: a single Euler step cannot follow a curved path and every sample lands in the middle of the ring. At 8 steps 20% of samples are still off the data; at 32 it is about 1%, which is the floor a perfect sample of this size would show anyway. This dial — quality traded against latency, at inference, after training — is the property that made diffusion win.
  3. Stochasticity buys robustness at moderate step counts. Turn on the Langevin correction: at 8 or 16 steps the sampler is measurably better because errors get corrected rather than accumulated; at 128 both are converged and it buys nothing. That is the ODE-versus-SDE trade in miniature.

The asymmetry that explains blur and collapse

The other axis is not cost but failure, and there is one idea underneath the two most famous failure modes in the field.

Fitting a model means minimizing a divergence, and KL divergence is not symmetric. Which direction you write it in changes what you get when the model cannot represent the data exactly — which is always.

\[ \underbrace{\text{KL}(p_{\text{data}} \,\|\, p_\theta)}_{\text{average over the \textit{data}}} \qquad\text{versus}\qquad \underbrace{\text{KL}(p_\theta \,\|\, p_{\text{data}})}_{\text{average over the \textit{model}}} \]
  • Forward KL is punished wherever the data has mass and the model has none. Its optimum covers everything, including the empty valley between the modes. This is maximum likelihood — what VAEs and autoregressive models optimize — and it is exactly why their samples look averaged. Blur is not a bug in the decoder. It is the objective getting what it asked for.
  • Reverse KL is punished only where the model puts mass, and is indifferent to data it never visits. Its optimum picks one mode and abandons the rest: every sample plausible, a whole part of the distribution gone. That is mode collapse, and stating it as a property of an objective rather than an accident of training is the point.

Say it once, use it seven times

Blurry and mode-collapsed are not unrelated defects. They are the two ends of one asymmetry. Every model in this module sits somewhere on it, and knowing where tells you what its samples will look like before you run it.

The trilemma, and how it was broken

For most of the 2010s the field looked like a hard three-way trade-off5:

  • High sample quality

  • Mode coverage and diversity

  • Fast sampling

GANs took quality and speed and gave up coverage. VAEs and flows took coverage and speed and gave up quality. Diffusion took quality and coverage and gave up speed — and that turned out to be the right corner to be in, because speed is the only one of the three you can buy back afterwards. Quality and coverage are decided during training; the number of sampling steps is decided at inference. The history of the last three years is largely the story of buying it back: better solvers, rectified flows, distillation into a handful of steps, then into one.

The seven chapters, and what each is for

Chapter Family Why it is here
17 · VAE Latent variable Not a competitive image generator, and not why it matters. It is the compressor every latent diffusion model runs on, and the clearest place to learn the latent-space idea.
18 · GAN Adversarial Displaced as a generator. Its loss survives everywhere — in autoencoder decoders, super-resolution, vocoders, and one-step distillation.
19 · CLIP Not generative The bridge. Learning a shared image–text space is what made conditioning on a sentence possible; everything after this chapter is text-conditioned.
20 · Stable Diffusion Diffusion The full system: latent space, U-Net, text conditioning, classifier-free guidance, samplers.
21 · Flow matching Flow The reformulation that made training simpler and paths straighter, and now underlies the state of the art.
22 · Diffusion Transformers Diffusion Replacing the U-Net with a Transformer, and getting scaling laws for image generation.
23 · Autoregressive generation Autoregressive Generating images as token sequences — the route to genuinely unified any-to-any models.

A short history, read as a sequence of bottlenecks

Each era is best understood by which of the four decompositions was affordable at the time.

When What happened The bottleneck it removed
1948 Shannon's n-grams generate text None yet — but the chain rule as a generative device
1985–2006 Boltzmann machines, deep belief nets1 Learning a distribution without labels. Sampling stayed intractable.
2013 VAE2 — the reparameterization trick Backpropagating through a random node. Latent-variable models became trainable by gradient descent.
2014 GAN3 Needing a tractable likelihood at all. Quality jumped; coverage and stability were the price.
2015–2020 Diffusion4 — from a curiosity to DDPM Sampling quality without adversarial training, by cutting the problem along time.
2021 CLIP, latent diffusion Conditioning on language, and running diffusion in a 64× smaller space — which is what made it affordable on one GPU.
2022–2023 Stable Diffusion, DALL·E 2, Midjourney Access. Open weights, and the beginning of the copyright and provenance arguments.
2023–2024 Flow matching, DiT, distillation Straighter paths, a scalable backbone, and step counts falling from 1000 to under 10.
2024–2026 Video, any-to-any, real-time Coherence over time, and one model that reads and writes every modality.

Two things this course will keep saying

Evaluation is genuinely unsolved. FID is the standard number and it is a poor proxy for what anyone cares about — it is sensitive to the feature extractor, it is not a distance you can trust across papers, and it is uncorrelated with human judgment past a point. Chapter 8.3 is about how to do better.

The training data is other people's work. Where the images came from, whether consent was given, whose style a model reproduces on request, and what happens to the people whose job it was — these are unresolved, they are actively litigated, and they are part of the engineering decision, not a separate ethics module bolted onto the end.

Key takeaways

  1. Every generative model decomposes an intractable sampling problem into tractable pieces. The cut you choose decides cost, stability and quality.
  2. Autoregressive cost scales with dimensions; diffusion cost scales with steps. That is why images went latent and why step counts are the thing everyone optimizes.
  3. KL is asymmetric. Forward KL covers modes and blurs; reverse KL seeks modes and collapses. Two famous failures, one cause.
  4. The trilemma — quality, coverage, speed — was broken by choosing the corner where the missing property could be bought back later. Diffusion gave up speed and then took it back.
  5. VAEs and GANs are still everywhere, as components rather than as generators. Read chapters 17 and 18 with that in mind.
  6. Evaluation and provenance are unsolved. Treat both as part of the work.

  1. Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines — in Parallel Distributed Processing. Energy-based generative modelling, decades before it was affordable. ↩

  2. Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes — ICLR. The ELBO and the reparameterization trick. ↩

  3. Goodfellow, I., et al. (2014). Generative Adversarial Networks — NeurIPS. ↩

  4. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models — NeurIPS. The paper that made diffusion competitive, building on Sohl-Dickstein et al. (2015). ↩

  5. Xiao, Z., Kreis, K., & Vahdat, A. (2022). Tackling the Generative Learning Trilemma with Denoising Diffusion GANs — ICLR. Where the trilemma is stated cleanly. ↩

  6. Song, Y., et al. (2021). Score-Based Generative Modeling through Stochastic Differential Equations — ICLR. The unification of diffusion and score matching, and the predictor–corrector sampler used in the panel above. ↩

  7. Bond-Taylor, S., Leach, A., Long, Y., & Willcocks, C. G. (2022). Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models — IEEE TPAMI. The survey to read alongside this module. ↩