Artificial Neural Networks and Deep Learning · Insper

Generative Models

Four ways to cut the problem

One question

You have a random number generator. You want a photograph.
How do you turn a distribution you can sample from into one you cannot?

  • Write down? Over a million pixels, no — and the constant that makes sum to 1 over every possible image is intractable on its own.
  • So every family makes the same move: cut the hard sampling problem into a sequence of easy ones.
  • They differ only in where they cut. That choice decides training stability, sample quality, whether you get a likelihood and how many forward passes an image costs.

Four ways to cut the problem

family the cut trained by passes per sample likelihood
autoregressive maximum likelihood one per dimension exact
latent variable (VAE) a bound (ELBO) one a bound
adversarial (GAN) none: a generator and a critic a minimax game one none
diffusion / flow time: noise → data in small steps regression tens to hundreds via the ODE

: the -th dimension (a pixel, a token); : the ones before it · : the latent, drawn from a simple · ELBO: a lower bound on · ODE: the differential equation that carries noise to data

Two of these rows are deprecated as headline methods. Two run the world.

Explicit or implicit density

You trained a model (weights ). Someone hands you a test image . Can the model tell you , the density it gives ?

  • Explicit: yes, or a bound on it — autoregressive (exact), VAE (bound), diffusion and flows (through the ODE).
  • Implicit: no — the GAN only samples. Noise in, image out; no density anywhere.
  • Consequence: a held-out likelihood exists for the first group only. For a GAN, the samples are the only evidence.

Three ways to train

  • Likelihood-based — maximize or its bound: autoregressive, VAE.
  • Adversarial — a critic stands in for the likelihood: GAN.
  • Regression onto a score or a velocity — diffusion and flow matching. Needs neither a normalized density nor a critic.

Score: , the direction in which the density rises. Velocity: , where a point should move at time of the noise → data path.

Left out of the table: normalizing flows — an invertible network with a cheap Jacobian, so change of variables gives an exact likelihood in one pass. The invertibility limits the architecture and they lost on quality; their continuous-time version is what flow matching trains (see the flow-matching class).

The autoregressive bill

= number of dimensions (pixels × channels); one factor per dimension.

One network pass per factor. How many passes for a 64 × 64 RGB image, pixel by pixel?

  • sequential passes (each factor waits for the previous one) — for one small image.
  • That is why autoregressive image models first compress the image into tokens: a 512 × 512, downsampled 16× per side, becomes 1,024 tokens (see the autoregressive-generation class).
  • The bill is the number of dimensions. It does not care how simple the image is.

The diffusion bill

  • Draw noise ; follow a learned velocity field from (noise) to (data).
  • With Euler steps, each step is : one network pass. You choose after training, at inference.
  • Few steps: fast and wrong. Many steps: slow and right. The bill is the number of steps.

Quality traded against latency, at sampling time, with the same network. That dial is the property that made diffusion win.

Simulator · the same target, three samplers

What the panel measures

Off the data = farther than from every mode (: the standard deviation of each Gaussian). In 2-D, : a perfect sample already shows .

Euler steps 1 4 8 16 32 128
ODE (flow matching) 100% 62% 20% 4% 1% 1%
+ Langevin corrector 100% 53% 16% 2% 1% 0%
network evaluations with corrector, 1 7 15 31 63 255
  • Past ~32 steps you are paying for nothing: 1% is the floor of a perfect sample, measured on 600 points per panel.
  • The corrector (Langevin: a small step up the score, plus noise) costs one score evaluation per step, except the last: .
  • The corrector helps at 8–16 steps, where integration error is large; at 128 there is none left to fix.

Why one step lands in the middle

With one Euler step, every sample lands in the centre of the ring — where there is no data. Why?

  • Flow matching trains on : the straight line from noise to a data point .
  • At the point is still : it says nothing about its mode.
  • So the best prediction (exact!) is the average: .
  • One step of size 1: , the centre of the ring.

E[x]z1 step4 steps128 steps

The path to a mode is curved; a straight step cannot follow it. Diagram: the ring's exact field, the same z with 1, 4 and 128 steps (with 4 it still stops between modes).

Simulator · one sample, two samplers, line by line

p(x1, x2) · the target
uniforms of the draw
noise z



  
  






  
  
  










The other axis: failure


Fitting = minimizing a divergence.  averages under its first argument — so it is not symmetric:




The model cannot represent the data exactly — which is always. Which direction gives what?








Simulator · forward and reverse KL






  
  
  
  












Forward KL: cover everything, blur


    

  • Punished wherever the data has mass and the model has none.
    • 

  • Its optimum covers every mode — including the empty valley between them.
    • 

  • This is maximum likelihood: . It is what VAEs and autoregressive models optimize.
    • 





Blur is not a bug in the decoder. It is the objective getting what it asked for.






In the lab: one Gaussian fitted by maximum likelihood to the ring leaves 78.7% of its samples off the data (farther than ) — and still touches all 8 modes (each gets ≥ 1% of the samples).








Reverse KL: pick a mode, drop the rest


    

  • Punished only where the model puts mass; indifferent to data it never visits.
    • 

  • Its optimum picks one mode: every sample plausible, a whole part of the distribution gone.
    • 

  • That is mode collapse — stated as a property of an objective, not an accident of training.
    • 





In the lab: the reverse-KL Gaussian lands on 1 of 8 modes, at the data's own width (). KL : on the chosen mode , so . Four starting points, four different modes.








Say it once, use it seven times




Blurry and mode-collapsed are not unrelated defects. They are the two ends of one asymmetry.




    

  • Every model in this module sits somewhere on it.
    • 

  • Knowing where tells you what its samples will look like before you run it.
    • 

  • The way out is not a better direction of KL: it is a model expressive enough not to have to choose.
    • 







The trilemma


For most of the 2010s: quality, coverage, speed — pick two.






family
kept
gave up






GAN
quality, speed
coverage




VAE, normalizing flows
coverage, speed
quality




diffusion
quality, coverage
speed








Which of the three would you give up, if you could only buy one back later?




Xiao, Kreis & Vahdat (2022), Tackling the Generative Learning Trilemma with Denoising Diffusion GANs, ICLR.






Speed is the one you can buy back


    

  • Quality and coverage are decided during training — they are what the model learned.
    • 

  • The number of sampling steps is decided at inference.
    • 

  • So diffusion took the corner where the missing property was recoverable — and then recovered it:
    • 

  • better solvers → rectified flows → distillation into a handful of steps → into one.
    • 

  • Step counts went from 1000 (DDPM, 2020) to under 10.
    • 







The seven classes, and what each is for






class
family
why it is here






VAE
latent variable
the compressor every latent diffusion model runs on




GAN
adversarial
its loss survives: decoders, super-resolution, one-step distillation




CLIP
not generative
the bridge: conditioning on a sentence




Stable Diffusion
diffusion
the full system: latent space, U-Net, guidance, samplers




Flow matching
flow
simpler training, straighter paths




Diffusion Transformers
diffusion
a Transformer backbone, and scaling laws




Autoregressive generation
autoregressive
images as tokens, any-to-any models










A history of bottlenecks






when
what
the bottleneck it removed






1985–2006
Boltzmann machines, deep belief nets
learning a distribution without labels




2013
VAE
backpropagating through a random node




2014
GAN
needing a tractable likelihood at all




2015–2020
diffusion, up to DDPM
quality without adversarial training




2021
CLIP, latent diffusion
conditioning on language; a 48× smaller space




2022–2023
Stable Diffusion, DALL·E 2, Midjourney
access: open weights




2023–2024
flow matching, DiT, distillation
straighter paths, a scalable backbone, < 10 steps






48× = (512 · 512 · 3) / (64 · 64 · 4) = 786,432 / 16,384: the pixels of a 512 × 512 image against Stable Diffusion's latent (VAE class).






Two things this course will keep saying




Evaluation is unsolved. FID is the standard number and a poor proxy: sensitive to the feature extractor and to resizing, not comparable across papers, and once models are good it tracks human judgment poorly (see the class on generative metrics).






The training data is other people's work. Consent, style on request, the people whose job it was: unresolved, litigated, and part of the engineering decision.








Measured in the lab · explicit densities


Eight Gaussians on a ring of radius 2, . NLL = mean of  over the test points, in nats (lower is better); the true density scores 1.476.






model
train NLL
test NLL
off the data
modes






one Gaussian (GMM )
3.547
3.546
78.7%
8/8




GMM 
1.479
1.478
1.0%
8/8




kernel estimate, 
0.891
2.581
0.9%
8/8




kernel estimate, 
1.439
1.500
1.2%
8/8






    

  • A training NLL below the truth is memorization: each point is scored by its own kernel.
    • 

  • Only the held-out likelihood is evidence.
    • 



GMM k: a mixture of k Gaussians · kernel: a Gaussian of width h on every training point · modes: how many get ≥ 1% of the samples


nat: the unit of a natural log (base e) · 1 nat ≈ 1.44 bits · every log in this deck is natural, so NLL and KL come out in nats






Measured in the lab · one model per family






model
test NLL
off the data
modes
passes






autoregressive, 64 × 64 bins
1.536
3.0%
8/8
2




GAN
—
7.6%
8/8
1




flow matching, 32 steps
1.514
4.7%
8/8
32




flow matching, 1 step

100%
0/8
1




sharp ring (): GAN, 3 seeds
—
22–46%
modes dip to 6–7
1




sharp ring: flow matching, 64 steps

6.4%
8/8
64






Passes: sequential network evaluations per sample (the autoregressive model pays 2 because the data has d = 2 dimensions; B = 64 bins per coordinate) · —: the GAN has no likelihood · blank: not measured. The GAN's off-data samples are bridges between the modes: a continuous generator from connected noise cannot make eight separate islands. The flow's NLL comes through the ODE (100 steps × 3 passes per point).






Lab · the map, measured


Colab notebook, ~2 h, synthetic 2-D data, CPU. Each section ends in a checkpoint with a check value.






    

  1. the data, the truth, two measurements
    2. 

  2. Gaussian mixtures and a kernel estimate
    3. 

  3. forward × reverse KL with one Gaussian
    4. 

  4. an autoregressive model on a grid
    5. 







    

  1. a GAN, and the sharp ring
    2. 

  2. flow matching: steps and the ODE likelihood
    3. 

  3. the map as a table
    4. 

  4. a rare mode with 2% of the data: which families keep it?
    5. 







open in Colab →






Takeaways


    

  1. Every generative model cuts an intractable sampling problem into tractable pieces; the cut decides cost, stability and likelihood.
    2. 

  2. Explicit models give  or a bound; the GAN is implicit and only samples.
    3. 

  3. Autoregressive cost scales with dimensions; diffusion cost with steps — a dial set at inference.
    4. 

  4. KL is asymmetric: forward covers and blurs, reverse seeks a mode and collapses.
    5. 

  5. The trilemma broke where the missing property could be bought back: speed.
    6. 

  6. VAEs and GANs live on as components. Evaluation and provenance are part of the work.
    7. 






Outline: (1) the question every generative model answers; (2) the four decompositions and what each
costs; (3) the same target, three samplers; (4) the KL asymmetry behind blur and mode collapse;
(5) the trilemma and the seven chapters that follow. Closes with the Colab lab.
This class is the map; the seven that follow are the territory. Let the class guess what "easy
pieces" could mean before the bullets: one pixel at a time, one latent and a decoder, one small
denoising step.
Ask which two before saying it: VAE and GAN no longer generate the headline images; autoregressive
(LLMs, image tokens) and diffusion/flow do. VAE and GAN come back later as components.
Every quantity is exact (score, velocity, discrete marginals): nothing is trained, so what you see
is the decomposition. Drag the steps to 1, then 8, then 32. Then turn on Langevin at 8 and at 128.
Shrink the autoregressive grid to 16 to see the staircase.
The corrector costs one extra (score) evaluation per step except the last: 2n − 1. This is the
ODE-versus-SDE trade in miniature (Song et al., 2021).
Four points on the corners of a square, so every number is checkable by hand. Lines 1–4: the AR
model reads p(x1) = 0.5, draws x1 = −1, then p(x2 | x1) = (0.8, 0.2), draws x2 = −1; log p = log 0.4,
in 2 passes. Lines 5–9 with n_steps = 4: x_hat starts at E[x] = (0, 0) and the sample stops at
(1, 0.82). Then "one step: the mean" (lands on (0, 0), the previous slide in numbers), n_steps = 8
(on the corner), and "unequal weights" (one step lands on (−0.6, −0.6): the mean, not the centre).
Two modes of different width and weight, one Gaussian model. Press "minimize KL(data‖model)":
it covers both modes with its peak in the valley. Press the other: it picks the narrow, heavy mode.
The landscape on the right shows both optima.
48×: 512·512·3 = 786,432 numbers against 64·64·4 = 16,384 in the latent. Shannon's n-grams (1948)
are the chain rule as a generative device, before any of this.