Artificial Neural Networks and Deep Learning · Insper

Generative Adversarial Networks

The architecture lost; the loss won

A loss you cannot write down

You want to generate faces. What is the loss?

  • against a target? There is no target — and the average of plausible faces is the blur of the VAE chapter.
  • A hand-written "realism" score? Every one ever written is satisfied by something that is not a face.
  • Goodfellow (2014): stop writing the loss — learn it. A second network learns to tell real from generated, and whatever it uses to tell them apart is the measure of realism.

Two players, one value function

A GAN: noise z goes through the generator G to a fake sample G(z); real samples x come from the training data. The discriminator D receives both and outputs the probability that its input is real. D wants D(x) near 1 and D(G(z)) near 0; G wants D(G(z)) near 1, and its gradient flows back through D. z ~ 𝒩(0, I) noise G generator G(z) fake x real · training data D discriminator D(·) ∈ (0, 1) probability of “real” D wants D(x) → 1 D wants D(G(z)) → 0 G wants D(G(z)) → 1 generator step: the gradient of D(G(z)) flows back through D into G

Neither network is told what "realistic" means. At the optimum and : a coin flip.

: average (over the real data or over the noise ) · : the distribution of the real data · : the distribution of the samples · : is 1/2 at every point.

What the discriminator learns

For a fixed generator, the best discriminator is

Where it comes from: at each , maximizes ; setting the derivative to zero gives .

At a point where the data are 3 times denser than the generator's samples, what is ?

  • : . Equal densities give ; generator mass where there are no data gives .
  • is a density-ratio estimator: its gradient points the generator to where it is short of mass.

Measured: a trained against the formula

Lab, section 2: data = ring of 8 Gaussians (standard deviation 0.05); a fixed "generator" = the same ring with standard deviation 0.2.

distance from a mode centre (formula) (trained)
0.00 0.941 0.957
0.10 0.710 0.791
0.15 0.191 0.380
0.30 0.000 0.001
  • At a mode's centre the data are times denser (a 2-D Gaussian of standard deviation peaks at ): .
  • Plugging back: . 's loss is ; the best possible here, 0.545; trained 0.599.

JS: Jensen–Shannon divergence, 0 when the two distributions are equal, when they do not overlap (0.42 here, by Monte Carlo).

The numbers to read a run by

At equilibrium , so:

quantity at equilibrium where it comes from
and 0.5 with :
discriminator loss
generator loss

A discriminator loss near zero is not progress: it means the discriminator separates real from fake perfectly.

One training step

  1. Discriminator step — real batch labelled 1, generated batch labelled 0. The fakes are detached (cut from the graph: no gradient reaches 's weights), because this step must not touch .
  2. Generator step — a fresh batch of fakes, scored by ; wants them called real.
  3. Repeat. Optionally discriminator steps per generator step.

Two networks, two optimizers, two opposite objectives. It is a game, not an optimization: each loss is measured against an opponent that keeps moving.

Two generator losses, one fixed point

With (: the logit, 's output before the sigmoid), the gradient with respect to :

loss gradient at
minimax (2014): minimize ?
non-saturating: minimize ?

The discriminator rejects a fake with . Fill in the column: which loss pushes harder?

To check the gradients: .

Two generator losses · answer

loss gradient: where it comes from at
minimax
non-saturating
  • Ratio : the non-saturating loss pushes 99 times harder, exactly when the generator is losing.
  • In general the ratio is , which grows without bound as .
  • Same fixed point: both want high. The 2014 paper itself recommends the second one.

Simulator · one training step, line by line

real x noise z D: w, b G: a, c

Measured: the saturation, and what Adam does with it

Discriminator given a 300-step head start, then the generator's gradient under each loss (lab, sections 4 and 8):

ring MNIST DCGAN
after the head start 0.0001 0.0005
ratio per fake, ≈ 10,000 ≈ 2,000
measured: ‖∇G‖ non-saturating / minimax ≈ 8,000× ≈ 1,900×
  • Generator on SGD: minimax does not move (; mean distance from the fakes to the nearest mode 1.91 → 1.91). Non-saturating: 1.91 → 0.68.
  • With Adam both move (minimax 1.91 → 0.29, non-saturating 1.91 → 0.39): Adam is invariant to the gradient's scale, so it hides the saturation.

‖∇G‖: norm of the gradient with respect to all of 's weights, over a batch of fakes; it mixes fakes with different , which is why it lands below the per-fake ratio. DCGAN: deep convolutional GAN, built on slide 22 · MNIST: 28 × 28 images of handwritten digits.

Simulator · a GAN in your browser

What training sees vs. what is happening

What training sees: , sitting near 0.5.

What is happening: coverage (how many of the ring's 8 modes get samples), measured apart by code the loop never runs.

Eight runs at the default settings, same orange line: 7, 5, 8, 6, 6, 8, 8, 8 modes.

A GAN has no loss you can read: no early stopping, no comparing runs by their curves, no telling whether a checkpoint is better without a separate metric.

That is why diffusion — an ordinary regression loss that goes down — replaced it.

Reading the curves

The discriminator loss falls to 0.05, the generator loss climbs to 6. Good news?

  • No: the discriminator wins outright. Generator loss → : nearly every fake is rejected.
  • Balanced is and .
  • The curves do not go down in a healthy run: each player chases a moving target.
  • And balanced curves do not mean good samples — next slide.

Mode collapse

Why does the objective allow a generator that makes only a few outputs?

  • Nothing in it rewards diversity: each fake is scored on its own.
  • If one output fools the current , putting all the mass there lowers 's loss.
  • Until catches up — and hops to the next accepted spot.
  • Behaviour like the reverse KL (Kullback–Leibler divergence) of the generative-models chapter, : it punishes fakes off the data, not forgotten modes. Mode-seeking, not mass-covering.

Measured: collapse provoked, invisible, fixed

Lab, section 5: the ring, 2,000 steps, three seeds each.

setting modes (3 seeds) loss D
balanced: both 7, 8, 7 0.45–0.47 1.24–1.42
slow D: (5× lower) 0, 0, 0 0.48–0.51 1.32–1.43
slow D, 6, 8, 7 0.47–0.50 1.22–1.32
  • Slow D: every sample in one clump between two modes, dragged along the ring.
  • Its curves look more balanced than the runs that work.

lr: Adam's learning rate · : steps per step · modes: how many of the 8 get samples, measured apart.

Why samples land between the modes

  • is a continuous function of .
  • A continuous map sends a connected set to a connected set.
  • To reach 8 separate modes it must stretch bridges between them — some samples land on the bridges.
  • With a 1-D latent the whole output is one curve.
  • In the panel: 2 of 8 modes, 57% of samples off the data — more than 3 standard deviations from every mode (our run).
  • With a 2-D latent: 18–52% still off the data.

The failure modes, named

failure looks like why
mode collapse a few outputs, whatever nothing rewards diversity
vanishing gradient stalls, won too completely
oscillation quality up, down, up games need not converge
D overfitting quality improves, then degrades memorizes the training set

The fixes that stuck — WGAN-GP, spectral norm, R1, ADA — do one thing: stop the discriminator from becoming too sharp.

WGAN-GP: Wasserstein GAN with gradient penalty, on the gradient of its discriminator (there called the critic) · spectral norm: bounds how fast can change · R1: penalizes 's gradient on real data · ADA: adaptive augmentation of what sees.

Why JS gives no direction, and what WGAN changed

  • Against an optimal , the generator minimizes .
  • Early on the two distributions do not overlap. Then JS — whether the fakes are a millimetre or a kilometre away.
  • A flat loss surface gives the generator no direction.
  • WGAN: the Wasserstein (earth mover's) distance still grows with the distance. Its critic must be 1-Lipschitz → gradient penalty (WGAN-GP), spectral normalization.

Why : , with ; with no overlap, where lives (and where lives), so each KL is . 1-Lipschitz: .

GAN Lab · what is on the screen

Kahng et al., IEEE VIS 2018 — an interactive GAN trained live in the browser on 2-D data.

element in this class
green / purple dots real and generated samples
background heatmap — an estimate of ; gray where
the warped grid (manifold) as a map of the noise square; darker = denser
pink lines the direction 's gradient pushes each fake
loss bars vs. Metrics (KL, JS) what training sees vs. a measurement done apart

GAN Lab · go and try

  1. Ring, play: watch the heatmap fade to gray as the fakes overlap the data.
  2. Step by step: update only a dozen times — the fakes climb towards green and overshoot. Then only .
  3. Non-convergence: both learning rates up; then lower one at a time. Which stops the oscillation?
  4. Collapse: disjoint with more layers and neurons; or the ring with 1D noise.
  5. Rebalance: updates per epoch, the learning-rate ratio, log loss vs. LeastSq. Judge by the samples and JS, not by the losses.

open GAN Lab →

A DCGAN on MNIST

DCGAN: deep convolutional GAN. Upsample + convolve (the CNN chapter: transposed convolutions leave a checkerboard).

generator output discriminator output
64 image 1 × 28 × 28
Linear(64, 6272) + BN + ReLU 128 × 7 × 7 Conv2d(1, 32, 4, 2, 1) + LReLU 32 × 14 × 14
Upsample + Conv2d(128, 64, 3, 1, 1) + BN + ReLU 64 × 14 × 14 Conv2d(32, 64, 4, 2, 1) + BN + LReLU 64 × 7 × 7
Upsample + Conv2d(64, 32, 3, 1, 1) + BN + ReLU 32 × 28 × 28 Linear(3136, 1) 1 logit
Conv2d(32, 1, 3, 1, 1) + Tanh 1 × 28 × 28

Conv2d(in, out, k, stride, padding) · BN: BatchNorm · LReLU: LeakyReLU · Upsample doubles height and width · outputs in channels × height × width.

Side in : , . In : by the Upsample; each 3×3 with padding 1 keeps the side. , .

Three epochs generator steps (batch of 128).

DCGAN · where the parameters come from

layer count parameters
Linear(64, 6272) + BN 420,224
Conv2d(128, 64, 3, 1, 1) + BN 73,920
Conv2d(64, 32, 3, 1, 1) + BN 18,528
Conv2d(32, 1, 3, 1, 1) 289
generator 512,961
Conv2d(1, 32, 4, 2, 1) 544
Conv2d(32, 64, 4, 2, 1) + BN 32,960
Linear(3136, 1) 3,137
discriminator 36,641

Conv: , the formula of the CNN chapter · Linear: inputs · outputs + outputs · BN: 2 per channel (scale and shift) · Upsample, ReLU, LReLU and Tanh: none.

Measured: what the loss cannot say

Judge: a CNN trained on real MNIST (test accuracy 0.979), over 2,000 samples.

loss D / loss G confident effective classes
real test images — — 0.91 9.94
DCGAN, 3 epochs 0.99 / 1.22 0.33 0.48 8.71
same, slow D (, 10× lower) 1.40 / 0.69 0.50 0.38 7.85
  • The DCGAN makes every digit, unevenly: 38 sixes in 2,000 (a balanced one: 2,000/10 = 200).
  • The slow-D run has the "perfect" curves (1.40 and 0.69, the equilibrium numbers) — and the worse samples.

confident: fraction of samples the judge labels with probability above 0.9 · effective classes: , with the entropy and the judge's mean output — 10 if the digits come out evenly, 1 if only one does.

The lineage, briefly

model the contribution
DCGAN (2015) convolutional players, rules that made training reproducible
Conditional GAN (2014) a label fed to both networks
pix2pix / CycleGAN (2017) image-to-image translation, even without pairs
WGAN / WGAN-GP (2017) another distance, a loss that tracks quality
StyleGAN 1–3 (2019–21) the best GAN faces; the editable latent
GigaGAN (2023) text-to-image, 0.13 s per 512 px image — after the field had moved

Where the adversarial loss lives in 2026

where what the discriminator does
VAE decoder of latent diffusion punishes the blur of the reconstruction loss (VQGAN recipe: reconstruction + perceptual + adversarial, VAE chapter)
super-resolution Real-ESRGAN: one low-res image, many sharp answers
neural vocoders HiFi-GAN: spectrogram → waveform
1–4 step distillation ADD (adversarial diffusion distillation), SDXL-Turbo: "looks real", not "matches the teacher's mean"
domain adaptation features the domain cannot be read from

Use it when the target is one of many valid answers and their average is itself a bad answer.

Lab · the game, measured

Colab notebook, ~2 h. Ring of 8 Gaussians on CPU, then MNIST on a T4 GPU. Each section ends in a checkpoint.

  1. the ring and a coverage meter
  2. the optimal discriminator, measured
  3. a GAN on the ring; reading its curves
  4. minimax × non-saturating, SGD × Adam
  5. mode collapse: provoked, invisible, fixed
  1. a DCGAN on MNIST
  2. which digits? a classifier as judge
  3. the saturation, on images
  4. your turn: break the DCGAN, fix it, and check what the curves said

open in Colab →

Takeaways

  1. A GAN learns its loss. The optimal discriminator is the density ratio .
  2. The 2014 generator loss saturates; the non-saturating has the same fixed point and, on the MNIST GAN after a head start, a gradient ~1,900× larger.
  3. No loss you can read: for 8 modes and for 0. Coverage is measured apart.
  4. Mode collapse is allowed by the objective; an unbalanced pair provokes it, rebalancing fixes it.
  5. Every fix that stuck keeps the discriminator from becoming too sharp.
  6. The architecture lost to diffusion; the adversarial loss lives on wherever the answer is one of many.

Outline: (1) a loss you cannot write down, and the two players; (2) what the discriminator learns; (3) the two generator losses; (4) the panel: what training sees vs. what happens; (5) mode collapse, provoked and fixed; (6) GAN Lab; (7) a DCGAN on MNIST, measured; (8) where the adversarial loss lives in 2026. Closes with the Colab lab.

The trained D is close, and smoother at the edge: a finite network trained for a finite time.

Default case = the previous slide: D rejects the first fake with D(G(z)) = 0.01. Step line by line: loss_d = 0.0295 (near zero: D is winning), D's own step is tiny. At the last line: σ − 1 = −0.99, c moves −1.00 → −0.71. Switch to minimax: −σ = −0.01, c only reaches −0.99; per fake 99× (and 30×). Then "long head start" (ratio ≈ 9,900, as in the lab), "equilibrium" (1.386 and 0.693, nothing moves) and "D fooled" (the fakes move left, towards where D says real, not towards the data).

Default first: point at the orange line (D(G(z))) near 0.5 and the green line (coverage). Then latent dimension 1: 2 of 8 modes in our run, orange line unchanged. Then retrain at identical settings a couple of times. Minimax trains here because D never gets far ahead.

About 20 minutes. The page has the same session, step by step, tied to the chapter's sections.