18. GANs

For seven years the GAN was how you generated an image. Then, in about eighteen months, it stopped being that — diffusion took the headline task and did not give it back. A chapter that presents GANs as the state of the art would be teaching 2019.

So this chapter does something more useful. The architecture lost; the idea did not. An adversarial loss — learn what "realistic" means instead of specifying it — is running right now inside the decoder of Stable Diffusion's autoencoder, inside the vocoder that turns your text-to-speech output into audio, and inside the distillation that makes a one-step image generator possible. The GAN is no longer the model. It has become a loss function, and it is everywhere.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Both simulators of this page run inside the slides. Open the slides in a new tab

The idea: a loss you cannot write down

Suppose you want to generate faces. What is the loss?

\(\ell_2\) against a target does not work — there is no target, and averaging over plausible faces gives you the blur from chapter 17. Hand-designed measures of "realism" fail immediately: every one anybody has written down is trivially satisfiable by something that looks nothing like a face.

Goodfellow's answer¹ is to stop writing the loss and learn it. Train a second network — a discriminator — whose only job is to distinguish real data from generated data. Whatever it finds to distinguish them by is the measure of realism, and it updates as the generator improves.

The generator maps noise to data; the discriminator judges. Neither is given a description of what "realistic" means. A gentle walkthrough of the same picture: ⁴.

Two players, one value function:

Here \(p_{\text{data}}\) is the distribution of the real data, \(p_z\) that of the noise (\(\mathcal{N}(0, I)\) in the diagram) and \(p_G\) that of the generated samples \(G(z)\). The discriminator maximizes \(V\) — pushing \(D(x) \to 1\) on real data and \(D(G(z)) \to 0\) on fakes. The generator minimizes it. At the theoretical optimum \(p_G = p_{\text{data}}\) and \(D \equiv 1/2\): the discriminator can do no better than a coin flip.

One training step of the lab's loop, line by line — a discriminator step, then a generator step — with a one-unit \(D\) and a straight-line \(G\) on 1-D data, so every number can be checked by hand. The default case is the example above (\(D(G(z)) = 0.01\)); switch the generator loss to see the \(-0.99\) become \(-0.01\):

What the discriminator learns

Fix the generator and ask which discriminator maximizes \(V\). Point by point, \(D(x)\) should maximize \(p_{\text{data}}(x)\log D + p_G(x)\log(1 - D)\), and setting the derivative to zero gives

The discriminator is a density-ratio estimator. Where the data are three times denser than the generator's samples, \(D^* = 3/(3+1) = 0.75\); where the two densities are equal, \(D^* = 1/2\); where the generator puts mass and the data have none, \(D^* = 0\). Its gradient therefore points the generator towards the regions where it is short of mass.

Substituting \(D^*\) back into \(V\):

So, against an optimal discriminator, the generator minimizes the Jensen–Shannon divergence, and the best discriminator loss (the binary cross-entropy, \(-V\)) is \(2\log 2 - 2\,\mathrm{JS}\). That gives the reference values for reading a training run. At equilibrium the discriminator loss sits at \(2\log 2 \approx 1.39\) and the non-saturating generator loss at \(\log 2 \approx 0.69\). A discriminator loss near zero is not progress: it means the discriminator separates real from fake perfectly.

Training one, and looking at what you cannot see

Below is a real GAN — two MLPs, hand-written backpropagation, Adam — trained in your browser on eight Gaussians arranged in a ring.

The left panel is worth watching on its own: the discriminator's decision surface does not settle, because there is no fixed target. It reshapes itself around wherever the generator currently puts mass, and the generator chases it. This is a two-player game, not an optimization, and a game has no value that must go down.

Now the third panel, which is the point of the whole exercise. The orange line is \(D(G(z))\) — what the training loop sees, and what both losses are computed from. It sits near 0.5 and stays there. The green line is mode coverage, measured separately, by code the training loop never runs.

Set the latent dimension to 1 and retrain. The generator is now a continuous function of a single number, so all its samples lie on one curve. In our first run at that setting it covered 2 of the 8 modes and 57% of its samples landed off the data — and the orange line did not move. With a 2-D latent, 18–52% of the samples still landed off the data across our runs: a continuous generator maps one connected latent space onto one connected set, so it has to stretch thin bridges between separate modes.

The other two controls are worth a run each. With three discriminator steps per generator step our run reached all 8 modes. The minimax loss trains here too, because in this panel the discriminator never gets far ahead (where \(D(G(z))\) stays near 0.5 both gradients have the same size). It fails when the discriminator wins the first steps, as it does on images; the lab measures that case.

The failure modes, named

Collapse does not need anything exotic. In the lab, the same small GAN on the ring covers 7–8 modes with both learning rates at \(10^{-3}\). Cut only the discriminator's rate to \(2 \times 10^{-4}\) and all three seeds end with 0 modes covered: the generator piles every sample into one clump between two modes and drags it along the ring as the slow boundary moves. Its \(D(G(z))\) is 0.48–0.51 and its discriminator loss 1.32–1.43 — the curves look more balanced than in the runs that work. Five discriminator steps per generator step bring coverage back to 6–8 modes.

The mitigations that stuck: WGAN-GP¹⁵⁷ (replace the JS divergence with a Wasserstein distance and constrain the gradient of the critic, as WGAN calls its discriminator — when the two distributions do not overlap, JS is stuck at \(\log 2\) however far apart they are, so it gives no direction, while the Wasserstein distance still grows with the distance), spectral normalization⁸ (bound the discriminator's Lipschitz constant, the most its output can change per unit of change in its input; one line of code), R1 regularization⁹ (penalize the discriminator's gradient on real data) and adaptive discriminator augmentation (ADA)¹⁰ (augment what the discriminator sees so it cannot memorize). Each one is, at bottom, the same move: stop the discriminator from becoming too sharp, because a discriminator that is perfectly confident gives the generator a gradient that points nowhere.

GAN Lab: the same game, with every knob

GAN Lab¹³ shows more of the machinery than the panel above, and it lets you change the hyperparameters while the model trains.

What is on the screen. Click the pencil icon next to the model overview to show the hyperparameters. On the right, the layered distributions view puts everything on one plane:

A guided session (about 20 minutes). Click a distribution to start from an untrained model; each one loads its own hyperparameters, so read them before you change anything.

1. The heatmap is \(D^*\). Pick the ring and press play. While the fakes are still off the ring, the heatmap is green on the data and purple on the fakes. As they overlap, it fades towards gray — \(D \approx 1/2\) wherever the two densities agree, as the formula for \(D^*\) says. Compare the JS curve in the Metrics chart with the loss bars: which one tells you that the fit is getting better?
2. A game, not an optimization. Pause, open the step controls (the skip icon) and update only the generator a dozen times. The fakes move up the heatmap towards green — not towards the data — and, with the heatmap frozen, they keep going. Now update only the discriminator: the fakes stay put and the boundary is redrawn around them. Alternate the two. This is the loop of the lab, one step at a time.
3. Non-convergence. Raise both learning rates to their largest values and play. Describe what the fakes do. Then bring back one learning rate at a time — which one stops the oscillation? (The GAN Lab paper reports that it was the generator's.) Relate it to the oscillation row of the failure table.
4. Mode collapse. Pick the distribution of three separate clusters (disjoint), add hidden layers and neurons to both networks and train several times; the paper reports that the fakes then often collapse into one point. Separately, on the ring, set the noise to 1D: the manifold becomes a single curve, the situation of the panel above with latent dimension 1. Count the clusters the fakes reach, and read what the loss bars said while it happened.
5. Rebalance. In a collapsed or oscillating run, change one thing at a time: the discriminator's updates per epoch (the \(k\) of the paper and of the lab), the ratio of the two learning rates, or the loss (log loss is the non-saturating loss of this chapter; LeastSq is the least-squares GAN¹⁶). Write down which change helped, and how you knew — from the samples and the JS curve, not from the losses.

GAN Lab has no minimax option: its generator always uses a non-saturating or least-squares loss. The saturation of the 2014 loss is measured in the lab below.

The lineage, briefly

Where the adversarial loss actually lives now

This is the part to remember, because you will use it without ever training a GAN end to end.

Lab: the game, measured

Every claim on this page is measured in a Colab notebook: first on the ring of eight Gaussians (on CPU, a few minutes), then with a small convolutional GAN on MNIST (choose a T4 GPU in Colab). Nine sections, each ending in a checkpoint with a check value:

1. the ring, and a coverage meter that the training loop never sees;
2. the optimal discriminator: a trained \(D\) against the formula \(p_{\text{data}}/(p_{\text{data}} + p_G)\), and the best loss against \(2\log 2 - 2\,\mathrm{JS}\);
3. a GAN on the ring, and how to read its loss curves;
4. minimax against non-saturating: the gradient at \(D = 0.01\), the gradient after a discriminator head start, and what SGD and Adam do with it;
5. mode collapse provoked by a slow discriminator, invisible in \(D(G(z))\), and fixed with more discriminator steps;
6. a DCGAN on MNIST (upsample + convolve), trained for three epochs;
7. which digits it makes, measured with a classifier;
8. the saturation again, on images;
9. your turn: break the DCGAN and fix it, and check what the curves said.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. A GAN learns its own loss. The discriminator's job is to discover what "realistic" means so you do not have to specify it. Against a fixed generator its optimum is the density ratio \(p_{\text{data}}/(p_{\text{data}} + p_G)\).
2. The generator loss in the original paper saturates; everyone uses the non-saturating form \(-\log D(G(z))\) instead. Same fixed point, a gradient that is large when the generator is losing.
3. The real problem is not instability, it is unmeasurability: \(D(G(z))\) sits near 0.5 whether the model is excellent or has collapsed. There is no loss to early-stop on. At equilibrium the losses sit at \(2\log 2\) and \(\log 2\); they do not go down.
4. Mode collapse is what the objective permits, not an accident — nothing in it rewards diversity. An unbalanced pair of players is enough to provoke it.
5. Every stabilization that stuck does the same thing: stop the discriminator from becoming too sharp (WGAN-GP, spectral norm, R1, ADA).
6. Diffusion won because it has an ordinary regression loss that goes down, is stable, and covers modes. GANs still win on inference speed — one forward pass.
7. The adversarial loss is everywhere: in latent-autoencoder decoders, super-resolution, vocoders, and few-step distillation. Use it when the answer is one of many and averaging is wrong.