20. Stable Diffusion

Chapters 16 to 19 assembled the parts. This chapter is where they become a system you can run.

Stable Diffusion⁴ is not one idea; it is four, stacked, and the stack is what made image generation go from a research demo requiring a cluster to something that runs on a laptop:

1. A diffusion process that turns generation into a long sequence of easy denoising problems.
2. A latent space, from a VAE, so that sequence runs in a space 48× smaller than pixels.
3. A text encoder, from CLIP, so the process can be told what to make.
4. Classifier-free guidance, so it actually listens.

The first three appear in most explanations. The fourth is usually skipped, and it is the one that turns a model which technically conditions on text into one that obeys a prompt.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. The three simulators of this page, and three of its diagrams, run inside the slides. Open the slides in a new tab

Diffusion Models

Diffusion models are trained to predict a way to slightly denoise a sample in each step; after enough iterations, a result appears. The idea goes back to 2015¹; the same machinery has since been applied to images, speech, 3D shapes, molecules and graphs.

The two chains run over the same frames. Step through them, or hover a frame to see its arrows; the equations of the next sections are the ones the captions use.

Forward diffusion

A sample \(x_0\) is corrupted with Gaussian noise over steps \(t = 1 \ldots T\):

where \(\mathcal{N}(x;\, m,\, \Sigma)\) is the Gaussian density of \(x\) with mean \(m\) and covariance \(\Sigma\), \(I\) is the identity (independent noise in every pixel) and \(\beta_t\), a small number that grows with \(t\), is the noise schedule. \(T = 1000\) in DDPM and in Stable Diffusion. Written this way it looks like a loop, and it would be useless if it were one. The step that makes diffusion trainable is that the composition of all those Gaussians is itself Gaussian, so any noise level can be reached in closed form:

Sample a random \(t\), jump straight there, and train. No simulation of the chain, no sequential dependency between training examples.

The training objective is one line

Given that, what should the network predict? The elegant answer, and the one DDPM² settled on: predict the noise that was added.

That is it. A mean-squared error between two vectors, where the target is a sample of Gaussian noise you drew yourself. \(\epsilon_\theta\) is the network (the U-Net), \(\theta\) its weights, and \(c\) is the conditioning (the text, in Stable Diffusion); the average runs over training images \(x_0\), noises \(\epsilon\) and steps \(t\) drawn uniformly from 1 to \(T\). It is ordinary supervised regression, which is precisely what a GAN (chapter 18) does not have, and the single largest reason diffusion displaced it: there is no adversary, so training is stable and the loss is a meaningful signal of progress.

Reverse diffusion

Sampling runs the process backwards:

\(\mu_\theta\) and \(\Sigma_\theta\) are the mean and the covariance of each backward step; DDPM fixes the covariance to \(\beta_t I\) and learns only the mean. (The text \(c\) is left out of \(\mu_\theta\) and \(\Sigma_\theta\) to keep them short.) We cannot use the true \(q(x_{t-1} \mid x_t)\) — it depends on the data distribution and is intractable. The network approximates it, and the U-Net¹³ is the usual architecture because denoising needs both fine detail and global structure, which is exactly what an encoder–decoder with skip connections provides.

The noise prediction is all a step needs. Inverting the closed form gives the network's current guess of the clean sample, and DDPM's mean takes a small step from \(x_t\) towards it:

\(\hat x_0\) is worth looking at: early in sampling it is a blurry average of everything the prompt allows, and it sharpens step by step. The lab decodes it at every step, and the Diffusion Explainer below shows the same progression.

Training and sampling, side by side

The same network runs in two loops of very different shape. Training, per step: pick a random \(t\) for each image, jump to that noise level with the closed form above, embed \(t\) and regress the noise. Sampling: start from noise and call the network \(T\) times in a row, each call taking the output of the previous one.

The noise schedule is not a detail

\(\beta_t\) decides how much signal survives at each \(t\), and therefore what the model spends its training budget learning. Two well-known bugs live in this one function.

Terminal SNR. Stable Diffusion 1.x ships a scaled-linear schedule whose \(\bar\alpha_T \approx 0.0047\) — not zero. Training therefore always leaves a faint trace of the real image in the last step (\(\sqrt{\bar\alpha_T} \approx 0.068\) of it), while sampling starts from actual noise. The mismatch has a famous consequence: SD 1.x cannot produce a genuinely black or genuinely white image, because the mean brightness of the training set leaks into the first step²³. Rescaling the schedule to zero terminal SNR fixes it.

Resolution. A noise level does not mean the same thing at 512 px and at 1024 px: more pixels means more redundancy, so the same \(\bar\alpha\) destroys proportionally less information. Reusing a 512-px schedule at higher resolution under-noises the image, the model is never asked to solve the structure-forming part of the problem, and the output looks like a tiled repetition of a smaller picture. SD3 makes the correction explicit as a resolution-dependent timestep shift²⁴.

Latent diffusion: the idea that made it affordable

Running diffusion on \(512 \times 512 \times 3 = 786{,}432\) numbers means a full U-Net pass at that resolution for every one of fifty steps — expensive enough that pixel-space systems such as Imagen and DALL·E 2 generate at 64 × 64 and then upsample in a cascade. Latent diffusion⁴ moves the entire process into the latent space of a VAE (the autoencoder of chapter 17):

The factor 48 is \(8 \times 8\) (each side shrinks 8×) times \(3/4\) (three colour channels become four latent channels). The split is the point: the autoencoder handles perceptual detail — texture, sharpness, high frequencies — and the diffusion model handles semantics and composition. Each does the part it is good at, and the expensive part runs in the small space.

The diagram below follows the figure of the latent diffusion paper⁴. Pick a path to see which parts training uses and which parts generation uses.

From here on the diffusion runs on the latent, so its variable is written \(z_t\) instead of \(x_t\): \(z_0\) is the encoded image and \(z_T\) the initial noise. Everything above holds unchanged.

The three components:

• Text encoder — CLIP (chapter 19) turns the prompt into a sequence of embeddings. Frozen.
• U-Net — denoises the latent, with cross-attention layers where the query comes from the latent and key/value from the text embeddings (chapter 11). This is the only place the prompt enters, and it is exactly the cross-attention of a Transformer decoder.
• VAE — encodes to latents during training, decodes to pixels at the end of sampling. Frozen.

Milestones

Eleven years in three lanes: the methods, the models built from them and the tools that grew around the models. Click a name to open its paper or release.

The pipeline, end to end

One image of SD 1.5 at 512 × 512 with 25 steps, stage by stage. The last step shows the path that training takes instead. The colour of a box says which space it works in; orange marks training.

One image, in shapes

The same path, with the tensor that leaves each stage, for SD 1.5 at 512 × 512 and 25 sampling steps. The parameter counts are the ones the lab prints.

Nearly all the compute is in one row: 25 steps × 2 passes = 50 U-Net evaluations per image, against one pass of the text encoder and one of the decoder. That is why the levers that make generation fast — fewer steps, distillation, a guidance-free model — all act on the U-Net row.

One image, in numbers

The same loop, line by line, with every number on screen. The latent has 4 numbers instead of 16,384 and the U-Net is a toy whose noise prediction is exact (each caption's latents are a Gaussian), so every value can be checked by hand; the schedule, the timesteps and the DDIM update are SD 1.5's. The guidance line is the subject of the next section.

Classifier-free guidance: the part that makes it obey

A model trained on (image, caption) pairs conditions on text, and if you sample from it honestly the prompt adherence is disappointing. Every text-to-image system you have used therefore does something that is not sampling from its own distribution.

Train the model with the caption dropped 10% of the time, so one network learns both the conditional noise prediction \(\epsilon_\theta(x_t, c)\) and the unconditional one \(\epsilon_\theta(x_t, \varnothing)\) (the step \(t\) is left implicit in this section). Then, at every sampling step, evaluate both and extrapolate past the conditional²⁵:

At \(w = 0\) you get the unconditional model. At \(w = 1\), plain conditional sampling. Past that you are amplifying the direction the prompt points in — which is not sampling from any distribution the model learned, and works anyway. In Stable Diffusion, "no prompt" \(\varnothing\) is the embedding of the empty string, not a vector of zeros, and the guidance_scale of the usual pipelines (7.5 by default) is this \(w\).

The panel samples a toy distribution (eight clusters on a ring, three of which the "prompt" asks for) with exact scores (the score is the gradient of the log-density, the noise prediction up to a scale: see SDE below), written in the velocity form of chapter 21; guidance combines the two predictions with the same line. It sweeps \(w\) and measures the three things that trade against each other. With 32 steps, prompt adherence is already 99% at \(w = 1\), because this toy's conditional is exact; a learned one is not, and that gap is what real pipelines pay \(w > 1\) to close. Diversity falls monotonically from the very first step past 1: the within-mode spread (the mean distance of a sample to the centre of its cluster, in units of σ, the standard deviation of a cluster) goes from 1.03σ at \(w = 1\) to 0.76σ at \(w = 3\) and 0.63σ at \(w = 7.5\). And the off-manifold share, 1% at \(w = 3\), climbs to 17% at \(w = 10\) and 63% at \(w = 15\) — which in a real image is the over-saturated, burnt, high-contrast look everyone recognizes from a guidance scale of 20.

Go and try: Diffusion Explainer

Diffusion Explainer¹⁵ is an interactive visualization of this pipeline, built for teaching by the Polo Club at Georgia Tech. It does not generate anything while you watch: the images were computed in advance for a set of preset prompts, each paired with a variant, at a few seeds and guidance scales. That is what makes it instant, and also its limit — you cannot type your own prompt; the lab is for that. Give it fifteen minutes.

1. Map its boxes onto this chapter. The tool names three stages. Before opening them, match each to a component of latent diffusion:

   Diffusion Explainer	This chapter	Output (SD 1.x)
   Text Representation Generator	tokenizer + CLIP text encoder	\(77 \times 768\)
   Image Representation Refiner	U-Net + scheduler, repeated over the timesteps, in the latent space	\(4 \times 64 \times 64\)
   Image Upscaler	VAE decoder	\(3 \times 512 \times 512\)

   Expand the first and find the number 77: what happens to a prompt shorter than that, and what would happen to one longer? Expand the second and find the two noise predictions it combines — that is the guidance line of the previous section.

2. Drag the timestep slider. It runs over the 50 steps of sampling. Before pressing play, predict: does the layout settle first or the texture? Then find the earliest step at which you can name the object, and the step after which only detail changes. The pictures it shows for each step are a cheap linear projection of the 4-channel latent to colour, not a pass of the VAE decoder; the lab measures how far a linear map gets.

3. Change the guidance scale. It offers 0, 1, 7 and 20. Predict each one from the formula before clicking: 0 is the unconditional model — which part of the prompt survives? — 1 is plain conditional sampling, 7 is close to the usual default, 20 is far past it. Where do you see the adherence of the guidance panel above, and where its off-manifold look?

4. Change the seed. The seed fixes the initial noise \(z_T\) and nothing else. Which boxes of the diagram change, and which stay the same (look at the text representation)? Does a new seed change the subject, or the composition?

5. Compare two prompts that differ by one modifier. Use the comparison mode with a preset pair whose prompts differ by a single phrase, such as a style or an adjective, and keep the seed and the guidance fixed. At which timestep do the two pictures start to differ, and how much of the composition survives to the end? Explain it with the pipeline: the initial noise is identical and the prompt enters only through cross-attention.

What it leaves out is as instructive: the number of steps, the scheduler and the resolution are fixed, and training does not appear at all — it shows the inference path only.

Samplers: the same model, five to fifty steps

Training gives you \(\epsilon_\theta\). Turning it into an image is a separate numerical problem: integrate a reverse ODE or SDE, and you may choose the solver. This is why the same checkpoint runs at 50 steps or 20 with almost no visible difference.

Distillation: from fifty steps to one

A separate line of work compresses the sampling trajectory into the weights.

Note what the ADD row implies: the GAN, displaced as a generator, came back as the thing that makes diffusion real-time (StreamDiffusion³¹ builds an interactive pipeline on such models). And note the honest cost — few-step models trade diversity for speed, which you will see immediately if you sample the same prompt twenty times.

Control: everything you actually do with it

Text is a poor interface for "put the subject here, in this pose". Five mechanisms cover almost every real workflow:

What changed after SD 1.5

The milestone chart above says when; it does not say what changed. Here is what each generation actually changed, which is more useful than the version numbers:

Four themes, and none of them is "more parameters":

1. The U-Net became a Transformer. Better scaling, and one architecture shared with everything else in the course.
2. Better text encoding. T5 alongside CLIP, because CLIP's text tower is small and behaves like a bag of words (chapter 19) — one reason SD 1.5 could not spell or handle long compositional prompts.
3. A richer latent. 4 → 16 channels: less lost in compression, which, together with T5, is where legible text in generated images came from.
4. Diffusion became flow matching. Straighter paths, fewer steps, simpler objective. That is the next chapter.

Lab: Stable Diffusion, opened up

The pipeline of this page is taken apart in a Colab notebook that runs Stable Diffusion 1.5 with Hugging Face diffusers¹⁰, one component at a time. It needs a GPU: in Colab, Runtime → Change runtime type → T4 GPU (the free tier is enough; the first cells download about 2 GB of weights, and the reference run of every cell took 11 minutes on an Apple M2 laptop GPU, slower than a T4). Eight sections, each ending in a checkpoint with a check value:

1. the three components and their parameter counts;
2. the VAE round trip: the 48× compression, measured, and what a linear map from latent to colour gets;
3. the text encoder: 77 tokens, a \(77 \times 768\) matrix, and what the empty prompt looks like;
4. the forward process on a latent, and the terminal SNR of SD 1.5, read off the scheduler;
5. the denoising loop written out — scheduler, two U-Net calls and the guidance line — checked against the library pipeline, with \(\hat x_0\) decoded at every step;
6. guidance scale and seed, measured: prompt adherence, saturation and diversity;
7. steps and distillation: the same model at 1 to 25 steps, and the same model with an LCM-LoRA at 4;
8. img2img written by hand, with a table to fill in.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Beyond the lab: the Hugging Face Diffusion Course¹¹ goes further with diffusers; shorter tutorials run SD from Python⁹, call SD3 through its API¹⁷ and survey the SD versions¹⁸. Illustrated walk-throughs of DDPM¹²⁸, of the U-Net¹⁴ and of Stable Diffusion⁶ cover the same ground with other figures.

Key takeaways

1. Diffusion turns generation into many easy denoising problems, and the closed form \(x_t = \sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\epsilon\) is what makes training parallel.
2. The loss is an ordinary MSE against noise you drew yourself. No adversary, stable training, and the main reason diffusion beat GANs.
3. The noise schedule decides what the model learns. Non-zero terminal SNR is why SD 1.x cannot render black; resolution changes what a noise level means, hence timestep shifting.
4. Latent diffusion runs the process in a 48×-smaller space. The VAE handles detail, the diffusion model handles semantics.
5. Classifier-free guidance is one line, doubles inference cost, and is what makes the model obey the prompt. It trades diversity for adherence, and too much of it pushes samples off-manifold.
6. The sampler is a separate choice from the model. DPM-Solver++ at 15–25 steps is the practical default; below ~10 steps you need distillation, not a better solver.
7. Distillation reaches 1–4 steps, and one of the winning methods uses an adversarial loss — the GAN, returning as a component.
8. ControlNet, IP-Adapter and LoRA all freeze the base and add a small path beside it. That is why they compose.
9. Since SD 1.5: Transformer backbone, better text encoders, 16-channel latents, rectified flow. Not "bigger".

Additional

DDPM vs DDIM

SDE

Song et al.⁷ take the number of steps to infinity. The forward chain becomes a stochastic differential equation (SDE) in continuous time \(t \in [0, T]\): \(f(x, t)\) is a drift, \(g(t)\) scales the noise and \(dw\) is an increment of Brownian motion, the continuous version of adding a little Gaussian noise. Call \(p_t\) the density of the noised data at time \(t\). Every forward SDE has a reverse one that runs from noise back to data (\(\bar w\) is Brownian motion running backwards in time), and its only unknown is the score \(\nabla_x \log p_t(x)\), the direction in which \(p_t\) grows fastest. A network trained like DDPM gives it, because the noise prediction is the score up to a scale:

DDPM's sampler is one discretization of the reverse SDE. DDIM is one of its deterministic twin, the probability-flow ODE \(dx = \big[f(x, t) - \tfrac12 g(t)^2 \nabla_x \log p_t(x)\big]\,dt\), which has no noise term and reaches the same distributions \(p_t\).

U-Net Architecture

U-Net is a convolutional neural network architecture originally designed for biomedical image segmentation¹³. It has since been widely adopted in image generation, including diffusion models like Stable Diffusion. Its shape is a U: an encoder (the contracting path) halves the resolution and adds channels level by level, a decoder (the expanding path) mirrors it, and skip connections carry each level of the encoder across to the decoder level of the same resolution, so the fine detail lost on the way down is available on the way up. Stable Diffusion's version adds the timestep and the text.

Below, the U-Net of Stable Diffusion 1.5, block by block. Every "Transformer" is self-attention, then cross-attention to the text, then a feed-forward layer. The timestep embedding is added inside every ResBlock. Channels and resolution are those of the diffusers configuration the lab loads. An up block has three ResBlocks, one per skip tensor that arrives at its resolution.

Videos