Artificial Neural Networks and Deep Learning · Insper

Stable Diffusion

Four ideas, stacked

Four ideas, stacked

  1. A diffusion process: generation as a long sequence of easy denoising problems.
  2. A latent space, from a VAE: the sequence runs in a space 48× smaller than pixels.
  3. A text encoder, from CLIP: the process can be told what to make.
  4. Classifier-free guidance: it actually listens.

Most explanations stop after the third. What goes wrong if you sample a text-conditioned model honestly, without the fourth?

Two chains over the same frames

The two chains of a diffusion model over the same four frames: x0, x t−1, x t and x T. Forward (top), a fixed process adds noise step by step, and a single jump reaches x t from x0. Between the rows, the true reverse step q(x t−1 | x t) is crossed out: it is intractable. Reverse (bottom), a U-Net predicts the noise and takes x t back to x t−1; sampling repeats that step T times from fresh noise. FORWARD · FIXED · ADDS NOISE REVERSE · LEARNED · REMOVES NOISE ⋯ q(xt​ | xt−1​) ⋯ one jump: xt​ = √ᾱt​ x0​ + √(1 − ᾱt) ε q(xt−1​ | xt) is unknown: it needs every image ⋯ ⋯ sampling: start from fresh noise and apply the learned step T times x0​ xt−1​ xt​ xT​ x0​ xt−1​ xt​ xT​ pθ(xt−1​ | xt)U-Netεθ(xt​, t)

Forward diffusion: noise in one jump

Training needs at a random between 1 and 1000. Do we have to run the chain times?

  • No: a composition of Gaussians is Gaussian.

  • Draw , draw , jump. Each example in a batch can sit at a different noise level.
  • The forward process is fixed: nothing in it is learned, and inference never uses it.

: the clean image · : the same image after noise steps, out of · : how much noise step adds (the schedule) · : Gaussian over with mean and covariance · : the identity, independent noise per pixel · : standard Gaussian noise.

The loss is one line

  • The target is noise you drew yourself; is the text.
  • Ordinary supervised regression: no adversary, stable training, a loss that means something.
  • That, more than image quality, is why diffusion displaced the GAN.

, and predictions are interchangeable given and the schedule. They differ in how the loss weighs the noise levels.

: the network (the U-Net, with weights ) · : average over drawn images, noises and · : a mix of the two targets.

The way back: from noise prediction to a step

The network outputs . How do you get its current guess of the clean image?

  • Invert the closed form:

  • Near , : any error in is amplified. Hence and velocity objectives.
  • The scheduler (the rule that turns the prediction into a step) moves a little towards . Repeat.

One network, two loops

Training and sampling share one U-Net. Left, training: a batch in which every image draws its own t, one jump to x t in closed form, and a loss between the drawn noise and the predicted one that updates the weights. Right, sampling: start from noise x T, take one step back with the predicted noise, and repeat T times until x0. TRAINING · ONE PASS PER IMAGE SAMPLING · T PASSES, IN ORDER ε εθ​ update θ εθ​ × T t = 1 done A batch: each image draws its own t t = 14 t = 350 t = 721 Jump to that noise level=√ᾱt​+√(1−ᾱt)xt​x0​ε Loss: ‖ε − εθ(xt​, t)‖²one gradient step on θ, then the next batch U-Net εθ(xt​, t)θ Start: xT​ ~ 𝒩(0, I), t = Tnoise drawn from the seed One step backxt−1​ = μθ(xt​, t) + σt​ zμθ​ removes a scaled share of εθ​σt​ z: fresh noise in DDPM; 0 in DDIM After the last step: x0​a new image

The noise schedule is not a detail

SD 1.x: "scaled-linear" ( on a straight line) from 0.00085 to 0.012, steps.

How much of the image survives at the very last training step, ?

  • , so of the image is still there — not zero.
  • Sampling starts from pure noise: a train/test mismatch.
  • Consequence: SD 1.x cannot make a truly black or truly white image. The mean brightness leaks in.
  • The fix: rescale the schedule to zero terminal SNR (Lin et al., 2024).

SNR (signal-to-noise ratio) , the variance of the signal over that of the noise in . Terminal: at . In SD 1.x, . The lab reads from the scheduler, which counts 0 to 999: .

Simulator · the signal-to-noise ratio along t

Latent diffusion: where the 48 comes from

pixels SD 1.5 latent
shape (height × width × channels) 512 × 512 × 3 64 × 64 × 4
numbers 512 · 512 · 3 = 786,432 64 · 64 · 4 = 16,384

786,432 / 16,384 = 48. Where does the factor 48 come from?

512 × 512 × 3image8 × 8 pixels × 3 colours= 192 numbersVAE encoder192 / 4 = 484 channels= 4 numbers64 × 64 × 4latent
  • Each side shrinks 8×: one latent cell sums up 8 × 8 pixels × 3 colours = 192 numbers in 4. .
  • Paid at every sampling step. The VAE costs one encode or one decode.

The VAE round trip, measured

Lab, section 2: one 512 × 512 photo, encode, decode.
latent 1 × 4 × 64 × 64
std, raw latent 4.12
std × 0.18215 0.75
reconstruction PSNR 27.7 dB
linear latent → RGB, R² 0.92
  • The scaling factor 0.18215 brings the latent to unit variance over the training set, the scale of the noise. For this photo: 4.12 × 0.18215 = 0.75.
  • The decoder puts back the fine detail; the latent keeps layout and colour.
  • A linear map from 4 channels to RGB already gives a recognizable preview — the trick behind the step previews of the Diffusion Explainer.

Shape: (images, channels, height, width). PSNR = 10 log₁₀(peak² / mean squared error), in dB: higher is more faithful. R²: the share of the variance of the photo's colours, downscaled to 64 × 64, that the linear map explains.

Latent diffusion, end to end

Latent diffusion in three spaces. Pixels: the image x, the VAE encoder E and decoder D, and the generated image x̃. Latent, 64 × 64 × 4: in training, z0 from the encoder, noise added at a random t, z t into the U-Net and the loss; in generation, z T from noise, the U-Net and the scheduler repeated T times, then z0 to the decoder. Text: the prompt, the CLIP text encoder and the embeddings c, which reach the U-Net through cross-attention. PIXELS · 512 × 512 × 3 LATENT · 64 × 64 × 4 TEXT · 77 TOKENS zt​, t εθ​ × T K, V xa photo ℰencoder(frozen) 𝒟decoder(frozen) x̃the image z0​ add noiserandom t, one jump zt​ loss‖ε − εθ‖² U-Net εθ(zt​, t, c)cross-attention:Q from zt​; K, V from c zT​𝒩(0, I) schedulerzt​ → zt−1​ z0​ “a cat in space”prompt text encoderCLIP · frozen c · 77 × 768

Three components, one of which does the work

component role parameters calls per image
CLIP text encoder prompt → (tokens × numbers) 123.1 M 2 (prompt and empty prompt)
U-Net predicts in the latent 859.5 M 50 (25 steps × 2)
VAE decoder latent → pixels 49.5 M 1
  • The text encoder and the VAE are frozen; only the U-Net was trained for diffusion.
  • Every speed-up you will see acts on the U-Net row: fewer steps, distillation, no guidance.

M = millions of parameters, counted in section 1 of the lab. The whole VAE has 83.7 M, but only the decoder (49.5 M) runs at generation. The U-Net is 81% of the total: 859.5 / (123.1 + 859.5 + 83.7).

The prompt, as the U-Net sees it

  • The tokenizer pads or truncates every prompt to 77 tokens: <start>, the words, <end>, then padding.
  • The text encoder returns one vector per token: (768 is the width of the CLIP ViT-L/14 text encoder).
  • "No prompt" is the encoding of the empty string — not zeros.
cosine between the whole matrices, flattened (lab, section 3)
"a lighthouse on a cliff at sunset" vs "…, watercolor painting" 0.78
"a lighthouse on a cliff at sunset" vs "a bowl of ramen on a wooden table" 0.24
"a lighthouse on a cliff at sunset" vs "" 0.20

The only door: cross-attention

  • from the latent features, at each resolution of the U-Net: 64², 32², 16² and 8² (the side halves per level).
  • from the 77 text vectors. : the dimension of each query and key.
  • The image asks, the text answers: the cross-attention of a Transformer decoder.

Two prompts that differ by one word, same seed. What is identical in the two runs, and what differs?

  • Identical: , the U-Net weights, the scheduler.
  • Different: only and .

The sampling loop, written out

latents = randn(1, 4, 64, 64, generator=seed)        # z_T: the seed decides it
emb = cat([encode(""), encode(prompt)])               # ∅ and c
for t in scheduler.timesteps:                         # 25 steps
    eps_u, eps_c = unet(cat([latents] * 2), t, emb).chunk(2)
    eps = eps_u + w * (eps_c - eps_u)                 # guidance: one line
    latents = scheduler.step(eps, t, latents).prev_sample
image = vae.decode(latents / 0.18215)
  • The lab writes this loop and checks it against the library pipeline: largest pixel difference 1 (out of 255).
  • 50 U-Net calls (25 steps × 2); 1.41 s per step on an Apple M2 (a T4 is faster).

chunk(2): splits the batch of 2 into the two predictions (∅ and c) · prev_sample: the latent of the next step · / 0.18215: undoes the VAE's scaling factor.

Simulator · one image, in numbers

iteration 







  
  
  










Classifier-free guidance


Train with the caption dropped 10% of the time: one network learns  and .




What do  and  give? And the pipelines' default, ?




    

  •  is the pipelines' guidance_scale. : the unconditional model, which ignores the prompt. : plain conditional sampling.
    • 

  •  extrapolates past the conditional: no distribution the model learned. It works anyway.
    • 

  • Cost: two U-Net passes per step, for any  other than 0 or 1.
    • 





: the U-Net's noise prediction with the prompt  (the step  left implicit) · : the empty prompt.








Simulator · guidance on a toy distribution






  
  












Guidance on the real model, measured


Lab, section 6: "a red fox sitting in a snowy forest, photograph", seeds 0, 1, 2, 20 DDIM steps.







CLIP score (prompt)
saturation
diversity across seeds






0
13.2
0.20
0.259




1
33.0
0.20
0.184




3
34.7
0.22
0.064




7.5
33.4
0.31
0.057




15
33.9
0.40
0.041






    

  • Adherence rises fast and then flattens; saturation keeps climbing.
    • 

  • Diversity across seeds falls at every step, 0.259 → 0.041: at large  the three seeds draw the same centred fox.
    • 



CLIP score: 100 × cosine between CLIP ViT-B/32 embeddings of the image and the prompt. Diversity: mean pairwise cosine distance between the CLIP embeddings of the three seeds. Saturation: mean HSV saturation.






Seed and prompt: who decides what


Lab, section 6: "a lighthouse on a cliff at sunset" ± ", watercolor painting", DDIM, .






pair
correlation between the two images, grey and downscaled to 64 × 64 (the layout)






same prompt, seeds 0 and 1
0.15




same seed 0, with and without the modifier
0.85






    

  • The seed fixes , and with it most of the layout, decided in the first steps.
    • 

  • The modifier changes style and colour through cross-attention.
    • 

  • The two trajectories'  already differ at step 1 (), yet the layouts end up correlated at 0.85.
    • 







Go and try: Diffusion Explainer






    

  1. Map its three boxes onto the chapter: text encoder → refiner (U-Net + scheduler) → decoder.
    2. 

  2. Timestep slider (50 steps): when can you name the object? When does only detail change?
    3. 

  3. Guidance 0, 1, 7, 20: predict each from the formula first.
    4. 







    

  1. Seed: which boxes change, and which do not?
    2. 

  2. Compare two prompts that differ by one modifier, same seed and guidance: when do they diverge?
    3. 



Images are precomputed: no prompt of your own — that is the lab. Lee et al., IEEE VIS 2024.






poloclub.github.io/diffusion-explainer →






Samplers: the same model, five to fifty steps






sampler
what it is
typical steps






DDPM
ancestral, stochastic
1000




DDIM
deterministic ODE; invertible, so it can edit
50




Euler / Heun
plain ODE solvers
20–30




DPM-Solver++
exponential integrator; the practical default
15–25




UniPC, DEIS
higher-order multistep
10–20






    

  • Training gives ; turning it into an image is a separate numerical problem.
    • 

  • Below ~10 steps no solver saves you: that regime needs distillation.
    • 



Ancestral: draws fresh noise at every step · ODE: a differential equation with no draws, so the seed fixes the image · higher order: uses several evaluations (or those of earlier steps) to take a more accurate step.






Distillation: from fifty steps to one






method
idea
steps






progressive distillation
a student takes two teacher steps at once; repeat
8192 → 4 (halved per round, 11 rounds)




LCM-LoRA
jump from any point of the trajectory to its end; ships as a LoRA
2–8




ADD (SDXL-Turbo)
add a discriminator: the GAN loss returns
1–4






Lab, section 7, same prompt and seed:







U-Net calls
CLIP score
time (M2)






DDIM, 1 step, 
2 (1 × 2)
13.1
2.4 s




DDIM, 4 steps, 
8 (4 × 2)
32.5
6.1 s




DDIM, 25 steps, 
50 (25 × 2)
34.1
34.1 s




LCM-LoRA, 4 steps, 
4 (4 × 1: no guidance)
33.9
4.5 s










Control: what people actually do with it






mechanism
what it does
how






img2img
start from an image
encode, noise to some , denoise; strength = which 




inpainting
redo a masked region
keep the noised original outside the mask at every step




ControlNet
pose, depth, edges
trainable copy of the encoder half, joined by zero convolutions




IP-Adapter
a reference image
extra cross-attention reading CLIP image embeddings




LoRA
a style, a character
low-rank update of the attention layers (the transfer-learning class)








All of them freeze the base and add a small path beside it. ControlNet and LoRA start it at zero, so training starts exactly at the base model. That is why they compose.








What changed after SD 1.5







SD 1.5 (2022)
SDXL (2023)
SD3 / FLUX (2024)






backbone
U-Net, 860M
U-Net, 2.6B + refiner
MMDiT (multimodal diffusion Transformer)




text encoder
CLIP ViT-L
two CLIP
T5-XXL + CLIP




latent
4 channels
4 channels
16 channels




objective
, DDPM

rectified flow




native size
512²
1024²
1024²+, shifted schedule






    

  • Four deep changes (Transformer, T5, 16 channels, flow); none of them is "more parameters".
    • 

  • Rectified flow is the next class; the Transformer backbone, the one after.
    • 







Where the training data came from




SD 1.x was trained on subsets of LAION-5B: image–URL pairs scraped from Common Crawl, filtered by CLIP similarity. Nobody whose work is in it was asked.




    

  • Litigation over training data, still open.
    • 

  • A named living artist's style, reproduced on request.
    • 

  • Documented memorization of individual training images.
    • 

  • Illegal material found in an audit, which took the dataset offline.
    • 





Provenance is an engineering property of your system, not a footnote.








Lab · Stable Diffusion, opened up


Colab notebook, ~2 h, T4 GPU (free tier), Stable Diffusion 1.5 with diffusers. Each section ends in a checkpoint.






    

  1. three components, counted
    2. 

  2. the VAE round trip and the 48×
    3. 

  3. the text encoder: 77 × 768
    4. 

  4. the forward process and the terminal SNR
    5. 







    

  1. the sampling loop, by hand, against the pipeline
    2. 

  2. guidance and seed, measured
    3. 

  3. steps, and an LCM-LoRA at 4 steps
    4. 

  4. img2img by hand: the table is yours
    5. 







open in Colab →






Takeaways


    

  1. Diffusion = many easy denoising problems; the closed form makes training parallel.
    2. 

  2. The loss is an MSE against noise you drew: stable, and the reason diffusion beat GANs.
    3. 

  3. The schedule decides what is learned:  is why SD 1.x cannot render black.
    4. 

  4. Latent diffusion: 48× fewer numbers per step; the VAE handles detail.
    5. 

  5. Guidance: one line, two U-Net passes, adherence bought with diversity.
    6. 

  6. The sampler is a separate choice; below ~10 steps, distillation (and the GAN returns).
    7. 

  7. ControlNet, IP-Adapter, LoRA: frozen base + small path. Since SD 1.5: Transformer, T5, 16 channels, flow.
    8. 






Outline: (1) diffusion: forward, loss, reverse; (2) the noise schedule; (3) latent diffusion and the
three components; (4) classifier-free guidance; (5) Diffusion Explainer; (6) samplers, distillation,
control; (7) what changed after SD 1.5. Closes with the Colab lab.
Answer at the guidance slides: prompt adherence is disappointing; w = 7.5 is not "honest" sampling.
Step through the six stages: the two directions, one forward step, the jump in closed form (the next slide), the true reverse step, which is intractable, the U-Net's guess and sampling. Hover a frame to see its arrows. The reverse row ends at a different picture: sampling makes a new image.
The lab decodes x̂0 at every step; the Diffusion Explainer shows the same progression.
Training (left): a batch where each image has its own t, the jump, the loss; one pass per image, all in parallel. Sampling (right): start from noise, one step back, repeat T times in a row. Same U-Net, same weights.
Start on the scaled-linear schedule: red dot at t = 1. Tick "force zero terminal SNR". Then raise
the resolution to 1024: the curve moves up, the image is under-noised. Raise the shift to see SD3's
correction (the dashed line is the unshifted curve at the same resolution).
Click Training, then Generation. Training uses the encoder and never the decoder; generation uses the decoder and never the encoder. Both use the U-Net and the text encoder. The next slide counts the parameters.
Numbers printed by section 1 of the lab.
The loop of the previous slide on a latent of 4 numbers, with an exact toy U-Net (each caption's latents
are a Gaussian) and SD 1.5's real schedule. Default: "a red apple", seed 3, w = 7.5, 25 steps. Step with
Next: 5 real tokens + 5 padding; ‖c‖ = 1.18, ‖∅‖ = 1.11 (not zeros); t = 961 … 1, ᾱ_961 = 0.00728.
Iteration 1: ε_∅ and ε_c almost equal (π ≈ 0.25 each), ε̃₁ = 1.2481, x̂₀₁ = −5.36 (÷ 0.0853: errors × 11.7).
Trajectory: x̂₀ converges, 10.5 → 0.23 → 0.008. Then change one control at a time: w = 1 (an honest sample,
0.41 from μ_c), w = 0 (seed 3 draws a snowy field), w = 15 (0.98 past μ_c, 64 of 192 values clipped),
N = 1 (t = 1: the noise itself), N = 4 (starts at t = 751, ends 0.18 from the 25-step result).
Exact scores, so w = 1 is already 99% on-prompt here; a learned conditional is not. With 32 steps:
spread 1.03σ at w = 1, 0.76σ at w = 3, 0.63σ at w = 7.5; off-manifold 1% at w = 3, 17% at w = 10,
63% at w = 15.