Interview preparation Neural networks Deep learning

Deep Learning Interviews

The 50 neural-network concepts that come up most in internship and job interviews, from the perceptron to diffusion: what to study, how to study it and how to answer. With short answers linked to the section of the class that teaches each one, a quiz, a checklist of your progress and references to go further.

The curve that rises at epoch 8

A scene that will happen to you.

Tuesday, 10 a.m. Second round of the interview for the computer-vision internship you wanted most. The interviewer shares the screen: two plots from an image classifier someone on the team trained. The training loss falls from start to finish. The validation loss falls until epoch 8, stops and starts to rise; by epoch 30 it is above where it began. He asks:

"What would you do?"
He does not want one word. He wants to see how you think.

There is a weak answer ("it's overfitting, add dropout") and a strong one, which starts with a question back: "was validation measured in model.eval()? How many images are in each set, and does the split keep them apart?". A validation curve that is wrong because dropout was on, or because the sets leak into each other, leads to a wrong diagnosis however right it sounds.

The difference between the two answers is not memorizing more layer names. It is knowing how to connect what you studied: the two curves, bias and variance, early stopping, the regularizers from cheapest to most expensive and the golden rule of the split. This page lays out that path. At the end we come back to epoch 8 with the full answer.

What a deep learning interview looks like

Three kinds of question, three skills and the same logic: reasoning out loud with shapes, curves and numbers.

Conceptual

"Why divide by √d_k?"

Checks that you explain in your own words, give the intuition and know where the idea breaks. There is almost always a small calculation behind it: a variance, a shape, a parameter count.

Case

"The loss went NaN"

Checks diagnosis and next steps. There is no single answer: there is a line of reasoning that rules out the cheap causes first and proposes one experiment per hypothesis.

Code

"Write the training loop"

Checks that you write a PyTorch loop without the classic bugs: zero_grad, model.eval(), torch.no_grad(), a softmax applied twice, shapes that do not match.

What separates candidates

For an entry-level role, hardly anyone expects you to know this week's architecture. What sinks candidates are the basic traps: not knowing the shape of the tensors, mixing up training and validation loss, evaluating with dropout on, choosing hyperparameters by looking at the test set and reporting accuracy on an imbalanced problem. Avoid those five and you are already ahead of most.

What the interviewer assesses

Shapes and arithmetic Reading training curves Debugging a run Choosing the architecture Evaluating without leaks Stating limits

Aspect	Conceptual	Case	Code
Right answer	An explanation, with the math behind it	Several valid answers	Code that runs and actually trains
Focus	Intuition and where it breaks	Diagnosis and trade-offs	Shapes and the forward–backward–step cycle
Typical length	1–3 min per question	10–20 min	15–30 min
Most common mistake	Reciting the definition without the trap	Switching architectures before looking at the data and the curves	Forgetting `zero_grad` or `model.eval()`
Where to practise	Key concepts	Case question	The training loop

How to answer: 8 principles

They apply to all three kinds and are what turns a correct answer into a good one.

1

Ask for the context before answering How many examples? What kind of data (tables, images, text)? Which metric, and what does each error cost? How was validation measured? A well-placed question shows more maturity than a perfect definition. It is what saved the epoch-8 answer.

2

Explain it in one sentence, in your own words "Dropout switches units off at random during training so the network cannot rely on any one of them." Short, no needless jargon. If you need three sentences for the core idea, you do not own it yet.

3

Draw The two loss curves, a network with the shape of every tensor, the attention matrix with its mask, the gradient's path through a residual connection. A ten-second drawing organizes your reasoning and the listener's.

4

Give a concrete example Ideally one you saw yourself: "in a 30-layer ReLU MLP with Xavier initialization, the signal reaches the last layer about 10⁻⁹ times smaller; with He it stays near 1". Lived examples are proof you did not memorize.

5

Name the trap Every good answer ends with "…but careful: …". BatchNorm computes differently in training and inference; a softmax before CrossEntropyLoss is applied twice; a perfect validation score is almost always a leak. This is where the interviewer sees depth.

6

Speak in numbers "448 parameters", "a 28×28 output", "initial loss near ln 10 ≈ 2.30 with ten classes", "the score matrix grows 4× when the sequence doubles". Numbers make your reasoning checkable and show you think like someone who has measured.

7

Always compare with something A number alone says nothing. Compare with a baseline (the majority class, a logistic regression, a frozen pretrained model), with the smaller network, with the spread across seeds. "0.81 against 0.74 for the logistic" is an answer; "0.81" is not.

8

Lead: propose the next step Do not wait for the next question. "The next step would be to check that the network can memorize a batch of 10 examples" or "I would divide the learning rate by 10 and look at the gradient norm". It shows you know what to do next, not just what things are.

Practical tip: make transitions explicit. When you move from diagnosis to solution, say it out loud: "so the problem looks like variance; I'll propose three ways to reduce it, from cheapest to most expensive". The interviewer follows your structure and you do not get lost.

The training loop from memory

The most common coding question is also the shortest: "write the training of a network in PyTorch". Know how to write this without looking anything up and say what every line does.

model = nn.Sequential(nn.Linear(d, 64), nn.ReLU(), nn.Linear(64, n_classes))
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
loss_fn = nn.CrossEntropyLoss()            # takes raw logits: no softmax in the model

for epoch in range(epochs):
    model.train()                          # dropout and BatchNorm in training mode
    for xb, yb in train_loader:            # xb: (B, d) · yb: (B,) with class indices
        opt.zero_grad()                    # without it the gradients accumulate
        loss = loss_fn(model(xb), yb)      # logits: (B, n_classes)
        loss.backward()
        opt.step()

    model.eval()                           # dropout off, BatchNorm uses its running statistics
    with torch.no_grad():                  # no graph: faster and no extra memory
        val_loss = sum(loss_fn(model(xb), yb).item() * len(yb)
                       for xb, yb in val_loader) / len(val_loader.dataset)

The bugs the interviewer looks for

Five lines that win the job

Forgetting zero_grad (gradients add up across steps); evaluating without model.eval() (dropout on during validation); applying softmax before CrossEntropyLoss; using MSE on the class index; mixing up dimensions, such as passing (B, 1) where the loss expects (B,).

Shape questions

Write the shape of every tensor

An nn.Linear(d, 64) takes (B, d) and returns (B, 64) with a (64, d) weight matrix. An nn.Conv2d(3, 16, 3) takes (B, 3, 32, 32) to (B, 16, 30, 30). Attention takes (B, n, d) to scores of (B, h, n, n). Whoever knows the shapes solves half the bugs by looking.

A case question, step by step

"How would you train a network to…": the most common open question. The answer is the flow of any project with neural networks.

1

Frame the problem and measure a baseline What is an input and what is the target? Classification or regression? Which metric reflects the cost of the errors? Before the network, measure what a trivial model (the majority class, the mean) and a simple one (logistic regression, gradient boosting on tables, a frozen pretrained model on images) already reach. Supporting material →

2

Split the data and lock the test set away Train, validation and test before any decision, splitting by patient, user or time when the examples are not independent. Scaling and encoding fit on training only. Supporting material →

3

Start with the smallest network that can work An MLP for tables, a small CNN or a pretrained model for images, a pretrained Transformer for text. Check the initial loss (near ln C with C classes) and whether the network can memorize a small batch: if it cannot, the problem is a bug, not a lack of capacity. Supporting material →

4

Train while watching both curves Adam at 1e-3 as a first attempt; tune the learning rate before any other hyperparameter. Track training and validation loss and keep the weights of the best epoch. Supporting material →

5

Regularize, tune on validation, test once If validation drifts away from training: weight decay, dropout, augmentation, early stopping, more data. Every choice on validation. Only at the end, one measurement on test, with an error analysis by subgroup and what the model does not support. Supporting material →

The scoping question

Always ask: "do I really need a network, or a network trained from scratch?" On small tables, gradient boosting usually wins. On images and text, starting from a pretrained model almost always beats training from scratch. Saying so before drawing the architecture shows you think about the problem, not the tool.

Quick diagnosis

The symptoms that show up in case questions, what they usually point to and what to do.

In case questions the interviewer describes a symptom and expects a diagnosis with the next step. Before proposing another architecture, rule out the cheap causes: a bug in the data pipeline, the learning rate, the evaluation mode, a leak between the sets.

Symptom	Likely diagnosis	Next step
Loss turns NaN or inf	Learning rate above the stability bound, exploding gradient, log(0) in the loss	Divide the rate by 10, gradient clipping, use the loss that takes logits
Loss flat from step 0	Rate far too small, dead ReLUs, broken data or labels	Rate ×10; check the network memorizes 10 examples; look at one batch by hand
Initial loss far from ln C	Wrong initialization or input scale, softmax applied twice	Check the input scale, the initialization and the loss
Train falls, validation rises	Overfitting (variance)	Early stopping, weight decay, dropout, augmentation, more data
Train and validation both high and close	Underfitting (bias)	More capacity, longer training, less regularization
Validation better than training	Dropout or augmentation only in training, or a leak between sets	Check `model.eval()` and how the split was built
Deep sigmoid network does not learn	Vanishing gradient: each layer multiplies by at most 0.25	ReLU, He initialization, residual connections, normalization
Many units always at zero	ReLUs killed by a high rate or a negative bias	Lower rate, LeakyReLU or GELU, check the initialization
Loss jitters violently	Batch too small or rate too high	Larger batch, momentum, a learning-rate schedule
Fine at first, diverges at epoch k	Rate too high for the sharper region training reached	Learning-rate schedule and warmup
Perfect validation score	A leak. Almost always a leak	Audit the split and the features

A full example

"The loss went NaN at epoch 3"

Ask: which optimizer and which rate? Is there a log or a division in the loss? Mixed precision?

Rule out the cheap: any NaN in the input data? Does the loss use BCEWithLogitsLoss, or a sigmoid and a log applied separately?

Diagnose: if the gradient norm shoots up before the NaN, the step is numerically unstable.

Propose, cheapest first: rate ÷10; gradient clipping; warmup; check the initialization and the input scale.

Tools by symptom

What to use when

The two loss curves: bias, variance and the best epoch.

Memorizing a small batch: tells a bug from a lack of capacity.

Gradient norm per layer: vanishing or exploding gradients.

Activation histogram: dead or saturated units.

Learning-rate sweep over powers of 10: the hyperparameter search that always pays off.

The 50 key concepts

The 50 topics in eight themes. Try to answer out loud before opening each theme. The numbers identify each topic in the checklist; the link at the end of each answer goes to the section of the class that teaches it.

[1]What can a perceptron not learn?

A perceptron draws a hyperplane: \(\hat y = \text{sign}(w \cdot x + b)\). It converges when the classes are linearly separable and never converges when they are not, as in XOR. Trap: the limit is the line's, not the training's: a hidden layer or the feature \(x_1 x_2\) solves it. ↗ Perceptron

[2]Why does a network need non-linear activations?

Without them, stacked linear layers collapse into one linear map: \(W_2(W_1x) = (W_2W_1)x\). ReLU is the default because it is cheap and does not saturate for positive inputs. Trap: a ReLU that only ever receives negative inputs has zero gradient and stops learning (a dead ReLU); sigmoid and tanh saturate at the ends. ↗ MLP

[3]What can one hidden layer already do?

With enough units, approximate any continuous function on a bounded domain (the universal approximation theorem). Trap: the theorem says the network exists, not that training finds it nor how many units it takes; deep networks represent the same function with far fewer units. ↗ MLP

[4]Explain backpropagation.

It is the chain rule applied from the loss backwards, layer by layer, reusing what the forward pass stored: a layer's error is the next layer's error multiplied by the transposed weights and by the activation's derivative. It costs about one forward pass. Trap: backpropagation computes gradients; the optimizer is what updates the weights. ↗ MLP

[5]Which output and which loss for each task?

Regression: one linear unit and MSE (or MAE/Huber). Binary: one logit and binary cross-entropy (BCEWithLogitsLoss). Multi-class: one logit per class and softmax cross-entropy (CrossEntropyLoss). Trap: PyTorch's losses already include the sigmoid or softmax; applying them before is a silent bug. ↗ MLP

[6]What is the shape of each tensor in an MLP?

With a batch of B examples of d features the input is \((B, d)\); a layer with h units has weights \((h, d)\), a bias \((h)\) and returns \((B, h)\). Parameters per layer: \(h \cdot d + h\). Trap: the batch is the first dimension almost everywhere in PyTorch; swapping the order gives a shape error or, worse, a broadcast that runs without a warning. ↗ MLP

[7]What does the learning rate do?

It scales the step against the gradient: \(w \leftarrow w - \eta \nabla L\). Too small and training crawls; too large and it oscillates or diverges. It is the hyperparameter that matters most. Trap: tune it before anything else, sweeping powers of 10. ↗ Optimization

[8]Batch, stochastic or mini-batch?

Batch uses every example per step (exact gradient, expensive); stochastic uses one (cheap, noisy); mini-batch uses tens to hundreds and is the default: it keeps the GPU busy and the noise still helps escape bad regions. Trap: changing the batch size calls for changing the learning rate with it. ↗ Optimization

[9]What is momentum?

A moving average of past gradients goes into the step, like a ball gaining mass: it speeds up along consistent directions and damps the ones that flip sign, as in a narrow valley. Trap: with momentum 0.9 the effective step is about 10× the rate; switching momentum on can make a rate that worked too large. ↗ Optimization

[10]How does Adam work? And AdamW?

Adam keeps two moving averages: of the gradient (momentum) and of the squared gradient, which gives every parameter its own step scale, with a bias correction at the start. AdamW applies weight decay directly to the weights, outside that scaling. Trap: in Adam the L2 penalty goes through the adaptive division and barely regularizes the parameters with large gradients; hence AdamW. ↗ Optimization

[11]Why schedule the learning rate? And warmup?

A high rate early moves fast; a lower rate late settles at the bottom of the valley. Cosine and steps are the common schedules. Warmup starts with a small rate and raises it over some steps, while Adam's statistics are still poor. Trap: without warmup, Transformers often diverge in the first steps. ↗ Optimization

[12]Why does initialization matter? Xavier or He?

Each layer multiplies the signal's variance by \(n\,\sigma^2\) (with n inputs); initialization picks \(\sigma^2\) to keep that factor near 1. Xavier uses \(1/n\) (symmetric activations such as tanh); He uses \(2/n\), because ReLU zeroes half the variance. Trap: with Xavier, 30 ReLU layers take the signal to about 10⁻⁹; with N(0, 1) weights, to 10³⁹. ↗ Deep networks

[13]What are vanishing and exploding gradients?

The gradient reaching the first layer is a product of L factors, one per layer; a product of many numbers goes to zero or to infinity. The sigmoid's derivative is at most 0.25, so 20 sigmoid layers shrink the gradient by an astronomical factor. Trap: this is decided before the first step, by the activation and the initialization, not by the optimizer. ↗ Deep networks

[14]Why do residual connections work?

With \(a \leftarrow a + f(a)\), each block's derivative is \(1 + \partial f/\partial a\): there is an identity path along which the gradient comes back without shrinking. That is what made networks of hundreds of layers trainable. Trap: on its own, the sum makes the signal grow at every block; residuals always come with normalization (pre-LN in Transformers). ↗ Deep networks

[15]Your run does not learn. Where do you start?

With the cheap checks: is the initial loss near \(\ln C\)? Can the network memorize 10 examples? If it cannot, it is a bug (data, labels, loss, shapes), not capacity. Then rate ×10 and ÷10. Trap: switching architectures before these tests hides the bug instead of finding it. ↗ Optimization

[16]Bias and variance in a neural network?

High bias: training and validation both poor and close (the network cannot represent the pattern). High variance: training good and validation poor, with the gap growing (the network memorizes the noise). You read it off the two loss curves. Trap: large networks with plenty of data and regularization often generalize better than small ones; "fewer parameters" is not the first remedy. ↗ Regularization

[17]L1 or L2?

L2 adds \(\lambda \sum w^2\) and shrinks all weights proportionally; L1 adds \(\lambda \sum |w|\) and pushes many weights to exactly zero, because its gradient does not fade near zero. Trap: in networks "weight decay" is the name of L2 and, with Adam, it is only equivalent in AdamW. ↗ Regularization

[18]How does dropout work?

In training, each unit is zeroed with probability p and the survivors are multiplied by \(1/(1-p)\), so the expected activation does not change; at inference nothing is dropped. It stops units from depending on each other. Trap: forgetting model.eval() leaves dropout on during evaluation. ↗ Regularization

[19]BatchNorm in training and in inference?

In training it normalizes each channel with the current batch's mean and variance and updates running averages; at inference it uses those running averages, so one example's output does not depend on the others. Trap: very small batches give noisy statistics, and the difference between the two modes becomes a bug when model.eval() is forgotten. ↗ Regularization

[20]Why do Transformers use LayerNorm?

LayerNorm normalizes each example (each token) by its own features: it does not depend on the batch, computes the same in training and inference and works with sequences of different lengths. Trap: BatchNorm would mix statistics of tokens from different sentences and of padding positions. ↗ Regularization

[21]What is early stopping?

Track the validation loss and stop when it has not improved for a few epochs (the patience), keeping the weights of the best epoch. It is the cheapest regularizer. Trap: the chosen epoch is a hyperparameter tuned on validation; the honest performance is still the test's. ↗ Regularization

[22]What is data augmentation?

Manufacturing new examples with transformations that do not change the label: crops, flips and colour changes on images, for instance. It teaches the network the invariances you know hold. Trap: the transformation has to preserve the label (flipping a "6" may not) and it is applied in training only. ↗ Regularization

[23]Train, validation and test: what is each for?

Training fits the weights; validation chooses everything that is not a weight (architecture, rate, epoch, threshold); test measures once, at the end. Trap: every decision made by looking at the test set turns it into validation and the reported score becomes optimistic. ↗ Splitting

[24]What is data leakage?

Information that would not be available at prediction time gets into training or evaluation: a column that is a consequence of the target, the same patient on both sides, the future in the training set, a scaler fit before the split. Trap: a result too good to be true is almost always a leak. ↗ Leakage

[25]Why scale a network's inputs?

Features on very different scales distort the loss surface: the learning rate that suits one is wrong for another and training gets slow or unstable. Standardizing (mean 0, standard deviation 1) is the default. Trap: the statistics come from training only; fitting the scaler on the test set is a leak. ↗ Scaling

[26]When does accuracy lie?

With imbalanced classes: with 1% fraud, always predicting "not fraud" scores 99% without learning anything. Report the prevalence, precision, recall and the PR curve. Trap: ROC-AUC can also look great on a rare problem with low precision. ↗ Classification metrics

[27]Precision, recall and the threshold?

Precision: of those flagged positive, how many were. Recall: of the actual positives, how many were found. Lowering the threshold raises recall and lowers precision. Trap: 0.5 is not sacred; the threshold comes from the cost of each kind of error and is chosen on validation. ↗ Classification metrics

[28]How do you handle class imbalance in a network?

In order: the right metric, the threshold from the costs, weighting the loss (pos_weight or focal loss) if negatives dominate the gradient, balanced batches if batches come up empty of positives and, whenever possible, more positives. Trap: resampling and weighting the loss decalibrate the probabilities. ↗ Imbalance

[29]MAE, RMSE or R²?

MAE: mean error in the target's unit, robust to outliers. RMSE: same unit, punishes large errors (always ≥ MAE). R²: compares with predicting the mean; it can be negative outside training. Trap: the loss you train (MSE or MAE) decides whether the network aims at the mean or the median. ↗ Regression metrics

[30]What does convolution buy over a dense layer?

Locality and weight sharing: a 3×3 kernel has 9 weights applied at every position, against thousands for the equivalent dense layer, which would also have to relearn the same detector at each position. Trap: it is a constraint (an inductive bias); with plenty of data, architectures with a weaker prior such as the ViT catch up. ↗ CNNs

[31]Compute a conv layer's output and parameters.

\(H_{out} = \lfloor (H_{in} + 2p - k)/s \rfloor + 1\). Parameters: \(C_{out}(C_{in} k^2 + 1)\), independent of the image size. Conv2d(3, 16, 3) has 448. Trap: most CNN bugs are shape bugs, nearly always at the first Linear after the Flatten: print the shape instead of guessing. ↗ CNNs

[32]What is the receptive field?

The region of the input that influences a unit. Each 3×3 conv with stride 1 grows it by 2 pixels; strides and pooling make it grow much faster. A network only recognizes an object if the receptive field of its last layers covers it. Trap: the effective receptive field is much smaller than the theoretical one: the centre weighs far more than the border. ↗ CNNs

[33]Equivariance, invariance and pooling?

Convolution is translation-equivariant: shifting the input shifts the feature map. What classification wants is invariance: the same class at any position, obtained gradually with pooling, strides and the global pooling at the end. Trap: stride breaks exact equivariance; shifting the image by one pixel can change the prediction. ↗ CNNs

[34]What are Q, K and V?

Three linear projections of each token. The query asks, the keys answer how well they match (the dot product becomes a weight through the softmax) and the values are what gets summed with those weights: \(\text{softmax}(QK^\top/\sqrt{d_k})\,V\). It is a dictionary with a soft lookup. Trap: the weights depend on the input, unlike a dense layer's. ↗ Q, K, V

[35]Why divide by √d_k?

With unit-variance entries, \(q \cdot k\) has variance \(d_k\); large scores make the softmax nearly one-hot and its gradient vanishes. Dividing by \(\sqrt{d_k}\) brings the variance back to 1. Trap: it is not to make the weights sum to 1; the softmax already does that. ↗ Q, K, V

[36]Causal mask and multi-head?

The causal mask sets the scores of future positions to \(-\infty\) before the softmax, so token i only sees earlier tokens when training in parallel. Multi-head splits \(d\) into h heads with their own projections, each free to look at a different relation. Trap: the padding mask is a different one, and the two are usually combined. ↗ Masks and heads

[37]Where does token position come from?

Attention alone does not see order: shuffling the tokens only shuffles the outputs. Position comes in through added positional embeddings (sinusoidal or learned) or, in current models, through RoPE, which rotates q and k according to position. Trap: learned positions do not extrapolate to sequences longer than those seen in training. ↗ Embeddings

[38]What does attention cost? What is the KV cache?

The score matrix is \(n \times n\): memory and compute in \(O(n^2)\). During generation the keys and values of earlier tokens do not change; the KV cache stores them so each new token computes only its own row. Trap: the cache trades compute for memory, and it, not the weights, is what limits long context when serving. ↗ The cost of attention

[39]Pre-LN or post-LN?

The 2017 Transformer normalized after the residual sum (post-LN); current models normalize inside the branch (\(a + f(\text{Norm}(a))\)) and leave the identity path clean for the gradient. They train deep without careful warmup. Trap: in pre-LN the residual stream grows with depth, which is why there is a final normalization before the output. ↗ What changed since 2017

[40]How does BPE tokenization work?

It starts from bytes and repeatedly merges the corpus's most frequent pair into a new token, up to the chosen vocabulary size. Common words become one token; rare words, several pieces. Trap: cost and context limits are counted in tokens, and Portuguese spends more tokens than English on the same text. ↗ Tokens

[41]ViT or CNN?

The ViT cuts the image into patches and treats them as tokens; it has less inductive bias than a CNN (no built-in locality), so it needs more data or pretraining to beat one. Trap: with little data and no pretraining the CNN usually wins; with large-scale pretraining the ViT scales better. ↗ Vision Transformers

[42]When should you not fine-tune?

Before changing weights, try what is cheaper: a better prompt, examples in the prompt, retrieval (RAG) or a simple classifier on the embeddings of a frozen model. Fine-tuning pays off when you need a format, a style or a domain those paths cannot reach. Trap: fine-tuning teaches behaviour better than it teaches new facts. ↗ Transfer learning

[43]What is LoRA?

The weights W stay frozen and a low-rank update is trained, \(W + BA\), with \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times k}\) and a small r: a tiny fraction of the parameters and of the optimizer's memory. QLoRA does the same on top of the model quantized to 4 bits. Trap: after training, BA can be added into W and inference does not get slower. ↗ Transfer learning

[44]Pretraining and post-training of an LLM?

Pretraining predicts the next token over a huge corpus, with no human labels; the result continues text. Post-training (supervised tuning on instructions, then preferences and reinforcement learning) turns it into an assistant that follows instructions. Trap: almost all the knowledge comes from pretraining; post-training mostly changes behaviour. ↗ LLMs

[45]Temperature, top-k and top-p?

The model returns a distribution over the vocabulary. Temperature divides the logits before the softmax: low sharpens it (near 0 it becomes greedy), high flattens it. Top-k keeps the k most probable tokens; top-p, the smallest set whose probability adds up to p. Trap: temperature 0 does not guarantee the same answer every time in a real service. ↗ Generation

[46]What does a VAE add to an autoencoder?

An autoencoder maps each input to a point of the latent space; there are holes between the points and decoding a random point gives garbage. A VAE encodes a distribution \(\mathcal N(\mu, \sigma^2)\) and a KL term pulls it towards \(\mathcal N(0, I)\): the space becomes continuous and you can sample from it. Trap: the samples come out blurry; today the VAE mostly serves as the compressor of latent diffusion. ↗ VAE

[47]The ELBO and the reparameterization trick?

The ELBO has two terms: reconstruction (the decoder rebuilds x from z) minus the KL between \(q(z|x)\) and the prior. Sampling z is not differentiable; writing \(z = \mu + \sigma \odot \varepsilon\), with \(\varepsilon \sim \mathcal N(0, I)\), turns the randomness into an input and the gradient flows through μ and σ. Trap: the β in front of the KL trades sharpness for a better-organized latent space. ↗ VAE

[48]How does a GAN train, and how does it fail?

A generator turns noise into samples and a discriminator tries to tell them from real ones; the two train in turns, in a zero-sum game. Failures: mode collapse (few distinct outputs), a vanishing gradient when the discriminator wins easily and oscillation without converging. Trap: the generator's loss does not tell you whether the samples improved; you have to look at them or measure with FID. ↗ GANs

[49]How does diffusion train? What is CFG?

Draw a step t and a noise ε, build \(x_t\) from \(x_0\) in closed form and the network learns to predict ε with MSE. Generation starts from pure noise and removes noise step by step. Classifier-free guidance combines the prediction with and without the prompt and pushes towards the prompt. Trap: high guidance obeys more and loses diversity and naturalness. ↗ Stable Diffusion

[50]How does CLIP train? What is zero-shot?

An image encoder and a text encoder map pairs into a shared space; in a batch of N pairs the contrastive loss pulls the N right pairs to a high cosine and pushes the \(N^2 - N\) wrong ones away, with a learned temperature. Zero-shot: write one sentence per class ("a photo of a cat") and pick the one with the highest cosine. Trap: zero-shot depends on the classes and the image style having shown up in pretraining. ↗ CLIP

Quiz

Twenty-four interview-style questions, three per theme. Filter by theme to review only what you need.

Study plan

How to study, and a plan for the two weeks before an interview.

Explain

Out loud, to someone

If you cannot explain backpropagation to a classmate who missed the class, you have not learned it yet. Studying in pairs, one interviewing the other with the case questions, is the best practice there is.

Show it in code

The loop and the shapes from memory

Write without looking anything up the training loop, an MLP, a small CNN and an attention head, noting the shape of every tensor. Run it, break it on purpose (no zero_grad, no eval) and see what changes.

Space it out

A day, a week, a month

Reviewing spaced out in time sticks far better than studying everything the night before. The study cards do it for you: the ones you miss come back first.

Tick honestly

The checklist is your thermometer

Tick an item only when you can explain it without looking at anything, trap included. A checklist full of items ticked out of optimism does not help in the interview.

Days	What to do
1–2	Fundamentals: perceptron, MLP and backpropagation by hand on a 2-layer network; the shape of every tensor
3–4	Optimization and depth: learning rate, Adam, initialization and residuals; play with the playground until you see the gradient vanish and come back
5–6	Regularization and evaluation: the two curves, dropout, BatchNorm, splitting, leakage and the metrics by hand
7–8	CNNs and attention: the shape and parameter arithmetic; one attention head by hand
9–10	Transformers, transfer learning, LLMs and generative models: one sentence and one trap per concept
11–12	Case questions in pairs, timed, with the quick diagnosis; the training loop written from memory
13–14	The full quiz and one last pass over the whole checklist, out loud

To review in the course's format: the study cards, with a challenge mode built on situations; the cheat sheet, with the formulas of every class; and the playground, to see in practice what the case questions describe.

Checklist

The 50 concepts and the interview habits. Your progress is saved in this browser.

Back to epoch 8

The full answer to "what would you do?"

1

Ask how the curve was measured"Was validation computed in model.eval()? Are there images of the same object or the same person in both training and validation?" A wrong curve leads to a wrong diagnosis. Principle 1, items [18], [24].

2

Give the diagnosis with the number"Up to epoch 8 the network learned the pattern; from there on the gap between the curves grows: it is memorizing the training noise. That is variance, not bias." Items [16], [21].

3

Propose, cheapest first"First, early stopping: I keep the epoch-8 weights. Then weight decay and data augmentation, which preserve the label on these images; dropout in the dense head; more data, if it can be collected. If the set is small, starting from a pretrained network is usually worth more than all of that." Items [17], [22], [18], [42].

4

Say how you will know it worked"I compare the variants on validation, over more than one seed, against the epoch-8 model and a baseline; only at the end do I measure once on test." Items [23], [26].

Four steps, one minute and nine of the 50 concepts used in a single answer. That is what the interviewer wanted to see: not what you memorized, but how you connect it.

References

To go deeper, grouped by kind.

Free books

Book · free

Understanding Deep Learning

Simon J. D. Prince (MIT Press, 2023). From MLPs to diffusion, with clear figures and notebooks. The best supporting read for almost every theme on this page.

udlbook.github.io/udlbook

Book · free, with code

Dive into Deep Learning

Zhang, Lipton, Li and Smola. Every chapter comes with PyTorch code that runs; great practice for the coding part of the interview.

d2l.ai

Book · free, classic

Deep Learning

Goodfellow, Bengio and Courville (MIT Press, 2016). The reference for optimization, regularization and the mathematical foundations.

deeplearningbook.org

Book · free, about interviews

Introduction to Machine Learning Interviews

Chip Huyen. How ML hiring processes work, and hundreds of real questions, from conceptual to case.

huyenchip.com/ml-interviews-book

Free online resources

Essay

A Recipe for Training Neural Networks

Andrej Karpathy. The step-by-step for debugging a training run, from "memorize one batch" to fine tuning. It is the case answer every interviewer would like to hear.

karpathy.github.io

Course notes

CS231n: Deep Learning for Computer Vision

Stanford. The notes on backpropagation, convolutional networks and babysitting a training run have been a reference for a decade.

cs231n.github.io

Documentation

PyTorch Tutorials

The training loop, Dataset and DataLoader, saving and loading models: the vocabulary that shows up in coding questions.

docs.pytorch.org/tutorials

The curve that rises at epoch 8

What a deep learning interview looks like

"Why divide by √dk?"

"The loss went NaN"

"Write the training loop"

What the interviewer assesses

How to answer: 8 principles

The training loop from memory

Five lines that win the job

Write the shape of every tensor

A case question, step by step

Quick diagnosis

"The loss went NaN at epoch 3"

What to use when

The 50 key concepts

Quiz

Study plan

Out loud, to someone

The loop and the shapes from memory

A day, a week, a month

The checklist is your thermometer

Checklist

Back to epoch 8

References

Free books

Free online resources

"Why divide by √d_k?"