Playground: NetLab
Loading NetLab… (it needs JavaScript)
How to use it
| To | Do |
|---|---|
| add a node | double-click the canvas and type its name, or drag it from the Nodes list (on a phone: + Node) |
| connect two nodes | drag from an output port (right) to an input port (left); the colours say what a port carries: blue a tensor, orange the target, purple a loss |
| move a wire | drag its input end and drop it somewhere else; dropped on empty space, it opens the search with the nodes that take it |
| delete | select a node or a wire and press Del (or the × on the node) |
| navigate | wheel or pinch to zoom, drag the background to pan, F to fit everything in view |
| train | ▶ Train (or Space), ⏭ Step for one mini-batch, ↺ Reset for new random weights |
| read the code | { } PyTorch; click a node to highlight its lines, click a line to find its node |
Nothing is sent anywhere: the graph is saved in this browser, and Export and Import move it as a JSON file. Each node has a ↗ that opens the class where it is taught.
A node with a red border has a problem, and the message says what to do: a cross-entropy whose last layer does not have one unit per class, a sum of two tensors of different widths, a port left unconnected. The wire labels (N×16) are the shapes the MLP class computes by hand: N rows, one per example of the batch, and 16 columns, one per unit. An image travels as N×C×H×W (channels, height, width), as in the CNN class: Conv2d and MaxPool2d take it, Linear does not, and a Flatten in between turns N×16×2×2 into N×64.
The templates
Each template is a short experiment. Train it, look at the viewers, then change one thing.
The menu groups the templates by class, in the order of the course, and building one opens a note on what to try with a link back to its class. Each class page has a box that opens its scenario directly (playground/?t=<name>); the graph you had stays one Ctrl+Z away.
| Template | What it shows | Where it is taught |
|---|---|---|
| MLP on the spiral | Two hidden layers of 32 ReLU units separate three interleaved spirals, which no straight line can do. The decision view shows the boundary bending as the loss falls. Try 4 units per layer, or tanh. | MLP |
| Perceptron on XOR | One Linear layer is a straight line, so it stays near 50% on XOR whatever the optimizer does. Switch the dataset's Features to + squares, product and the same perceptron solves it: the product \(x_1 x_2\) is the feature XOR needs. | Perceptron |
| Too deep, with sigmoid | Six sigmoid layers with the default initialization: the Gradient flow viewer shows the gradient shrinking layer after layer, the loss stays flat near \(\ln 2\) and the wires go dark towards the input. | Deep Networks |
| Same depth: ReLU + residual | The same six layers with ReLU, He initialization and a skip connection around each block: the bars of the gradient flow are level and the loss falls. | Deep Networks |
| Same depth: sigmoid + BatchNorm | The six sigmoid layers of Too deep, with a BatchNorm between each Linear and its sigmoid: every layer gets inputs of mean 0 and variance 1 again, the sigmoids leave their flat tails and the loss falls. While training, BatchNorm uses each batch's own mean and variance; when evaluating, the running averages it kept (the model.train() and model.eval() of the code). | Deep Networks |
| Overfitting vs. dropout and weight decay | Two copies of a large network trained on 50 noisy points. The plain one drives its training loss to zero while its validation loss climbs (its best epoch is one of the first); the one with dropout and weight decay keeps the two curves close, its validation loss keeps falling and its boundary is smoother. | Regularization |
| Regression on a sine | A tanh network fits a noisy sine with MSE. The second decision view, on the first hidden layer, shows the 32 curves the output layer adds up. | Metrics: regression |
| CNN on digits | Two blocks of Conv2d, ReLU and MaxPool2d, then a Flatten and one Linear layer, on the 8×8 handwritten digits (1,797 images, 10 classes): about 98% on validation within a few seconds, with 1,898 parameters. The first Feature maps viewer shows a validation digit and the 8 channels of the first block; the second, on the output, a page of validation digits with the digit the network picks, green when right and red when wrong. The Confusion matrix shows which digits it mixes up, and Saliency shows, under each validation digit, the pixels where more ink would raise the score of that digit (blue) or lower it (orange). Adversarial (FGSM) moves every pixel by \(\epsilon\) in the direction that raises the loss: digits that look the same are misread, and \(\epsilon = 0.1\) of the pixel range takes the accuracy from about 98% to about 56%. Try a single block, or one Linear layer on the flattened pixels. | CNN |
| VAE on digits | An encoder takes each digit to the mean \(\mu\) and the log-variance \(\log \sigma^2\) of a 2-D Gaussian. Sample z draws \(z = \mu + \sigma \epsilon\) (the reparameterization trick, so the gradient reaches \(\mu\) and \(\sigma\)) and a decoder turns \(z\) back into 64 pixels. The VAE loss adds the reconstruction error to \(\beta\) times the KL to \(\mathcal{N}(0, I)\); the reconstruction is the squared error or the binary cross-entropy. The first Latent space viewer puts each validation digit at its \(\mu\), coloured by its class (the clusters form without the labels); the second runs the decoder on a grid of \(z\), which is what the VAE generates. Feature maps on the output shows each digit over its reconstruction. Try \(\beta = 0\): the KL climbs and the codes spread far beyond the dashed circles of the prior. | VAE |
| Autoencoder on digits | The VAE's network without Sample z and without the KL: the Reconstruction loss only compares the output with \(x\), so the code is a point, not a Gaussian. Compare its Latent space with the VAE's: the codes spread far beyond the circles of \(\mathcal{N}(0, I)\) and the decoded grid around the origin covers only part of them, which is why a plain autoencoder does not generate. The decoder can also be convolutional: Unflatten reads a flat vector as \(C \times H \times W\) and Upsample ×2 followed by a Conv2d grows it back to 8×8. | VAE |
| Flow matching on moons | A Flow pair takes each data point \(x_1\), draws a noise \(x_0 \sim \mathcal{N}(0, I)\) and a time \(t\), and gives the network the point \(x_t = t\,x_1 + (1 - t)\,x_0\) on the line between them, with the target \(v = x_1 - x_0\). A small MLP regresses \(v\) with MSE; the loss stays well above 0, because many lines cross the same \((x_t, t)\) and the network can only learn their average. The Sampler integrates the learned velocity from noise at \(t = 0\) to the data at \(t = 1\) with Euler steps: its first view draws the paths, the second the velocity field at one time. Switch the pair to the cosine path and to predicting \(\epsilon\) and it is a diffusion model, sampled with DDIM. Try 1, 2 or 5 steps. | Flow Matching |
| Mini ViT on digits | A Patches node cuts each 8×8 digit into 16 squares of 2×2; a Linear layer turns each one into a token of 16 numbers (the same layer for every token) and Position adds a learned vector per place. One Transformer block follows: LayerNorm, Attention with 2 heads and a residual Add, then LayerNorm, an MLP and another Add. Mean over tokens feeds a Linear layer with 10 outputs: about 93% on validation after 15 epochs, with 2,730 parameters. The Attention map shows, on a validation digit, where each head looks: from one patch, or how much attention each patch receives. Compare with the CNN template on the same digits. | Vision Transformers |
| RNN vs. LSTM: the adding problem | Sequences of 40 steps of two numbers: a value between 0 and 1 and a mark, which is 1 at two steps (one in each half) and 0 elsewhere. The target is the sum of the two marked values, so the network has to keep the first one for 20 to 40 steps while ignoring all the others. The same head is trained on a plain RNN (tanh) and on an LSTM, 16 units each. The RNN stays at R² near 0 or below: at best it learns to answer the mean, 1. The LSTM reaches R² ≈ 0.99 within the 40 epochs. The Through time viewer shows why, on a validation sequence: the input, the state \(h_t\) of each unit and \(\lVert \partial L / \partial h_t \rVert\) at each step. Through the RNN the gradient falls by three to thirteen orders of magnitude from the last step back to the first, and its states sit at ±1; in the LSTM it falls by less than two orders of magnitude, and a unit switches on at the first mark and holds its value to the end (Show the memory gate to see the forget gate \(f_t\)). At the start the LSTM's gradient fades too, since its forget gates begin near 0.5: watch its bars level out as the loss drops. With Length T at 10 the RNN learns too (R² ≈ 0.93); at 20 it already fails. The GRU cell learns at 40 and at 80. The learning rate is 0.03 and the gradient is clipped at norm 1: with 0.01 and no clipping, the LSTM sometimes stays on its first plateau for all 40 epochs (6 runs in 20). If its curve stays flat, ↺ Reset and train again. | Attention: recurrence |
| LSTM reads digits row by row | Patches set to rows (1×W) turns each 8×8 digit into a sequence of 8 steps of 8 pixels, read from top to bottom. An LSTM of 32 units reads it and a Linear layer on its last state gives the 10 classes: about 98% on validation within 20 epochs, with 5,706 parameters, close to the CNN template. Eight steps make a short sequence, so a plain RNN learns it too (switch the Cell: 93–96%, a few points below); the LSTM's clear advantage needs distance, as in the template above. | Attention: recurrence |
| Transfer learning on the spiral | An MLP learns the spiral's three arms with every label (1,050): train it for about 30 epochs, to about 99%. Then give it a finer task on the same points: set the Dataset's Task to arm, inner or outer (6 classes: each arm split at the middle of its length, which the three-arm task never asked for) and Labels to 12, two per class. The old head cannot answer it, since its three outputs say which arm and not which half. Press the Loss's button that sets fc3 to 6 units (a new head; the rest keeps its weights), set fc1 and fc2 to frozen and train: about 87% on the 450 validation points. Now press Reset, set them back to trainable and train from scratch on the same 12 labels: about 70%, and the decision view shows why (12 points do not draw a spiral). Fine-tuning everything without Reset lands in between (about 84%): 12 labels are enough to spoil part of the features. The frozen layers were trained only to tell the arms apart, yet they still carry where along the arm a point lies, and the new head reads both. With inner or outer alone the arms are of no use and no regime is reliably ahead (67% frozen, 59% fine-tuned, 60% from scratch). Transfer pays when the source task learned what the new one needs (in PyTorch, over 5 seeds: 0.91 frozen, 0.86 fine-tuned, 0.80 from scratch; with inner or outer, 0.75, 0.79 and 0.81). | Transfer Learning |
| World model: RSSM (Dreamer) | One step of the recurrent state-space model of PlaNet and Dreamer, on the Room data: a dot in a room with walls, seen as an 8×8 image, a step \(a\) and the next frame. A Transition splits each row into \(o\), \(a\) and \(o'\); Stack sends \(o\) and \(o'\) through the same encoder in one batch and Split in two takes their embeddings \(e\) and \(e'\) apart. The deterministic state is \(h = f(e, a)\) (a GRU in Dreamer; a layer here, since one step needs no memory). From \(h\) alone come the prior's \(\mu\) and \(\log\sigma^2\) (fc4, fc5); from \(h\) and \(e'\), the posterior's (fc6, fc7), which Sample z draws \(s'\) from; the decoder draws \(o'\) from \((h, s')\). The RSSM loss adds DreamerV3's KL terms to the reconstruction: \(0.5\,\mathrm{KL}(\mathrm{sg}(q)\,\|\,p)\) trains the prior, \(0.1\,\mathrm{KL}(q\,\|\,\mathrm{sg}(p))\) the posterior (sg: stop-gradient), neither below 1 nat (free bits). Next frame shows four validation steps: \(o\) with its step, the true \(o'\), the frame decoded from the posterior and the dream, the decoder fed the prior's mean, which is all the model has when it imagines. After 35 epochs: 0.40 per frame from the posterior, 0.43 dreaming (in PyTorch, 0.41 and 0.44). Set the KL term to none and press Reset: the posterior improves to 0.32, since it sees \(o'\), and the dream degrades to 1.61. | World Models |
| LeWorldModel: latent + SIGReg | The same Room without a decoder: one encoder for \(o\) and \(o'\) (8 numbers each), a predictor from \((z, a)\) to \(\hat z'\) and the Latent loss \(\|\hat z' - z'\|^2 + \lambda\,\mathrm{SIGReg}(Z)\) with \(\lambda = 0.1\), with no stop-gradient and no EMA. SIGReg projects the batch of embeddings on 64 random directions and measures on each how far it is from \(\mathcal{N}(0, 1)\) (the Epps–Pulley statistic). The Latent probe draws \(z\) on its two main directions in the colour of where the dot really is (the colours of the Dataset's room), the spread σ of each dimension, the effective rank and a linear probe of the position: \(R^2 \approx 0.86\) after 30 epochs (0.83 to 0.91 in PyTorch, over 3 seeds). Set \(\lambda = 0\) and press Reset: the prediction error goes to 0 because every frame gets the same \(z\) (σ 0.006, \(R^2\) 0.07), a collapse. With \(\lambda = 0.03\) the probe reaches 0.94. LeWM's predictor reads the last three latents; the dot needs one, since its position is its whole state. Plan in the latent, on the predictor's output, uses the model as LeWM, DINO-WM and PLDM do: for 100 pairs of a start and a goal frame it encodes both, searches the 5 steps whose imagined latent ends nearest the goal's (the cross-entropy method: 64 candidate plans, the 8 best set the next 64, 4 rounds), then does the plan in the real room. After 30 epochs it brings about half of the goals within 0.2 (47% to 57% over 3 runs; 30% to 39% in PyTorch, where the draw of the plans alone moves it by about 8 points), against 3% with random actions; with the collapsed latent of \(\lambda = 0\), 4%. The loss's Target z′ gives the other way out of collapse, the asymmetry of BYOL and JEPA: with \(\lambda = 0\) and a stop-gradient on \(z'\), or \(z'\) from an EMA encoder (an average of the past weights, momentum 0.99), nothing collapses (\(R^2\) 0.91 to 0.95) and the plans reach 33% to 40% (stop-gradient) and 44% to 53% (EMA) of the goals. | Temporal Discriminative World Models |
| GAN on moons | A Noise z feeds a generator (64, 64, 2 units); the GAN pair stacks a batch of real points over the generator's fakes, labelled 1 and 0, and the discriminator (64, 64, 1) reads them all. The GAN loss has two ports: loss D (real → 1, fake → 0) for the discriminator's optimizer and loss G (non-saturating: fake → 1) for the generator's, which goes through the discriminator but changes only the generator. The GAN view draws, as GAN Lab does, the data, the samples and the discriminator's verdict over the plane; its second distance (data → samples) grows when the generator collapses onto part of the data. The curves oscillate: that is the game, not a bug. | GANs |
| A line through a sine | One Linear unit, the squared error and SGD: the three pieces every template is made of. A line cannot follow a sine (\(R^2 \approx 0.72\), however long it trains); put a hidden Linear with a tanh in between and it can. | Concepts |
| Train, validation and noise | 120 noisy points of the moons, half of them for validation, and a network large enough to memorize the other half: the training loss goes to 0 while the validation loss climbs past 2. Change the number of points, the noise, the validation share or the seed of the split and see how much the validation number moves. | Data |
| Three neurons on the moons | A hidden layer of 3 tanh units: each one is a line through the plane and the output combines the three (about 91% on validation). Try 1 unit, a single line, and 8. | Neural Networks |
| SGD vs. Adam | The MLP of the spiral twice, trained with SGD (rate 0.1) and with Adam (0.01). In the same 14 seconds, SGD was at 75% and Adam at 94%. Change the rates, or try momentum and RMSProp. | Optimization |
| Digits: the confusion matrix | An MLP on the 8×8 digits (about 98% on validation) read through the Confusion matrix: a row's cells off the diagonal are that digit's misses (its recall), a column's are the false alarms (its precision). | Metrics: classification |
| Attention: the adding problem | The adding problem of the RNN template, by attention: a Linear turns each step into a token of 16, Attention (2 heads) lets every token reach the two marked steps in one hop and Mean over tokens feeds the answer. \(R^2 \approx 0.94\) within 10 epochs at \(T = 40\), where the plain RNN stays near 0. No position is needed: the sum does not depend on the order. The Attention map shows where the heads look. | Attention |
| A Transformer block | The same sequences through a whole pre-LN block: LayerNorm, attention and a residual Add, then LayerNorm, an MLP with GELU and another Add: \(R^2 \approx 0.99\) within 7 epochs. Pull out a residual wire and train again. | Transformers: the block |
| A tiny GPT: the next character | The Text dataset: sentences of a subject, a verb and an object ("the cat jumped on the floor."), cut into windows of 20 characters; the input is each window one-hot and the target the same window shifted by one. A Linear embeds each character, Position adds where it is, and one pre-LN block follows, its Attention set to the causal mask (each token sees only the past); a last Linear gives one score per character at every position, and the Loss takes the cross-entropy of all of them. About 89% of the next characters right after 5 epochs. The Next character viewer shows the prediction at each position of a validation window and then writes text one character at a time from it, with a temperature. Set the mask to none: the loss drops near 0 because every position but the last reads its answer, and the number stops measuring prediction. | Autoregressive Generation |
| Two blocks: a bigger language model | The same recipe with two blocks and a stream of 48: about three times the parameters, each step dearer. Compare it with the one-block model at the same epoch, and cut the number of windows to see training and validation part. | LLMs |
| CLIP on digits | Two encoders: one for the image (64 pixels → 64 → 16) and one for its caption, which here is the digit's class written as a one-hot vector (One-hot caption → Linear of 16). The Contrastive loss (CLIP) normalizes both embeddings, divides the N×N matrix of cosines by the temperature \(\tau\) and takes the cross-entropy of every row (image → its caption) and every column (caption → its image). Its accuracy is the share of images whose most similar caption is their own; on the validation set, where every class is present, that is zero-shot classification: about 98% after 20 epochs. The loss stays near 4 and that is expected: a batch has many images of the same class, which share a caption, so the right pair can never take all the probability. | CLIP |
| Diffusion on moons | The graph of Flow matching on moons with the Flow pair on the cosine path and predicting the noise \(\epsilon\): a diffusion model, which the Sampler runs with DDIM. Compare its paths with flow matching's and try few sampling steps. | Stable Diffusion |
| I-JEPA on digits | I-JEPA with NetLab's nodes. Patches cuts each digit into 16 tokens, a Linear and Position embed them, and the JEPA mask draws two target blocks of 2×2 patches per image and sends the batch twice, [context view; full view], so that one encoder computes both. Its Attention set to JEPA context keeps the context half from looking at the targets (the same as leaving them out). Split in two separates the halves: the context goes to Mask tokens (a learned token plus a learned position at each target) and a predictor block; the full view, through a LayerNorm, is the target. The JEPA loss compares them at the targets only, with a stop-gradient on the target. The Representation probe reads the full view: spread, effective rank and a linear probe against the raw pixels. Measured: the untrained encoder's probe (mean of the tokens) gives about 56%, a dozen epochs about 65% with the spread doubling, the raw pixels 92%. At this scale, as in the JEPA chapter's lab, predicting representations does not beat pixels. With the probe on every token side by side, even the untrained encoder reaches about 97%: more numbers, not better features, which is why a probe is read against the untrained encoder. With the gradient on both sides the loss goes to 0.000 and the spread to 0.01 (collapse), while the effective rank stays near 9, blind to a representation that simply shrank. | JEPA |
| JEPA: collapse and the stop-gradient | The graph of LeWorldModel with \(\lambda = 0\) and the target behind a stop-gradient: a JEPA held up by asymmetry alone. Switch Target z′ to the gradient on both sides and the latent collapses; the EMA encoder, or SIGReg with \(\lambda = 0.1\), prevents it again. | JEPA |
The optimizers are the ones of the Optimization class: SGD, SGD with momentum, RMSProp and Adam, with the learning rate and the weight decay on the node. The Optimizer node also shows the ratio between the size of the last update and the size of the weights: around \(10^{-3}\) is a healthy step; much larger and the training jumps around, much smaller and it crawls. It can also change the learning rate along the epochs (steps of ×0.1, a cosine, or a warm-up followed by the cosine, as PyTorch's lr_schedulers do; the curve on the node marks the current epoch) and clip the norm of the gradient (clip_grad_norm_), reporting how often it did.
Your data (CSV)
In the Dataset node, choose Your data · CSV and press Choose a file…, or drop the file on the node. The file must be ready: the node trains on what it gets and does very little to it. Step by step, it does only this (the node lists the same steps under What this node does with the file, with the numbers of your file):
- Reads the file in this browser; nothing is sent anywhere. The first row names the columns; the separator (
,,;or tab) is detected, and so is a decimal comma (3,14) when the separator is not a comma. - Target: the column chosen in Target (the last one by default). With Task on auto, it is a classification when the target is text or has at most 10 whole values (numbered 0, 1, … in sorted order), and a regression otherwise. Up to 10 classes.
- Inputs: every numeric column except the target. Text columns are left out, and so is a column named
id,indexorUnnamed: 0. - Missing values: a row with an empty cell (or
NA,NaN,null,?and the other markers pandas reads as missing) in the target or in an input is dropped, not filled in. - Rows and split: at most Points rows (2,000 at the most, drawn with the seed when the file has more), split into training and validation (stratified by class).
- Standardization: each input gets the mean and standard deviation of the training set, as a
StandardScalerfitted on it alone. In a regression the target is standardized too, so the loss is in units of its standard deviation; the R² does not change.
It does not impute values, encode categories (one-hot), transform columns (log, powers), remove outliers or pick columns. Do that first in the ML simulator, which has a Preparation block for each of these, or in a spreadsheet, and save the result as CSV. A file up to 10 MB is accepted. It is kept in this browser for the next visit, and Export puts it inside the graph's JSON. The PyTorch code repeats the same steps with pd.read_csv, dropna and StandardScaler, and runs with the file next to it.
Two checks, with the MLP template and the iris flowers (150 rows, 8 of them with a missing value): 142 rows remain, and the network reaches about 95% on validation (98% in PyTorch). On scikit-learn's diabetes data (442 rows, 10 inputs, a regression), a single Linear layer reaches \(R^2 \approx 0.46\) (scikit-learn's linear regression gets 0.44 to 0.50, depending on the split); the 32 + 32 MLP overfits it within a hundred epochs, a good moment to try dropout or weight decay. The decision view only draws a file with 2 inputs (1 in a regression).
What runs underneath
Everything is computed in the page, in JavaScript: the forward pass of each node, the backward pass that sends the gradient back along the wires (a wire that feeds two nodes adds up the two gradients coming back) and the update of the optimizer. The datasets are small (up to 2,000 points) and the layers have up to 256 units, so a step takes a fraction of a millisecond. A convolution costs more: on the digits a step of the CNN takes a few milliseconds, and the loss curve gets a point less often. The digits are scikit-learn's load_digits (UCI, CC BY 4.0), loaded with the page.
The PyTorch code describes the same network, loss, optimizer and data, and runs as it is. The numbers it prints will not be the same as here: the random initial weights and the order of the batches are drawn differently.