3. MLP

Activity: Understanding Multi-Layer Perceptrons (MLPs)

This activity is designed to test your skills in Multi-Layer Perceptrons (MLPs).

The thread running through the activity is what a hidden layer buys you: one exercise by hand to see backpropagation move every parameter, then a dataset no straight line can touch, then more classes — and finally a fourth exercise that asks whether stacking a second hidden layer buys you anything at all.

Exercise 1

One Step of Backpropagation, by Hand

Consider an MLP with 2 input features, 1 hidden layer of 2 neurons, and 1 output neuron. Use \(\tanh\) as the activation on both layers. The loss is the mean squared error over a single sample, so \(N = 1\):

Use these values:

• Input and target: \(\mathbf{x} = [0.5, -0.2]\), \(y = 1.0\)

• Hidden weights: \(\mathbf{W}^{(1)} = \begin{bmatrix} 0.3 & -0.1 \\ 0.2 & 0.4 \end{bmatrix}\)

• Hidden biases: \(\mathbf{b}^{(1)} = [0.1, -0.2]\)

• Output weights: \(\mathbf{W}^{(2)} = [0.5, -0.3]\)

• Output bias: \(b^{(2)} = 0.2\)

• Learning rate: \(\eta = 0.3\)

Show every derivation and every intermediate number, to at least 4 decimal places. This exercise is done on paper (or in Markdown) — not by calling a library.

A — Forward pass

1. Hidden pre-activations: \(\mathbf{z}^{(1)} = \mathbf{W}^{(1)} \mathbf{x} + \mathbf{b}^{(1)}\)
2. Hidden activations: \(\mathbf{h}^{(1)} = \tanh(\mathbf{z}^{(1)})\)
3. Output pre-activation: \(u^{(2)} = \mathbf{W}^{(2)} \mathbf{h}^{(1)} + b^{(2)}\)
4. Output: \(\hat{y} = \tanh(u^{(2)})\), and then the loss \(L\).

B — Backward pass

Compute the gradient of the loss with respect to every weight and every bias. Start from \(\displaystyle \frac{\partial L}{\partial \hat{y}}\) and chain outward:

1. \(\displaystyle \frac{\partial L}{\partial u^{(2)}}\), using the \(\tanh\) derivative;
2. Output layer: \(\displaystyle \frac{\partial L}{\partial \mathbf{W}^{(2)}}\) and \(\displaystyle \frac{\partial L}{\partial b^{(2)}}\);
3. Propagate back: \(\displaystyle \frac{\partial L}{\partial \mathbf{h}^{(1)}}\) and \(\displaystyle \frac{\partial L}{\partial \mathbf{z}^{(1)}}\);
4. Hidden layer: \(\displaystyle \frac{\partial L}{\partial \mathbf{W}^{(1)}}\) and \(\displaystyle \frac{\partial L}{\partial \mathbf{b}^{(1)}}\).

C — Parameter update

With \(\eta = 0.3\), apply gradient descent to all eight parameters:

Report the numerical value of every updated parameter, and state whether the loss would go up or down on a second forward pass — and why.

Exercise 2

Binary Classification a Straight Line Cannot Do

A — Generate the data

Class 1 is split into two clusters that sit on opposite sides of Class 0, so no straight line can separate them — which is exactly the situation a hidden layer exists for. Use these parameters, unchanged:

from sklearn.datasets import make_blobs

centers = np.array([[0., 0.], [3., 3.], [-3., -3.]])
X, c = make_blobs(n_samples=[500, 250, 250], centers=centers,
                  cluster_std=1.2, random_state=42)
y = (c > 0).astype(int)      # class 0: 1 cluster | class 1: 2 clusters

Split 80% train / 20% test with train_test_split(..., test_size=0.2, random_state=42, stratify=y).

Produce Figure 1: a scatter plot of the 1000 points colored by y.

B — Establish the baseline

Before building anything, fit a single linear boundary to the training set and report its test accuracy. You may use sklearn.linear_model.LogisticRegression here — this is the baseline, not your model. It should land near 48%: worse than guessing. Keep that number; item D asks about it.

C — Implement the MLP

Write an MLP from scratch. You choose the architecture, but it must have at least one hidden layer, and you must implement yourself:

• the forward pass;
• the loss (binary cross-entropy is the natural choice);
• the backward pass — every gradient derived, not autodiffed;
• the parameter update.

Write it as a function or class that takes the layer sizes as an argument. Exercises 3 and 4 reuse this same code, so a hard-coded two-layer network will cost you there.

Train for a reasonable number of epochs (a few hundred to a few thousand), recording the training loss each epoch. Then:

1. Report the architecture, the learning rate, the number of epochs, and the test accuracy. A correct implementation lands around 90–93%.
2. Produce Figure 2: training loss \(\times\) epoch.
3. Produce Figure 3: the decision boundary over the test points — evaluate your network on a grid and shade the two regions.

D — Analysis

Why does the linear baseline score below 50% while your MLP clears 90%? Point at Figure 3: describe the shape of the region your network assigns to Class 1, and explain why no single straight line can produce it.

Exercise 3

Multi-Class, Same Network

A — Generate the data

from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1500, n_features=4, n_informative=4,
                           n_redundant=0, n_repeated=0, n_classes=3,
                           n_clusters_per_class=2, class_sep=1.0,
                           random_state=42)

That is 1500 samples, 4 informative features, 3 classes, 2 clusters each. Split 80/20 with the same seed and stratify=y.

B — Adapt the network

Extend your Exercise 2 implementation to three classes. The changes you need are in the output layer and the loss:

• the output layer now has 3 units;
• softmax turns those into probabilities;
• categorical cross-entropy replaces binary cross-entropy.

Derive \(\partial L / \partial u\) for the softmax + cross-entropy pair and show the derivation in the report — the result is famously simple, and knowing why is the point of this item.

C — Train and evaluate

1. Report the architecture, hyperparameters, and test accuracy. Expect roughly 83–86%.
2. Produce Figure 4: training loss \(\times\) epoch.
3. Produce Figure 5: the confusion matrix on the test set.

D — Analysis

Which pair of classes does the network confuse most, according to Figure 5? The data has 4 informative dimensions and 6 clusters total — offer an explanation for the confusion that refers to that structure. Also report the logistic-regression baseline on this data (around 66%) and say what the hidden layer added.

Exercise 4

Does a Second Hidden Layer Help?

Same data as Exercise 3, same training budget, same seed. The only change is depth: at least 2 hidden layers.

This is a controlled comparison, not a hunt for a better score. Change one thing at a time and keep everything else fixed, or the comparison means nothing.

A — Train the deeper network

Report the architecture and the test accuracy, trained with the same number of epochs and the same learning rate as Exercise 3.

B — Compare

Produce Figure 6: the training-loss curves of the Exercise 3 network and the Exercise 4 network on the same axes. Then fill in this comparison in the text:

Run both with at least 3 different seeds and report the mean, so you are comparing models rather than luck.

C — Analysis

Explain the result. Useful angles: how many clusters the data actually has and how complex a boundary they need; what a second layer adds in representational terms versus what it costs in optimization; whether the training loss tells the same story as the test accuracy. Say what kind of data would make the extra depth pay off.

Results summary

Close the report with this table, filled in:

Evaluation Criteria

The deliverable for this activity is a report that includes:

1. Exercise 1 worked entirely by hand, every step shown to at least 4 decimals.
2. The code for your MLP and for the data generation, commented.
3. Figures 1 to 6, numbered as requested.
4. Your answers to the analysis questions in items D (and 4C).
5. The Results summary table.

Important Notes:

• The deliverable is a GitHub Pages site backed by a public repository — see Submission Format for the required layout, front matter and checklist;

• There is a strict no-plagiarism policy. Any form of plagiarism will result in a zero grade for the activity and may lead to further disciplinary action under the university's academic integrity policies;

• The deadline for each activity is not extended — NO EXCEPTIONS will be made for late submissions.

• AI collaboration is allowed, but each student MUST UNDERSTAND and be able to explain every part of the submitted code and analysis. Any use of AI tools must be properly cited. ORAL EXAMS may be conducted.

• All deliverables for individual activities should be submitted through the course platform insper.blackboard.com.

Grading Criteria:

Each row is worth the points indicated, awarded in full, partially (half), or not at all: in full when the item is complete and correct; partially when it is implemented but missing the requested analysis, or when the analysis lacks the numerical result that supports it; zero when absent or incorrect.

The accuracy value itself is not graded. The datasets are fixed and seeded, so the numbers are what they are; what is graded is the method, the figures, and reporting honestly what you got — including in Exercise 4, where the expected finding is that depth does not help.

Exercise 1 — Backpropagation by hand (2 points):

Exercise 2 — Binary classification (3 points):

Exercise 3 — Multi-class (3 points):

Exercise 4 — Depth comparison (2 points):