5. Multi-Layer Perceptron

Fast Forward: Multi-Layer Perceptrons (MLPs)

Every neuron does the same two things: it takes a weighted sum of everything that reaches it (plus a bias), and it passes that sum through a non-linear activation. The zoom panel above shows those two operations; the network is that one neuron repeated and stacked.

where:

• \( y_k \) is the output for the \( k \)-th output neuron.
• \( x_i \) are the input features.
• \( w_{ji} \) is the weight connecting the \( i \)-th input to the \( j \)-th hidden neuron — the receiving neuron comes first, so that \( w_{ji} \) is the entry in row \( j \), column \( i \) of \( \mathbf{W} \).
• \( v_{kj} \) is the weight connecting the \( j \)-th hidden neuron to the \( k \)-th output neuron.
• \( b^{h}_{j} \) is the bias for the \( j \)-th hidden neuron.
• \( b^{y}_{k} \) is the bias for the \( k \)-th output neuron.
• \( m \) is the number of hidden neurons.
• \( n \) is the number of input features.
• \( \sigma \) is the activation function applied to the weighted sums at each layer, such as sigmoid, tanh, or ReLU.

Matrix representation of the MLP architecture:

Activation Functions

All three share the same horizontal axis, so the dashed derivatives can be compared directly. Note where the vanishing-gradient problem comes from: the sigmoid derivative never exceeds 0.25, so every layer a gradient crosses shrinks it by at least a factor of four, whereas ReLU's derivative is exactly 1 wherever the unit is active.

Backpropagation is the algorithm used to train multi-layer perceptrons (MLPs) by adjusting the weights and biases based on the error between the predicted output and the actual target. The process involves two main steps:

1. Forward Pass: The input data is passed through the network, layer by layer, to compute the output. The output is compared to the target value to calculate the loss (error).
2. Loss Calculation: Calculate the loss (error) between the predicted output and the actual target using a loss function, such as mean squared error or cross-entropy.
3. Backward Pass: The error is propagated backward through the network to compute the gradients of the loss with respect to each weight and bias. These gradients are then used to update the weights and biases using an optimization algorithm, such as stochastic gradient descent (SGD) or Adam.

Feedforward

Consider a Multi-Layer Perceptron (MLP) with:

• 2 input neurons: \(x_1\) and \(x_2\)
• 1 hidden layer with 2 neurons: \(h_1\) and \(h_2\)
• 1 output neuron: \(y\)

We assume sigmoid activation functions for both the hidden and output layers:

, with derivative

The architecture, and the forward pass through it, can be walked step by step. The values shown are the ones used in the Numerical Simulation below, so the diagram and the arithmetic are the same example:

In mathematical terms, the feedforward process can be described as follows:

or, more canonical for our simple MLP:

1. Hidden layer pre-activation:

1. Hidden layer activations:

1. Output layer pre-activation:

1. Output layer activation:

where \( \sigma \) is the activation function, \( w_{ji} \) is the weight from input \( i \) to hidden neuron \( j \), and \( v_{kj} \) is the weight from hidden neuron \( j \) to output neuron \( k \) — receiving neuron first, matching the rows of \( \mathbf{W} \) and \( \mathbf{V} \). The biases \( b^h_1, b^h_2, \) and \( b^y_1 \) are added to the respective layers. \( \hat{y} \) is the predicted output of the MLP.

Loss Calculation

The loss function quantifies the difference between the predicted output and the actual target. For regression tasks, a common loss function is the Mean Squared Error (MSE):

where \( N \) is the number of samples, \( y_i \) is the true output, and \( \hat{y}_i \) is the predicted output.

Backpropagation: Computing Gradients

The backpropagation algorithm is a method used to train multi-layer perceptrons (MLPs) by minimizing the error between the predicted output and the actual target. It involves two main steps: the forward pass and the backward pass. The update of weights and biases is done using the gradients computed during the backward pass.

Backpropagation computes the partial derivatives of \(L\) with respect to each parameter using the chain rule, starting from the output and propagating errors backward.

Update Rule

To update parameters during training (e.g., via gradient descent with learning rate \(\eta\)), for each weight/bias \(p\):

This derivation assumes a single example; for batches, average the gradients. For other activations or losses (e.g., softmax + cross-entropy), the deltas would adjust accordingly, but the chain rule structure remains similar.

The five steps below are the same walk the diagram takes. The network is the one from the forward pass, with the values it produced still on the neurons — backpropagation does not recompute them, it reuses them:

Step 1: Output Layer Error

The error term (delta) for the output is:

Step 2: Gradients for Output Weights and Bias

Using \(\sigma_y\):

\(\begin{align} \frac{\partial L}{\partial v_{11}} &= \sigma_y \cdot h_1 \\ \\ \overbrace{\frac{\partial L}{\partial u} \cdot \frac{\partial u}{\partial v_{11}}}^{\text{chain rule}} &= \sigma_y \cdot h_1 \end{align}\)

Similarly:

\(\begin{align} \frac{\partial L}{\partial v_{12}} &= \sigma_y \cdot h_2 \\ \\ \overbrace{\frac{\partial L}{\partial u} \cdot \frac{\partial u}{\partial v_{12}}}^{\text{chain rule}} &= \sigma_y \cdot h_2 \end{align}\)

For the bias:

\(\begin{align} \frac{\partial L}{\partial b^y_1} &= \sigma_y \cdot 1 \\ \\ \overbrace{\frac{\partial L}{\partial u} \cdot \frac{\partial u}{\partial b^y_1}}^{\text{chain rule}} &= \sigma_y \end{align}\)

Step 3: Hidden Layer Errors

Propagate the error back to the hidden layer. For each hidden neuron:

\(\begin{array} \displaystyle \sigma_{h_1} &= \displaystyle \frac{\partial L}{\partial z_1} \\ &= \displaystyle \frac{\partial L}{\partial h_1} & \displaystyle \cdot \frac{\partial h_1}{\partial z_1} \\ &= \displaystyle \overbrace{ \left( \frac{\partial L}{\partial u} \cdot \frac{\partial u}{\partial h_1} \right)}^{\text{chain rule}} & \cdot \sigma'(z_1) \\ &= (\sigma_y \cdot v_{11}) & \cdot \sigma'(z_1) \\ &= (\sigma_y \cdot v_{11}) & \cdot h_1(1 - h_1) \end{array}\)

Similarly:

\(\sigma_{h_2} = (\sigma_y \cdot v_{12}) \cdot h_2(1 - h_2)\)

Step 4: Gradients for Hidden Weights and Biases

Using the hidden deltas:

\(\begin{align} \frac{\partial L}{\partial w_{11}} &= \frac{\partial L}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_{11}} &= \sigma_{h_1} \cdot x_1 \end{align}\)

\(\begin{align} \frac{\partial L}{\partial w_{12}} &= \frac{\partial L}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_{12}} &= \sigma_{h_1} \cdot x_2 \end{align}\)

\(\begin{align} \frac{\partial L}{\partial w_{21}} &= \frac{\partial L}{\partial z_2} \cdot \frac{\partial z_2}{\partial w_{21}} &= \sigma_{h_2} \cdot x_1 \end{align}\)

\(\begin{align} \frac{\partial L}{\partial w_{22}} &= \frac{\partial L}{\partial z_2} \cdot \frac{\partial z_2}{\partial w_{22}} &= \sigma_{h_2} \cdot x_2 \end{align}\)

similarly for biases:

\(\begin{align} \frac{\partial L}{\partial b^h_1} &= \sigma_{h_1} \cdot 1 &= \sigma_{h_1} \end{align}\)

\(\begin{align} \frac{\partial L}{\partial b^h_2} &= \sigma_{h_2} \cdot 1 &= \sigma_{h_2} \end{align}\)

Step 5: Update Weights and Biases

Finally, update the weights and biases using the computed gradients and a learning rate \(\eta\):

\(\begin{align} v_{11} & \leftarrow v_{11} - \eta \cdot \frac{\partial L}{\partial v_{11}} \\ v_{12} & \leftarrow v_{12} - \eta \cdot \frac{\partial L}{\partial v_{12}} \\ \\ w_{11} & \leftarrow w_{11} - \eta \cdot \frac{\partial L}{\partial w_{11}} \\ w_{12} & \leftarrow w_{12} - \eta \cdot \frac{\partial L}{\partial w_{12}} \\ w_{21} & \leftarrow w_{21} - \eta \cdot \frac{\partial L}{\partial w_{21}} \\ w_{22} & \leftarrow w_{22} - \eta \cdot \frac{\partial L}{\partial w_{22}} \\ \\ b^h_1 & \leftarrow b^h_1 - \eta \cdot \frac{\partial L}{\partial b^h_1} \\ b^h_2 & \leftarrow b^h_2 - \eta \cdot \frac{\partial L}{\partial b^h_2} \\ \\ b^y_1 & \leftarrow b^y_1 - \eta \cdot \frac{\partial L}{\partial b^y_1} \end{align}\)

Numerical Simulation

Based on the MLP architecture and backpropagation steps described above, we can implement a simple numerical simulation demonstrate the training process of a multi-layer perceptron (MLP) using backpropagation.

Initizalization

The weight matrices and bias vectors are initialized as follows (randomically in \([0,1]\)):

Forward Pass

For the sample:

1. Compute hidden layer pre-activation:

   \[ \begin{array}{ll} \mathbf{z} &= \mathbf{W} \mathbf{x} + \mathbf{b}^h \\ &= \begin{bmatrix} 0.2 & 0.4 \\ 0.6 & 0.8 \end{bmatrix} \begin{bmatrix} 0.5 \\ 0.8 \end{bmatrix} + \begin{bmatrix} 0.1 \\ 0.2 \end{bmatrix} \\ &= \begin{bmatrix} 0.2*0.5 + 0.4*0.8 + 0.1 \\ 0.6*0.5 + 0.8*0.8 + 0.2 \end{bmatrix} \\ \mathbf{z} &= \begin{bmatrix} 0.52 \\ 1.14 \end{bmatrix} \end{array} \]

2. Compute hidden layer activations:

   \[ \begin{array}{ll} \mathbf{h} &= \sigma(\mathbf{z}) \\ &= \sigma \left( \begin{bmatrix} 0.52 \\ 1.14 \end{bmatrix} \right) \\ &= \begin{bmatrix} \displaystyle \frac{1}{1 + e^{-0.52}} \\ \displaystyle \frac{1}{1 + e^{-1.14}} \end{bmatrix} \\ \mathbf{h} &\approx \begin{bmatrix} 0.627 \\ 0.758 \end{bmatrix} \end{array} \]

3. Compute output layer pre-activation:

   \[ \begin{array}{ll} u &= \mathbf{V} \mathbf{h} + b^y \\ &= \begin{bmatrix} 0.3 & 0.5 \end{bmatrix} \begin{bmatrix} 0.627 \\ 0.758 \end{bmatrix} + 0.4 \\ &= 0.3*0.627 + 0.5*0.758 + 0.4 \\ u &\approx 0.967 \end{array} \]

4. Compute output layer activation:

   \[ \begin{array}{ll} \hat{y} &= \sigma(u) \\ &= \sigma(0.967) \\ &= \displaystyle \frac{1}{1 + e^{-0.967}} \\ \hat{y} &\approx 0.725 \end{array} \]

Loss Calculation

Using Mean Squared Error (MSE):

Backward Pass

1. Compute output layer gradients:

   \[ \begin{array}{ll} \displaystyle \delta_y = \frac{\partial L}{\partial u} &= \displaystyle \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial u} \\ &= 2(\hat{y} - y) \cdot \underbrace{\hat{y} (1 - \hat{y})}_{\sigma'(u)} \\ &\approx 2(0.725 - 0) \cdot 0.725 \cdot (1 - 0.725) \\ &\approx 0.289 \end{array} \]

   The gradient is positive: raising \( u \) would raise the loss, so the update below will lower it — which is what pushes \( \hat{y} \) towards the target \( y = 0 \).

2. Compute hidden layer gradients:

   \[ \begin{array}{ll} \boldsymbol{\delta}_h &= \displaystyle \frac{\partial L}{\partial \mathbf{z}} = \underbrace{\left( \delta_y \, \mathbf{V}^T \right)}_{\text{back through } \mathbf{V}} \odot \underbrace{\mathbf{h} \odot (1 - \mathbf{h})}_{\sigma'(\mathbf{z})} \\ &\approx 0.289 \cdot \begin{bmatrix} 0.3 \\ 0.5 \end{bmatrix} \odot \begin{bmatrix} 0.627 \\ 0.758 \end{bmatrix} \odot \begin{bmatrix} 0.373 \\ 0.242 \end{bmatrix} \\ &\approx \begin{bmatrix} 0.087 \\ 0.145 \end{bmatrix} \odot \begin{bmatrix} 0.234 \\ 0.184 \end{bmatrix} \\ &\approx \begin{bmatrix} 0.020 \\ 0.027 \end{bmatrix} \end{array} \]

   where \( \odot \) is the element-wise product.

3. Compute weight gradients — the outer product of the deltas with the layer input:

   \[ \begin{array}{ll} \displaystyle \frac{\partial L}{\partial \mathbf{W}} &= \boldsymbol{\delta}_h \, \mathbf{x}^T \\ &\approx \begin{bmatrix} 0.020 \\ 0.027 \end{bmatrix} \begin{bmatrix} 0.5 & 0.8 \end{bmatrix} \\ &\approx \begin{bmatrix} 0.010 & 0.016 \\ 0.013 & 0.021 \end{bmatrix} \end{array} \]
   \[ \begin{array}{ll} \displaystyle \frac{\partial L}{\partial \mathbf{V}} &= \delta_y \, \mathbf{h}^T \\ &\approx 0.289 \cdot \begin{bmatrix} 0.627 & 0.758 \end{bmatrix} \\ &\approx \begin{bmatrix} 0.181 & 0.219 \end{bmatrix} \end{array} \]

4. Compute bias gradients — a bias always sees an input of 1, so its gradient is the delta itself:

   \[ \frac{\partial L}{\partial b^y} = \delta_y \approx 0.289, \qquad \frac{\partial L}{\partial \mathbf{b}^h} = \boldsymbol{\delta}_h \approx \begin{bmatrix} 0.020 \\ 0.027 \end{bmatrix} \]

5. Update the parameters:

   \[ \begin{array}{ll} \mathbf{W} &\leftarrow \mathbf{W} - \eta \displaystyle \frac{\partial L}{\partial \mathbf{W}} \\ &\leftarrow \displaystyle \begin{bmatrix} 0.2 & 0.4 \\ 0.6 & 0.8 \end{bmatrix} - 0.7 \cdot \displaystyle \begin{bmatrix} 0.010 & 0.016 \\ 0.013 & 0.021 \end{bmatrix} \\ &\leftarrow \begin{bmatrix} 0.193 & 0.389 \\ 0.591 & 0.785 \end{bmatrix} \\ \mathbf{V} &\leftarrow \mathbf{V} - \eta \displaystyle \frac{\partial L}{\partial \mathbf{V}} \\ &\leftarrow \displaystyle \begin{bmatrix} 0.3 & 0.5 \end{bmatrix} - 0.7 \cdot \displaystyle \begin{bmatrix} 0.181 & 0.219 \end{bmatrix} \\ &\leftarrow \begin{bmatrix} 0.173 & 0.347 \end{bmatrix}\\ \\ b^y &\leftarrow b^y - \eta \displaystyle \frac{\partial L}{\partial b^y} \\ &\leftarrow 0.4 - 0.7 \cdot 0.289 \\ &\leftarrow 0.198 \\ \mathbf{b}^h &\leftarrow \mathbf{b}^h - \eta \displaystyle \frac{\partial L}{\partial \mathbf{b}^h} \\ &\leftarrow \begin{bmatrix} 0.1 \\ 0.2 \end{bmatrix} - 0.7 \cdot \begin{bmatrix} 0.020 \\ 0.027 \end{bmatrix} \\ &\leftarrow \begin{bmatrix} 0.086 \\ 0.181 \end{bmatrix} \end{array} \]

   Feeding the same sample through the updated network gives \( \hat{y} \approx 0.638 \) and \( L \approx 0.407 \), down from \( 0.525 \) — one step of gradient descent did what it promised.

6. Repeat the training process for each sample or multiple epochs. About the training process, there are two main approaches: online learning and batch learning:

   • Online learning is a method of training multi-layer perceptrons (MLPs) where the model is updated after each training example. This approach allows for faster convergence and can be more effective in scenarios with large datasets or when the data is not stationary.

   • Batch learning, on the other hand, involves updating the model after processing a batch of training examples. This method can lead to more stable updates and is often used in practice due to its efficiency in utilizing computational resources.

Additional

For a more intuitive understanding of neural networks, I highly recommend the following video series by 3Blue1Brown, which provides excellent visual explanations of the concepts: https://www.3blue1brown.com/lessons/neural-networks

Interactive: MLP Forward Pass Visualizer

Watch activations propagate through a 2-input → 3-hidden → 2-output network. Drag the input sliders and see the values flow layer by layer.