4. Perceptron

Biological Inspiration

Since the inception of artificial neural networks (ANNs), their design has been heavily influenced by the structure and function of biological neural networks. The human brain, with its complex network of neurons, serves as a foundational model for understanding how ANNs can process information. The base of an ANN is a neuron, which mimics the behavior of biological neurons. Each neuron receives inputs, processes them, and produces an output, similar to how biological neurons communicate through synapses.

The biological neuron consists of a cell body (soma), dendrites, and an axon. Dendrites receive signals from other neurons, the soma processes these signals, and the axon transmits the output to other neurons. This structure allows for complex interactions and information processing, which is essential for learning and decision-making in biological systems. The signal produced by a neuron is known as an action potential, which is an electrical impulse that travels along the axon to communicate with other neurons. The action potential is generated when the neuron receives sufficient input from its dendrites, leading to a change in the electrical potential across its membrane. This process is known as neural activation and is crucial for the functioning of both biological and artificial neurons.

Based on this biological inspiration, McCulloch and Pitts¹ proposed the first mathematical model of a neuron in 1943. This model laid the groundwork for the development of artificial neurons, which are the building blocks of ANNs. The McCulloch-Pitts neuron is a simple binary model that outputs a signal based on whether the weighted sum of its inputs exceeds a certain threshold.

This equation describes how the output of neuron \(N_i\) at time \(t+1\) is determined by the weighted sum of its inputs \(N_j\) at time \(t\), adjusted by a threshold \(\theta_i(t)\). The function \(H\) is a step function that activates the neuron if the input exceeds the threshold. Note, the results of the McCulloch-Pitts model are binary, meaning the output is either 0 or 1, which corresponds to the neuron being inactive or active, respectively - see a connection to symbolic logic.

This model, while simplistic, captures the essence of how neurons process information and has been foundational in the development of more complex neural network architectures.

Mathematical Foundations

The mathematical foundations of artificial neural networks are built upon linear algebra, calculus, and probability theory. These areas provide the tools necessary to understand how ANNs operate, how they learn from data, and how they can be optimized for various tasks.

The Perceptron, introduced by Rosenblatt² in 1958, is one of the earliest and simplest forms of an ANN. It consists of a single layer of neurons that can classify linearly separable data. The Perceptron algorithm adjusts the weights of the inputs based on the error in the output, allowing it to learn from examples.

The Perceptron can be mathematically described as follows:

The Perceptron computes the output \(y\) as follows:

A perceptron is a simple artificial neuron that takes multiple inputs, applies weights to them, sums them up, adds a bias, and passes the result through an activation function (typically a step function) to produce a binary output (e.g., 0 or 1). It’s used to solve linearly separable classification problems³.

The perceptron’s output is computed as:

• Input: a vector of features \( \mathbf{x} = [x_1, x_2, \dots, x_n] \).

• Weights: a vector \( \mathbf{w} = [w_1, w_2, \dots, w_n] \) representing the importance of each input.

• Bias: a scalar \( b \) that shifts the decision boundary.

• Output:

  \[ y = \text{activation}(\mathbf{w} \cdot \mathbf{x} + b) \]

  where \( \mathbf{w} \cdot \mathbf{x} = w_1x_1 + w_2x_2 + \dots + w_nx_n \), and the activation function is typically a step function:

  \[ \text{activation}(z) = \begin{cases} 1 & \text{if } z \geq 0 \\ 0 & \text{if } z < 0 \end{cases} \]

The goal of training is to find the optimal weights \( \mathbf{w} \) and bias \( b \) so the perceptron correctly classifies the training data.

Perceptron Training Process

The perceptron training algorithm adjusts the weights and bias iteratively based on errors in classification. Here’s a step-by-step explanation:

1. Initialize Weights and Bias

Start with small random values for the weights \( \mathbf{w} \) and bias \( b \), or initialize them to zero. These initial values don’t need to be perfect, as the algorithm will adjust them during training.

2. Provide Training Data

The training dataset consists of input-output pairs \( \{(\mathbf{x}^{(i)}, y^{(i)})\} \), where:

• \( \mathbf{x}^{(i)} \) is the feature vector for the \( i \)-th example.
• \( y^{(i)} \) is the true label (0 or 1).

The data must be linearly separable for the perceptron to converge (i.e., a straight line or hyperplane can separate the two classes).

3. Forward Pass: Compute Prediction

For each training example \( \mathbf{x}^{(i)} \):

Calculate the weighted sum:

\( z = \mathbf{w} \cdot \mathbf{x}^{(i)} + b \).

Apply the activation function to get the predicted output \( \hat{y}^{(i)} \):

Compare the predicted output \( \hat{y}^{(i)} \) with the true label \( y^{(i)} \).

4. Compute Error

The error is the difference between the true label and the predicted label:

Since \( y^{(i)} \) and \( \hat{y}^{(i)} \) are binary (0 or 1), the error can be:

• \( 0 \) \( \implies \) correct prediction (no update needed),
• \( 1 \) \( \implies \) predicted 0, but true label is 1 (1 - 0 = 1),
• \( -1 \) \( \implies \) predicted 1, but true label is 0 (0 - 1 = -1).

5. Update Weights and Bias

If the prediction is correct (\( \text{error} = 0 \)), no update is needed.

If the prediction is incorrect, adjust the weights and bias using the perceptron learning rule², an error-driven relative of Hebb's rule⁴:

where:

\( \eta \) (eta) is the learning rate, a small positive number (e.g., 0.01) that controls the size of the update.

The update moves the decision boundary to reduce the error for the current example.

6. Iterate

Repeat steps 3–5 for all training examples in the dataset (one pass through the dataset is called an epoch).

7. Convergence

If the data is linearly separable, the perceptron is guaranteed to converge to a solution that correctly classifies all training examples.

Intuition Behind the Training

• The perceptron learns by adjusting the decision boundary (a hyperplane defined by \( \mathbf{w} \cdot \mathbf{x} + b = 0 \)) to separate the two classes.
• Each weight update moves the hyperplane slightly to reduce misclassification errors.
• The learning rate \( \eta \) sets the size of each step, but for the perceptron it matters less than it seems: the hyperplane is \( \mathbf{w} \cdot \mathbf{x} + b = 0 \), and multiplying \( \mathbf{w} \) and \( b \) by a constant does not move it. Starting from zero weights, every update is \( \eta \) times what it would be with \( \eta = 1 \), so the boundaries, the mistakes and the number of updates are the same for any \( \eta > 0 \). Starting from random weights, \( \eta \) only sets how much the first updates weigh against that start.

Example: Training a Perceptron

Suppose we have a dataset with two features \( \mathbf{x} = [x_1, x_2] \) and binary labels (0 or 1). Let’s train a perceptron to classify points.

Dataset:

Step-by-Step:

1. Initialize: Set \( \mathbf{w} = [0, 0] \), \( b = 0 \), and \( \eta = 0.1 \).

2. First Example: \( \mathbf{x}^{(1)} = [1, 1] \), \( y^{(1)} = 1 \).

   • Compute: \( z = \underbrace{0}_{w_1} \cdot \underbrace{1}_{x_1} + \underbrace{0}_{w_2} \cdot \underbrace{1}_{x_2} + \underbrace{0}_{b} = 0 \), so \( \hat{y}^{(1)} = 1 \) (since \( z \geq 0 \)).
   • Error: \( y^{(1)} - \hat{y}^{(1)} = 1 - 1 = 0 \). No update needed.

3. Second Example: \( \mathbf{x}^{(2)} = [2, 2] \), \( y^{(2)} = 1 \).

   • Compute: \( z = \underbrace{0}_{w_1} \cdot \underbrace{2}_{x_1} + \underbrace{0}_{w_2} \cdot \underbrace{2}_{x_2} + \underbrace{0}_{b} = 0 \), so \( \hat{y}^{(2)} = 1 \).
   • Error: \( \underbrace{1}_{y^{(2)}} - \underbrace{1}_{\hat{y}^{(2)}} = 0 \). No update.

4. Third Example: \( \mathbf{x}^{(3)} = [-1, -1] \), \( y^{(3)} = 0 \).

   • Compute: \( z = \underbrace{0}_{w_1} \cdot \underbrace{-1}_{x_1} + \underbrace{0}_{w_2} \cdot \underbrace{-1}_{x_2} + \underbrace{0}_{b} = 0 \), so \( \hat{y}^{(3)} = 1 \).
   • Error: \( \underbrace{0}_{y^{(3)}} - \underbrace{1}_{\hat{y}^{(3)}} = -1 \).
   • Update:
   \[ \mathbf{w} = \underbrace{[0, 0]}_{\mathbf{w}} + \underbrace{0.1}_{\eta} \cdot \underbrace{-1}_{\text{error}} \cdot \underbrace{[-1, -1]}_{\mathbf{x}^{(3)}} = [0, 0] + [0.1, 0.1] = [0.1, 0.1] \]
   \[ b = \underbrace{0}_{b} + \underbrace{0.1}_{\eta} \cdot \underbrace{-1}_{\text{error}} = -0.1 \]

5. Fourth Example: \( \mathbf{x}^{(4)} = [-2, -1] \), \( y^{(4)} = 0 \).

   • Compute: \( z = \underbrace{0.1}_{w_1} \cdot \underbrace{-2}_{x_1} + \underbrace{0.1}_{w_2} \cdot \underbrace{-1}_{x_2} + \underbrace{-0.1}_{b} = -0.4 \), so \( \hat{y}^{(4)} = 0 \).

   • Error: \( \underbrace{0}_{y^{(4)}} - \underbrace{0}_{\hat{y}^{(4)}} = 0 \). No update.

6. Repeat: Continue iterating through the dataset, updating weights and bias when errors occur, until all examples are correctly classified or a maximum number of epochs is reached.

Visualizing the Decision Boundary

The decision boundary is the hyperplane where \( \mathbf{w} \cdot \mathbf{x} + b = 0 \). For a 2D dataset (\( x_1, x_2 \)), this is a line:

During training, the weights and bias are adjusted to move this line so that it separates the two classes correctly.

Practical Considerations

• Preprocessing: Normalize or standardize input features to ensure they’re on similar scales, which helps the perceptron learn more effectively.
• Epochs: Set a maximum number of epochs to prevent infinite loops if the data is not linearly separable.
• Extensions: To handle multi-class problems, you can use multiple perceptrons (one-vs-rest) or move to more advanced models like MLPs or support vector machines (SVMs).

Limitations of the Perceptron

While the perceptron is a foundational model in machine learning, it has several limitations:

1. Linearly Separable Data: The perceptron can only solve problems where the classes are linearly separable. If the data cannot be separated by a straight line (or hyperplane in higher dimensions), the perceptron will fail to converge. A classic example of this limitation is the XOR problem⁵, which cannot be solved by a single-layer perceptron.

2. Binary Classification: The basic perceptron is designed for binary classification tasks. While it can be extended to multi-class problems using techniques like one-vs-rest, it is not inherently suited for multi-class classification.

3. No Probability Estimates: The perceptron outputs binary decisions (0 or 1) without providing probability estimates for its predictions. This can be a limitation in applications where understanding the confidence of predictions is important.
4. No Margin: The perceptron stops at the first hyperplane that separates the training data, which can pass right next to some points. A maximum-margin classifier (a support vector machine) picks the hyperplane farthest from both classes, which usually generalizes better.

Interactive playground: train a perceptron

Click the area below to add points (plain click = class A, Shift+click = class B). The perceptron updates its weights at every epoch with Rosenblatt's learning rule.