8. Regularization

Regularization techniques to prevent overfitting in MLPs, such as dropout, L1 and L2 regularization, batch normalization, and early stopping.

Dropout

Dropout¹ is a regularization technique where, during training, a random subset of neurons (or their connections) is "dropped" (set to zero) in each forward and backward pass. This prevents the network from relying too heavily on specific neurons.

During training, each neuron has a probability \( p \) (typically 0.2 to 0.5) of being dropped. This forces the network to learn redundant representations, making it more robust and less likely to memorize the training data. At test time, all neurons are active, but their weights are scaled by \( 1-p \) to account for the reduced activation during training.

Dropout acts like training an ensemble of smaller subnetworks, reducing co-dependency between neurons. It introduces noise, making the model less sensitive to specific patterns in the training data, thus improving generalization.

Practical tips

• Common dropout rates: 20–50% for hidden layers, lower (10–20%) for input layers.

• Use in deep networks, especially in fully connected layers or convolutional neural networks (CNNs). Avoid dropout in the output layer or when the network is small (it may hurt performance).

L1 and L2 Regularizations

• L1 regularization (Lasso) adds a penalty term to the loss function based on the absolute values of the model’s weights, encouraging sparsity in the weight matrix. This means that some weights can become exactly zero, effectively performing feature selection.

  The loss function is modified to include an L1 penalty:

  \[\text{Loss} = \text{Original Loss} + \lambda \sum |w_i|\]

  where \( w_i \) are the model’s weights, and \( \lambda \) (regularization strength) controls the penalty’s impact. During training, this encourages the model to focus on the most important features, potentially improving generalization.

• L2 regularization (Weight Decay) adds a penalty term to the loss function based on the magnitude of the model’s weights, discouraging large weights that can lead to complex, overfitted models.

  The loss function is modified to include an L2 penalty:

  \[\text{Loss} = \text{Original Loss} + \lambda \sum w_i^2\]

  where \( w_i \) are the model’s weights, and \( \lambda \) (regularization strength) controls the penalty’s impact.

  During optimization, this penalty encourages smaller weights, simplifying the model.

  Large weights amplify small input changes, leading to overfitting. L2 regularization constrains weights, making the model smoother and less sensitive to noise. It effectively reduces the model’s capacity to memorize training data.

Key Differences

• L1: Encourages sparse models (some weights = 0), useful for feature selection or when you want a simpler model with fewer active connections.

• L2: Produces small but non-zero weights, often leading to better generalization in many cases, especially in deep neural networks.

Practical tips

• Common \( \lambda \): \( 10^{-5} \) to \( 10^{-2} \), tuned via cross-validation.
• Works well in linear models, fully connected NNs, and CNNs.
• Combine with other techniques (e.g., dropout) for better results.
• The regularization affects the weights (not biases, typically) of the chosen layers.

Batch Normalization

Batch normalization² is a technique to improve the training of deep neural networks by normalizing the inputs to each layer. It reduces internal covariate shift, allowing for faster training and potentially better performance.

During training, batch normalization standardizes the inputs to a layer for each mini-batch:

where \( \mu \) is the batch mean, \( \sigma \) is the batch standard deviation, and \( \epsilon \) is a small constant for numerical stability.

After normalization, the layer can learn a scale ( \( \gamma \) ) and shift ( \( \beta \) ) parameter:

This allows the model to maintain the representational power while benefiting from the normalization.

Practical tips

• Use batch normalization after convolutional layers or fully connected layers.
• It can be used with other regularization techniques (e.g., dropout) for improved generalization.
• Consider using it in the training phase only, and not during inference, to avoid introducing noise.

Early Stopping

Early stopping is a regularization technique used to prevent overfitting in machine learning models, particularly in deep learning. The idea is to monitor the model's performance on a validation set during training and stop the training process once the performance starts to degrade.

The key steps involved in early stopping are:

1. Split the Data: Divide the dataset into training, validation, and test sets.
2. Monitor Performance: During training, periodically evaluate the model on the validation set and track the performance metric (e.g., accuracy, loss).
3. Set Patience: Define a patience parameter, which is the number of epochs to wait for an improvement in the validation performance before stopping training.
4. Stop Training: If the validation performance does not improve for a specified number of epochs (patience), stop the training process.

Early stopping helps to find a balance between underfitting and overfitting by allowing the model to train long enough to learn the underlying patterns in the data while preventing it from fitting noise.

Practical tips

• Use early stopping in conjunction with other regularization techniques (e.g., dropout, weight decay) for better results.
• Monitor multiple metrics (e.g., training loss, validation loss) to make informed decisions about stopping.

Additional