Overfitting on purpose: dropout, weight decay, and early stopping in one runnable lab

LEARN · NEURAL NETWORKS FROM THE GROUND UP

Machine learning tutorials often introduce overfitting as a failure mode: the model memorizes the training data, performs brilliantly on examples it has already seen, and then collapses on new data.

That description is correct, but incomplete.

Overfitting is also a diagnostic tool. A model that can overfit a dataset proves that it has enough capacity to learn meaningful patterns. The problem is not that the model learns too much; the problem is that it learns the wrong things too confidently.

In this lab, we will intentionally create an overpowered neural network, watch it memorize a small dataset, and then apply three of the most important regularization techniques used in modern deep learning:

dropout
weight decay
early stopping

By the end, you will have a repeatable workflow for answering a practical question:

Is my model underpowered, or is it learning too aggressively?

The entire experiment can run on a laptop using Python and PyTorch.

The mental model: learning signal versus memorization

A neural network does not understand the difference between useful patterns and accidental patterns.

Suppose you train a model to predict whether a customer will buy a product. A useful pattern might be:

customers who viewed several product pages are more likely to purchase

A meaningless pattern might be:

customers whose order IDs end in certain digits purchased more often in the training data

If the model has enough parameters, it can store these accidental correlations. This is memorization.

The goal of regularization is not to prevent learning. It is to encourage the model to discover patterns that generalize.

A useful way to think about training:

Too little capacity: the model cannot capture the real relationship.
Appropriate capacity: the model learns useful structure.
Too much capacity without constraints: the model memorizes noise.

The sweet spot is not always the smallest model. Modern deep learning often uses very large models combined with strong regularization.

Building a deliberately overfit experiment

Real-world datasets are often messy, large, and difficult to debug. For this experiment, we will create a synthetic retail-style dataset.

The dataset represents customers and their shopping behavior:

number of products viewed
time spent browsing
number of previous purchases
discount exposure
device type features

The target is a simple purchase score.

The important part is not the prediction problem itself. The important part is controlling the difficulty so we can observe overfitting clearly.

A small dataset plus a large neural network is a reliable recipe for producing the effect.

Project setup

A simple project layout:

regularization-lab/
├── train.py
├── requirements.txt
└── results/
    ├── baseline.png
    ├── dropout.png
    ├── weight_decay.png
    └── early_stopping.png

Create an environment and install dependencies:

python -m venv .venv

Activate it:

source .venv/bin/activate

On Windows:

.venv\Scripts\activate

Install packages:

pip install torch numpy matplotlib scikit-learn

Creating the dataset

We will generate neutral synthetic commerce data. There is no need for a huge dataset to demonstrate regularization.

The key idea is intentionally limiting training examples while giving the neural network plenty of parameters.

Create train.py:

import numpy as np
import torch
from torch import nn
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
torch.manual_seed(42)

samples = 600
features = 20

X = np.random.randn(samples, features)

true_weights = np.random.randn(features)

signal = X @ true_weights
noise = np.random.randn(samples) * 3.0

y = signal + noise

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=42
)

scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

X_train = torch.tensor(X_train, dtype=torch.float32)
X_test = torch.tensor(X_test, dtype=torch.float32)

y_train = torch.tensor(y_train, dtype=torch.float32).reshape(-1, 1)
y_test = torch.tensor(y_test, dtype=torch.float32).reshape(-1, 1)

Standardization matters because neural networks train more reliably when input features have similar scales.

Designing a network that can overfit

The model below is intentionally larger than necessary.

A small regression problem does not need hundreds of thousands of parameters. We are creating this imbalance on purpose.

class LargeNetwork(nn.Module):
    def __init__(self):
        super().__init__()

        self.layers = nn.Sequential(
            nn.Linear(features, 256),
            nn.ReLU(),
            nn.Linear(256, 256),
            nn.ReLU(),
            nn.Linear(256, 256),
            nn.ReLU(),
            nn.Linear(256, 1)
        )

    def forward(self, x):
        return self.layers(x)

The training objective is mean squared error:

loss = (prediction - target)² averaged over examples

In practice, PyTorch computes this with a built-in loss function:

criterion = nn.MSELoss()

Training the baseline model

First, we remove all regularization.

This gives us a reference point.

import matplotlib.pyplot as plt


def train_model(model, optimizer, epochs=300):
    train_history = []
    test_history = []

    criterion = nn.MSELoss()

    for epoch in range(epochs):
        model.train()

        optimizer.zero_grad()

        prediction = model(X_train)
        loss = criterion(prediction, y_train)

        loss.backward()
        optimizer.step()

        model.eval()

        with torch.no_grad():
            test_prediction = model(X_test)

            test_loss = criterion(
                test_prediction,
                y_test
            )

        train_history.append(loss.item())
        test_history.append(test_loss.item())

    return train_history, test_history


baseline = LargeNetwork()

optimizer = torch.optim.Adam(
    baseline.parameters(),
    lr=0.001
)

baseline_train, baseline_test = train_model(
    baseline,
    optimizer
)

plt.plot(baseline_train, label="train")
plt.plot(baseline_test, label="test")
plt.legend()
plt.xlabel("epoch")
plt.ylabel("MSE")
plt.title("Baseline")
plt.savefig("results/baseline.png")
plt.close()

You should observe a common pattern:

training loss keeps decreasing
test loss improves initially
test loss eventually starts increasing

That gap is the signature of overfitting.

The model is becoming better at reproducing the training examples while becoming worse at predicting unseen examples.

Dropout is one of the most recognizable regularization techniques in deep learning.

During training, dropout randomly disables a percentage of activations.

For example, with dropout probability 0.3:

each training pass randomly removes about 30% of selected neuron outputs
the network cannot rely on a single pathway
the model learns more distributed representations

The surprising part is that dropout makes the training problem harder. The network is forced to become more robust.

During evaluation, dropout is disabled.

Adding dropout layers

Modify the network:

class DropoutNetwork(nn.Module):
    def __init__(self):
        super().__init__()

        self.layers = nn.Sequential(
            nn.Linear(features, 256),
            nn.ReLU(),
            nn.Dropout(0.3),

            nn.Linear(256, 256),
            nn.ReLU(),
            nn.Dropout(0.3),

            nn.Linear(256, 256),
            nn.ReLU(),

            nn.Linear(256, 1)
        )

    def forward(self, x):
        return self.layers(x)

Train it:

dropout_model = DropoutNetwork()

optimizer = torch.optim.Adam(
    dropout_model.parameters(),
    lr=0.001
)

dropout_train, dropout_test = train_model(
    dropout_model,
    optimizer
)

The training curve may not look as impressive. That is expected.

A regularized model often has:

higher training error
lower validation error

The purpose is not winning on the training set. The purpose is winning on new data.

Weight decay takes a different approach.

Instead of randomly removing parts of the network, it discourages excessively large weights.

The optimizer effectively adds a penalty:

new objective = prediction error + weight penalty

The model is encouraged to use simpler solutions.

In PyTorch, AdamW is commonly used because it implements decoupled weight decay.

Training with weight decay

weight_decay_model = LargeNetwork()

optimizer = torch.optim.AdamW(
    weight_decay_model.parameters(),
    lr=0.001,
    weight_decay=0.01
)

weight_decay_train, weight_decay_test = train_model(
    weight_decay_model,
    optimizer
)

Weight decay is especially useful when:

the network is large
the dataset is not large enough
you want smoother model behavior
you are training transformer-based architectures

Many modern deep learning training recipes rely heavily on optimizer settings like learning rate schedules and weight decay.

Sometimes the best model is not the final model.

During training, performance often follows this pattern:

The model learns useful patterns.
Validation performance improves.
The model starts memorizing details.
Validation performance declines.

Early stopping simply keeps the best checkpoint.

Adding early stopping logic

def train_with_early_stopping(model, optimizer, patience=30):
    criterion = nn.MSELoss()

    best_loss = float("inf")
    best_state = None
    waiting = 0

    train_history = []
    test_history = []

    for epoch in range(300):

        model.train()

        optimizer.zero_grad()

        prediction = model(X_train)

        loss = criterion(
            prediction,
            y_train
        )

        loss.backward()
        optimizer.step()

        model.eval()

        with torch.no_grad():
            test_prediction = model(X_test)

            test_loss = criterion(
                test_prediction,
                y_test
            )

        train_history.append(loss.item())
        test_history.append(test_loss.item())

        if test_loss.item() < best_loss:
            best_loss = test_loss.item()
            best_state = model.state_dict()
            waiting = 0
        else:
            waiting += 1

        if waiting >= patience:
            break

    model.load_state_dict(best_state)

    return train_history, test_history

Run it:

early_model = LargeNetwork()

optimizer = torch.optim.Adam(
    early_model.parameters(),
    lr=0.001
)

early_train, early_test = train_with_early_stopping(
    early_model,
    optimizer
)

Early stopping is simple, but it is one of the highest-value techniques in practical machine learning.

These methods solve the same problem from different angles.

Technique	Main idea	Typical effect
Dropout	Remove random activations during training	Reduces reliance on specific neurons
Weight decay	Penalize large weights	Encourages simpler solutions
Early stopping	Stop before memorization dominates	Keeps the best generalizing model

They are often combined.

A production neural network might use:

dropout inside layers
AdamW with weight decay
early stopping based on validation metrics

A phenomenon called double descent challenges the traditional assumption that bigger models always overfit more.

The classic view:

small model → underfit
medium model → best performance
large model → overfit

Double descent adds another behavior:

small model → underfit
medium model → worst generalization region
very large model → sometimes improves again

Researchers observed this in modern deep learning systems where extremely large models can fit training data perfectly while still achieving strong test performance.

The explanation is still an active research area, but one idea is that sufficiently large models can find simpler, more stable solutions among many possible solutions.

This does not mean “always use the biggest model.” It means the relationship between capacity and generalization is more complicated than the old textbook curve suggests.

Machine learning systems are not only algorithms. They are software stacks.

A model training pipeline may include:

Python packages
image libraries
data processing tools
GPU drivers
serialization formats
deployment servers

A reminder came from Google Chrome and other software ecosystems when attackers exploited vulnerabilities in common dependencies.

One notable example was CVE-2023-4863, a heap buffer overflow in the widely used libwebp image library that was exploited in the wild. The vulnerability demonstrated that even a small dependency can become a major security boundary.

For machine learning teams, the lesson is practical:

pin dependency versions
monitor security advisories
avoid loading untrusted serialized models
scan containers before deployment
keep training environments reproducible

A well-regularized model is valuable, but a secure model pipeline is equally important.

When your model performs poorly, do not immediately add more layers.

Check:

Does training accuracy become excellent while validation accuracy stalls?
Does validation loss increase while training loss decreases?
Is the dataset too small?
Are labels noisy?
Are features leaking information from the future?
Is the model much larger than the data requires?

Then try:

Add a validation split.
Plot training and validation curves.
Reduce model size.
Add dropout.
Add weight decay.
Use early stopping.
Collect better data.

Regularization is not a magic button. It is a way of expressing a preference:

Prefer models that learn reusable patterns instead of memorizing examples.

Modify the lab and experiment:

Increase the network size and observe when overfitting begins.
Change dropout from 0.3 to 0.5.
Compare Adam with AdamW.
Add learning-rate scheduling.
Replace synthetic data with your own non-sensitive dataset.

The fastest way to understand regularization is not by memorizing definitions. Run the experiment, break the model on purpose, and then teach it how to generalize. Build this lab, inspect the curves, and use the lessons in your next machine learning project.