Why your model fails: understanding the bias-variance tradeoff with runnable plots

Machine learning models rarely fail because they are too simple or too complex by accident. More often, they fail because they learn the wrong amount of information from data.

A model with too much bias misses important patterns. A model with too much variance memorizes details that do not generalize. The tension between these two problems is the bias-variance tradeoff — one of the most important ideas in MachineLearning and a foundation of practical model development.

This guide explains the concept from first principles, then builds a runnable scikit-learn example that visualizes underfitting, overfitting, and the sweet spot in between.

The core idea: learning patterns without memorizing noise

Suppose you are training a model to predict house prices.

The training data contains:

• Real patterns:

  • Location affects price.

  • Size affects price.

  • Number of rooms affects price.

• Noise:

  • A seller’s unusual motivation.

  • A temporary market fluctuation.

  • Measurement errors.

A useful model learns the patterns. A problematic model learns the noise.

The bias-variance tradeoff describes this balance:

Problem	Meaning	Typical symptom
High bias	Model is too simple	Poor performance on training and test data
High variance	Model is too sensitive to training data	Excellent training performance but poor test performance

The goal is not the lowest training error. The goal is the lowest error on new, unseen data.

Bias: when a model is too simple

Bias is the error caused by incorrect assumptions in the learning algorithm.

Examples:

• A linear model trying to fit a highly nonlinear relationship.

• A decision tree restricted to only one level.

• A neural network with too little capacity for a complex task.

A high-bias model usually has:

• High training error.

• High validation error.

• Similar performance across datasets.

Imagine trying to draw a curve through a set of points using only a straight line. The model may capture the general direction, but it cannot represent the real shape.

Variance: when a model memorizes the training set

Variance measures how much a model changes when trained on slightly different datasets.

A high-variance model:

• Fits the training examples extremely well.

• Captures random fluctuations.

• Performs poorly on new data.

Common causes:

• Too many features compared with the amount of data.

• An overly complex model.

• Too little regularization.

• Training for too long in some model types.

A classic example is a deep decision tree. It can keep splitting until it creates almost one rule per training example.

The bias-variance curve

The relationship usually looks like this:

• Very simple models:

  • High bias.

  • Low variance.

• Very complex models:

  • Low bias.

  • High variance.

• Medium complexity:

  • Better generalization.

The best model is usually somewhere in the middle.

This does not mean “always choose a medium-sized model.” It means use validation data, cross-validation, and evaluation metrics to find the right level of complexity.

A practical example with scikit-learn

The following example creates synthetic nonlinear data, trains polynomial regression models with different complexity levels, and plots their behavior.

It uses current scikit-learn APIs:

import numpy as np
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error


rng = np.random.RandomState(42)

X = np.sort(6 * rng.rand(80, 1) - 3, axis=0)
y = 0.5 * X[:, 0] ** 2 + np.sin(X[:, 0]) + rng.normal(
    scale=1.0,
    size=X.shape[0]
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.35,
    random_state=42
)

degrees = [1, 3, 10, 20]

plt.figure(figsize=(12, 8))

x_plot = np.linspace(X.min(), X.max(), 300).reshape(-1, 1)

for degree in degrees:
    model = make_pipeline(
        PolynomialFeatures(degree=degree),
        Ridge(alpha=0.1)
    )

    model.fit(X_train, y_train)

    train_error = mean_squared_error(
        y_train,
        model.predict(X_train)
    )

    test_error = mean_squared_error(
        y_test,
        model.predict(X_test)
    )

    print(
        f"Degree {degree}: "
        f"train MSE={train_error:.3f}, "
        f"test MSE={test_error:.3f}"
    )

    plt.plot(
        x_plot,
        model.predict(x_plot),
        label=f"Degree {degree}"
    )

plt.scatter(
    X_train,
    y_train,
    color="black",
    label="Training data"
)

plt.xlabel("Feature")
plt.ylabel("Target")
plt.title("Bias-Variance Tradeoff Example")
plt.legend()
plt.show()

What you should observe:

• Degree 1 usually underfits:

  • The curve is too rigid.

  • Bias is high.

• Moderate complexity captures the pattern:

  • Better balance.

• Very high complexity becomes unstable:

  • The model starts following noise.

How to diagnose bias versus variance

When your model performance is disappointing, compare training and validation results.

High bias pattern

Example:

• Training accuracy: 75%

• Validation accuracy: 74%

The model performs poorly everywhere.

Possible fixes:

• Use a more powerful model.

• Add useful features.

• Reduce excessive regularization.

• Train longer if the model has not converged.

High variance pattern

Example:

• Training accuracy: 99%

• Validation accuracy: 80%

The model learned the training data too specifically.

Possible fixes:

• Collect more data.

• Reduce model complexity.

• Apply regularization.

• Use feature selection.

• Use cross-validation.

Regularization: controlling variance

Regularization intentionally limits model flexibility.

For linear models, common approaches include:

• L1 regularization:

  • Encourages some weights to become exactly zero.

  • Useful for feature selection.

• L2 regularization:

  • Reduces large weights.

  • Often improves generalization.

In the example above, Ridge applies L2 regularization:

from sklearn.linear_model import Ridge

model = Ridge(alpha=1.0)

model.fit(
    X_train,
    y_train
)

Increasing alpha generally creates a simpler model. Decreasing it allows the model to fit the training data more closely.

The correct value is usually selected with validation or cross-validation rather than guessing.

The surprising overfitting story: a Kaggle competition lesson

A famous example of overfitting happened during the Netflix Prize competition, where teams competed to improve movie recommendation accuracy.

One of the key lessons from the competition was that repeatedly tuning against a public leaderboard could cause teams to optimize for the leaderboard itself rather than the underlying problem.

This phenomenon appears frequently in machine learning competitions, including Kaggle competitions:

• A model may improve on the public leaderboard.

• The improvement may come from exploiting quirks in the public evaluation set.

• The model may perform worse on truly unseen data.

This is a practical example of variance at the workflow level: not just a model overfitting its training examples, but a human repeatedly adapting to a fixed benchmark.

The solution is the same idea behind good ML practice:

• Keep a private test set.

• Avoid repeatedly checking final evaluation data.

• Use proper cross-validation.

• Treat benchmarks as measurements, not targets to memorize.

Bias-variance tradeoff in modern machine learning

Although the term comes from classical statistics, the tradeoff still matters in modern systems:

• Large language models:

  • Need enough capacity to represent complex patterns.

  • Need training strategies that prevent memorizing undesirable data.

• Computer vision:

  • Deep networks can fit millions of parameters.

  • Data augmentation and regularization help generalization.

• Tabular ML:

  • Tree ensembles often perform well because they balance flexibility and robustness.

Modern tools do not remove the bias-variance tradeoff. They provide better ways to manage it.

A practical checklist for model improvement

When a model is not performing well:

If training and validation scores are both bad

Check for high bias:

• Add better features.

• Increase model capacity.

• Reduce overly strong regularization.

• Verify that the problem is learnable.

If training is excellent but validation is poor

Check for high variance:

• Simplify the model.

• Add more training examples.

• Tune regularization.

• Remove noisy features.

• Use stronger validation methods.

If results vary a lot

Check your evaluation process:

• Use multiple folds.

• Fix random seeds during experiments.

• Avoid tuning repeatedly on the final test set.

Key takeaways

The bias-variance tradeoff is not about choosing between simple and complex models. It is about finding the level of complexity that captures real patterns without memorizing noise.

Remember:

• High bias means the model is too limited.

• High variance means the model is too sensitive.

• Validation data reveals whether your model generalizes.

• Regularization and careful evaluation are essential tools.

Understanding BiasVariance is one of the most valuable Fundamentals skills for anyone building reliable machine learning systems.

Continue learning

Build your intuition by changing the polynomial degree, regularization strength, and dataset size in the example above. Plot the results, observe the errors, and experiment until you can recognize underfitting and overfitting by sight.

The fastest way to master the bias-variance tradeoff is to train models, measure them, and learn from what they do.