Linear regression is not boring: assumptions, diagnostics, and when it beats deep learning

LEARN · CLASSICAL MACHINE LEARNING

Why linear regression still matters

If you’ve been around machine learning for a while, you’ve probably seen linear regression dismissed as the “hello world” of ML. That sells it short.

LinearRegression remains one of the strongest baselines in ClassicalML. It’s fast, interpretable, requires almost no hyperparameter tuning, and often performs surprisingly well on structured, tabular datasets.

More importantly, learning why linear regression works builds intuition that transfers to every predictive model you’ll use later.

In this guide, you’ll learn:

What linear regression actually assumes
How to train a model with scikit-learn
How to diagnose problems using residuals
When linear regression is the right tool
When a more sophisticated model is worth the extra complexity

What is linear regression?

Linear regression predicts a continuous target by modeling it as a weighted sum of input features.

The prediction is:

prediction = intercept + weight₁ × feature₁ + weight₂ × feature₂ + ...

Training finds the coefficients that minimize the sum of squared errors (SSE):

SSE = Σ(actual − predicted)²

For ordinary least squares, the solution is well understood and efficiently computed, making training extremely fast even on reasonably large datasets.

When linear regression shines

Linear regression is often an excellent choice when:

Relationships are approximately linear
Interpretability matters
The dataset is relatively small or medium-sized
Training speed is important
You need a trustworthy baseline before trying more complex models

Typical applications include:

House price prediction
Sales forecasting
Energy consumption estimation
Manufacturing quality control
Scientific experiments
Financial risk modeling

Even if you eventually deploy gradient-boosted trees or a neural network, comparing against a linear baseline is considered good machine learning practice.

The assumptions that actually matter

Many people memorize the assumptions without understanding why they matter. Here’s the practical interpretation.

1. Linearity

The relationship between features and the target should be approximately linear.

Good examples:

House size → price
Advertising spend → sales
Years of experience → salary (within a reasonable range)

Less suitable:

Image classification
Natural language processing
Highly nonlinear physical systems

If the relationship curves significantly, feature engineering or nonlinear models may work better.

2. Independent observations

Each row should represent an independent sample.

Usually fine:

Different houses
Different customers
Different patients

Potential problem:

The same sensor sampled every second

Time-series data often violates this assumption because nearby observations are correlated.

3. Constant error variance (homoscedasticity)

Prediction errors should have roughly the same spread across the prediction range.

A funnel-shaped residual plot often indicates increasing prediction uncertainty as predictions grow.

Predictions may still be useful, but statistical inference becomes less reliable.

4. Residuals should look random

Residuals are simply:

Residual = actual − predicted

A good residual plot resembles random noise centered around zero.

Watch for:

Curved patterns
Funnel shapes
Distinct clusters
Systematic bias

Patterns usually mean the model is missing important structure.

5. Limited multicollinearity

Features shouldn’t carry nearly identical information.

For example:

Height in centimeters
Height in inches

Highly correlated features can make coefficients unstable even when prediction quality remains acceptable.

Build your first model with scikit-learn

The following example uses the California Housing dataset bundled with scikit-learn.

If needed, install or upgrade scikit-learn first:

pip install -U scikit-learn

You can also verify your installed version:

import sklearn

print(sklearn.__version__)

Now train a simple linear regression model:

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

housing = fetch_california_housing(as_frame=True)

X = housing.data
y = housing.target

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)

mae = mean_absolute_error(y_test, predictions)
rmse = mean_squared_error(y_test, predictions) ** 0.5
r2 = r2_score(y_test, predictions)

print(f"MAE : {mae:.3f}")
print(f"RMSE: {rmse:.3f}")
print(f"R²  : {r2:.3f}")

Each metric answers a different question:

MAE: Average prediction error.
RMSE: Penalizes large mistakes more heavily.
R²: Fraction of variance explained.

Don’t rely on just one metric.

Understanding the coefficients

One of linear regression’s biggest strengths is interpretability.

import pandas as pd

coefficients = (
    pd.Series(model.coef_, index=X.columns)
    .sort_values(key=lambda s: s.abs(), ascending=False)
)

print(coefficients)

Keep these caveats in mind:

Positive coefficients increase the prediction.
Negative coefficients decrease the prediction.
Correlation is not causation.
Correlated features can distort coefficient values.
Scaling changes coefficient magnitudes but not the underlying predictions when applied consistently.

Residual diagnostics: the most overlooked step

Many beginners stop after computing R².

That’s a mistake.

Residual analysis often reveals problems that summary metrics hide.

import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

housing = fetch_california_housing(as_frame=True)

X_train, X_test, y_train, y_test = train_test_split(
    housing.data,
    housing.target,
    test_size=0.2,
    random_state=42,
)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)
residuals = y_test - predictions

plt.figure(figsize=(6, 4))
plt.scatter(predictions, residuals, alpha=0.5)

plt.axhline(0, color="red", linestyle="--")

plt.xlabel("Predicted value")
plt.ylabel("Residual")
plt.title("Residual plot")

plt.tight_layout()
plt.show()

You’re looking for a roughly random cloud centered around zero.

If you see clear structure, the model is probably missing nonlinear relationships or important features.

Common beginner mistakes

Ignoring feature engineering

Linear models can capture surprisingly rich relationships when you provide informative features.

Useful transformations include:

Squared terms
Log transforms
Ratios
Interaction features

A well-engineered linear model can outperform a poorly tuned complex model.

Trusting R² alone

Always inspect:

MAE
RMSE
Residual plots
Performance on unseen data

A high R² does not automatically mean the model is useful.

Forgetting train/test splits

Evaluating on training data produces overly optimistic results.

Always reserve unseen data or use cross-validation.

Assuming coefficients imply causality

Linear regression discovers statistical relationships.

It does not prove that changing one variable causes another to change.

Cherry on the cake: simple baselines still surprise people

One recurring lesson from recent tabular machine learning benchmarks—including updates and follow-up analyses built on the widely used TabZilla benchmark suite published over the past few years—is that deep neural networks are not consistently the best choice for structured tabular data. Depending on the dataset, carefully tuned classical methods can match or outperform them, and a well-designed linear baseline is often much stronger than newcomers expect.

The takeaway isn’t that linear regression always wins—it doesn’t.

The lesson is more practical:

Always establish a strong baseline before reaching for a more complex model.

If a sophisticated model improves your error by only a tiny amount while costing far more to train, deploy, and explain, the simpler model may be the better engineering decision.

When should you move beyond linear regression?

Linear regression is an excellent starting point, but not always the finish line.

Consider more advanced models when:

Residual plots show strong nonlinear patterns.
Prediction accuracy plateaus.
Feature interactions dominate.
Relationships are clearly non-additive.

Natural next steps include:

Ridge Regression
Lasso Regression
Elastic Net
Random Forests
HistGradientBoostingRegressor
XGBoost
LightGBM
CatBoost

Notice that the first three are still linear models—they simply improve robustness through regularization.

A practical workflow

A disciplined workflow looks like this:

Explore the data.
Train a LinearRegression baseline.
Measure MAE, RMSE, and R².
Inspect residual plots.
Engineer better features.
Re-evaluate.
Only then compare against more sophisticated models.

This process saves time, improves interpretability, and gives every future model a meaningful benchmark to beat.

Key takeaways

Linear regression is far more than an introductory algorithm.

It teaches core machine learning skills that remain valuable throughout your career:

Build strong baselines first.
Understand model assumptions.
Diagnose failures using residuals.
Interpret coefficients responsibly.
Add complexity only when the data justifies it.

Many production systems still rely on linear models because they are fast, transparent, easy to debug, and surprisingly effective on structured datasets.

The next time you start a regression project, begin with a LinearRegression baseline, inspect the residuals, improve your features, and let the evidence—not the hype—decide whether a more complex model is actually necessary.

What’s next?

Run the examples in this article on one of your own datasets, inspect the residual plots, then compare the results with a tree-based model. You’ll build stronger intuition, create better baselines, and make more informed modeling decisions from the very beginning.