Data leakage: the silent killer of ML projects

Machine learning projects rarely fail because the algorithm is not advanced enough. More often, they fail because the model learned from information it would never have at prediction time.

This problem is called data leakage.

Data leakage happens when information from outside the intended prediction workflow accidentally enters training, validation, or feature engineering. The result is a model that looks excellent in experiments but breaks when exposed to real-world data.

A leaked model is dangerous because it creates confidence. Dashboards look green, metrics look impressive, and teams may ship a system that has learned shortcuts instead of useful patterns.

This guide covers:

• what data leakage is and why it matters

• the most common leakage patterns

• a runnable example that creates and fixes leakage

• MLOps practices that prevent leakage

• a real-world leakage lesson from healthcare ML

What is data leakage?

The core rule is simple:

A feature can only be used if it exists at the exact moment the prediction is made.

Imagine building a customer churn model.

Valid features:

• account age

• historical product usage

• previous support requests

• past billing activity

Leaked features:

• cancellation reason entered after leaving

• account closure date

• refund approval created after cancellation

The leaked features do not predict churn. They reveal it.

The model is not learning customer behavior. It is learning information that became available after the outcome already happened.

Why leakage is so dangerous

Most software bugs announce themselves:

• a service crashes

• a deployment fails

• an API returns errors

Data leakage often does the opposite. It produces a successful-looking result.

Common warning signs:

• validation accuracy is suspiciously high

• a simple model performs better than expected

• production results are much worse than experiments

• important features seem too closely related to the target

• performance disappears when new data arrives

The most difficult leakage problems are hidden in ordinary engineering decisions:

• a database join

• a timestamp field

• a preprocessing step

• a feature aggregation job

The important question is always:

Would this information exist when the prediction is made?

Four common types of ML leakage

1. Train-test contamination

A classic mistake is allowing the test set to influence training.

Incorrect workflow:

1. Load all data.

2. Normalize all features.

3. Split into training and testing data.

4. Train the model.

The scaler already learned information from the test data.

Correct workflow:

1. Split the data.

2. Fit preprocessing only on training data.

3. Apply those transformations to validation and test data.

4. Evaluate the model.

This is why production ML systems often rely on pipelines: they make the correct order easier to enforce.

2. Target leakage

Target leakage happens when a feature contains information created from the label.

Example:

A hospital wants to predict whether a patient will develop a serious infection.

A feature called:

antibiotics_prescribed

may look useful.

However, if antibiotics are prescribed after clinicians recognize the condition, the feature is not an early predictor. It is a delayed signal of the answer.

The model is not predicting the future. It is reading the outcome from the past.

3. Temporal leakage

Time-dependent problems require strict boundaries.

Common examples:

• forecasting

• fraud detection

• recommendations

• demand prediction

A bad split:

• training data: random samples from all months

• test data: random samples from all months

This allows future information to influence past predictions.

A better split:

• training: older historical data

• validation: later period

• testing: newest unseen period

Your evaluation should imitate how the model will run in production.

4. Group leakage

Sometimes the same entity appears multiple times.

Examples:

• multiple medical records from one patient

• multiple images from one person

• repeated transactions from one customer

A random split can put related examples in both training and testing.

The model then learns identity-specific patterns instead of general behavior.

A practical leakage example

The following example creates a fake churn dataset.

The mistake is the account_closed_days feature. It is created after a customer leaves, but the model treats it as if it was available before the churn prediction.

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

data = pd.DataFrame(
    {
        "monthly_usage": [90, 10, 85, 20, 75, 15, 95, 25],
        "support_calls": [1, 8, 2, 7, 1, 9, 0, 6],
        "account_closed_days": [0, 5, 0, 7, 0, 10, 0, 4],
        "churned": [0, 1, 0, 1, 0, 1, 0, 1],
    }
)

X = data.drop(columns=["churned"])
y = data["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42,
    stratify=y,
)

model = RandomForestClassifier(random_state=42)

model.fit(X_train, y_train)

predictions = model.predict(X_test)

print("Accuracy:", accuracy_score(y_test, predictions))

The score may look excellent because account_closed_days effectively gives the model the answer.

A real version of this mistake can happen when teams combine data from different systems:

• CRM data

• billing systems

• support platforms

• analytics databases

A timestamp or status field added later can quietly become a leakage source.

Fixing the leakage

The solution is not a larger model.

The solution is removing information that would not exist at prediction time.

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

data = pd.DataFrame(
    {
        "monthly_usage": [90, 10, 85, 20, 75, 15, 95, 25],
        "support_calls": [1, 8, 2, 7, 1, 9, 0, 6],
        "account_closed_days": [0, 5, 0, 7, 0, 10, 0, 4],
        "churned": [0, 1, 0, 1, 0, 1, 0, 1],
    }
)

features = [
    "monthly_usage",
    "support_calls",
]

X = data[features]
y = data["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42,
    stratify=y,
)

model = RandomForestClassifier(random_state=42)

model.fit(X_train, y_train)

predictions = model.predict(X_test)

print("Accuracy:", accuracy_score(y_test, predictions))

The score may decrease.

That is a good outcome.

A realistic evaluation score is more valuable than a perfect score based on impossible information.

Use pipelines to reduce leakage risk

Modern ML workflows should combine preprocessing and modeling into one pipeline.

A pipeline ensures transformations are fitted only on training data during evaluation.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y,
)

pipeline = Pipeline(
    [
        ("scaler", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=1000)),
    ]
)

pipeline.fit(X_train, y_train)

print("Test score:", pipeline.score(X_test, y_test))

The pipeline enforces the correct sequence:

• fit the scaler using training data

• transform training data

• train the model

• transform test data using the same scaler

• evaluate

This pattern is easier to review and harder to break accidentally.

How MLOps teams prevent leakage

Leakage prevention is an engineering practice, not just a modeling technique.

Define the prediction moment

Before creating features, document:

• when the prediction happens

• what information exists at that moment

• what information arrives later

This creates a clear boundary for feature engineering.

Track feature ownership

Production features should have:

• data source

• owner

• creation timestamp

• update frequency

• business definition

• availability delay

A feature without clear ownership is a future leakage risk.

Match evaluation to production

Use realistic validation strategies:

• forecasting → time-based splits

• customer models → customer-level splits

• medical imaging → patient-level splits

• fraud detection → future transaction testing

Investigate unusually good results

Unexpected performance is a debugging signal.

Review:

• feature definitions

• timestamps

• database joins

• label creation logic

• duplicate entities

The problem may not be the algorithm. It may be the dataset.

Cherry on the cake: a healthcare model lesson

One of the clearest leakage discussions involved the Epic Sepsis Model, a widely deployed healthcare prediction system.

Researchers examining machine learning practices have highlighted a leakage concern: using variables such as whether antibiotics were prescribed can introduce information that typically occurs after clinicians recognize or treat sepsis. Those variables can act as proxies for the outcome rather than true early warning signals.

Independent evaluation of the Epic Sepsis Model also found substantially lower performance than originally reported, showing why real-world validation matters before deploying clinical prediction systems.

The broader lesson applies everywhere:

A model can achieve impressive metrics while solving the wrong problem.

A leakage checklist before deployment

Before shipping a model, ask:

• Can every feature exist at prediction time?

• Was preprocessing fitted only on training data?

• Are time boundaries respected?

• Can the same entity appear in training and testing?

• Are labels created independently from features?

• Does performance remain realistic on future data?

• Has someone outside the modeling team reviewed the feature list?

If any answer is unclear, investigate before deployment.

Final thoughts

Data leakage is one of the most expensive ML mistakes because it creates confidence without capability.

Strong ML systems are not built by chasing the highest validation score. They are built by designing evaluation environments that match reality.

Build models that learn from the past, predict the future, and use only information that truly exists at decision time.

Start today: audit one dataset in your ML pipeline, document the prediction moment, review every feature for leakage risk, and make realistic evaluation practices a standard part of your MLOps workflow.