Missing data is a message: patterns, mechanisms, and honest imputation

LEARN · EXPLORATORY DATA ANALYSIS & STATISTICS

Why missing data deserves first-class attention

Missing values are often treated as an inconvenience: fill them, drop them, move on.

That approach can quietly introduce bias, erase important signals, or even reverse your conclusions.

A better mindset is this:

Missing data is data.

The fact that a value is missing may itself contain valuable information about the process that generated your dataset. Before choosing an imputation strategy, understand why values are missing, where they occur, and how your downstream analysis will be be affected.

In this guide, you’ll learn how to:

Recognize missingness patterns
Understand the three missing-data mechanisms
Visualize missing values
Decide when dropping rows is appropriate
Compare simple and model-based imputation
Preserve missingness as a predictive feature
Avoid data leakage
Build reproducible preprocessing pipelines

This article uses current Python tooling (tested against the 2025 ecosystem):

pandas 2.3+
scikit-learn 1.7+
matplotlib 3.10+
missingno 0.5.2

Step 1: Measure before you fix

Never start with imputation.

First, answer these questions:

How many values are missing?
Which columns are affected?
Are values missing together?
Are certain groups affected more than others?
Is missingness increasing over time?

Start with a summary.

import pandas as pd

df = pd.read_csv("data.csv")

summary = (
    df.isna()
      .sum()
      .rename("missing")
      .to_frame()
)

summary["percent"] = (
    summary["missing"] / len(df) * 100
).round(2)

print(summary.sort_values("percent", ascending=False))

Percentages are usually more informative than raw counts because datasets vary in size.

Step 2: Visualize missingness

Tables tell you how much data is missing.

Visualizations tell you where and why.

Install the required packages:

pip install "pandas>=2.3" "matplotlib>=3.10" "missingno==0.5.2"

Then generate a missingness matrix and correlation heatmap.

import matplotlib.pyplot as plt
import missingno as msno
import pandas as pd

df = pd.read_csv("data.csv")

fig, axes = plt.subplots(2, 1, figsize=(12, 8))

msno.matrix(df, ax=axes[0])
axes[0].set_title("Missingness Matrix")

msno.heatmap(df, ax=axes[1])
axes[1].set_title("Missingness Correlation")

plt.tight_layout()
plt.show()

These plots often reveal:

Broken ETL jobs
Sensor outages
Optional survey questions
Entire failed ingestion batches
Columns that become missing together because they share an upstream dependency

Patterns are almost always more informative than isolated counts.

Step 3: Understand why values are missing

Before selecting an imputation strategy, identify the likely missingness mechanism.

MCAR — Missing Completely At Random

Missingness is unrelated to any observed or unobserved variable.

Example:

A laboratory analyzer randomly fails 2% of the time.

In this case, removing incomplete rows may be acceptable if:

enough observations remain,
the removed rows still represent the population reasonably well,
and the loss of statistical power is acceptable.

MAR — Missing At Random

Missingness depends on variables you do observe.

Example:

Older customers are less likely to disclose income.

Income itself isn’t causing the missingness.

Age is.

Because age is available, predictive models can often estimate missing income reasonably well.

Many practical imputation methods assume MAR.

MNAR — Missing Not At Random

Missingness depends on the value itself.

Example:

People with exceptionally high salaries deliberately skip the income question.

The missing value is directly related to the probability of being missing.

No generic imputation algorithm can fully solve MNAR.

Here, domain knowledge and data collection practices matter as much as machine learning.

Step 4: Don’t treat row deletion as a default

A common rule of thumb says:

“If less than 5% of values are missing, just drop the rows.”

Treat that as a heuristic—not a law.

Whether row deletion is appropriate depends on:

Why values are missing
Whether missingness is concentrated in important groups
Available sample size
The downstream task

Dropping 3% of rows can still bias results if those rows belong to one customer segment or one time period.

If row deletion is appropriate:

clean = df.dropna()

# Or require only critical columns
clean = df.dropna(subset=["age", "salary"])

Always inspect which rows disappear—not just how many.

Step 5: Build strong baselines

Simple imputers are valuable baselines.

Numeric columns:

Mean
Median
Constant value

Categorical columns:

Most frequent value
Explicit "Unknown" category

Median is usually more robust than mean because it is less sensitive to outliers.

from sklearn.impute import SimpleImputer

median_imputer = SimpleImputer(strategy="median")

df["salary"] = median_imputer.fit_transform(
    df[["salary"]]
)

Simple methods are often surprisingly competitive.

But never assume they preserve statistical relationships.

Cherry on the cake: when mean imputation flips the conclusion

Suppose you’re evaluating whether a treatment improves recovery scores.

The treatment group has many missing observations because patients with the highest scores skipped the follow-up questionnaire.

Here’s a reproducible example.

import numpy as np
import pandas as pd

control = np.array([70, 72, 74, 75, 76, 77, 78, 79])

treatment = np.array([75, 77, 80, 82, np.nan, np.nan, np.nan, np.nan])

df = pd.DataFrame({
    "group": ["control"] * len(control) + ["treatment"] * len(treatment),
    "score": np.concatenate([control, treatment])
})

print("Drop missing:")
print(df.groupby("group")["score"].mean())

overall_mean = df["score"].mean()

filled = df.copy()
filled["score"] = filled["score"].fillna(overall_mean)

print("\nMean imputation:")
print(filled.groupby("group")["score"].mean())

In this scenario:

Dropping missing values suggests the treatment group performs substantially better.
Mean imputation pulls those missing high scores toward the global average.
The apparent treatment effect shrinks dramatically—even though nothing about the underlying patients changed.

This illustrates an important lesson:

Imputation changes data. Therefore, it can change conclusions.

Always compare analyses across multiple imputation strategies before reporting results.

Step 6: Preserve the fact that a value was missing

Sometimes missingness itself predicts the outcome.

Instead of only filling values, add a missing-value indicator.

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(
    strategy="median",
    add_indicator=True
)

X = imputer.fit_transform(df)

This is especially useful in:

Healthcare
Credit scoring
Manufacturing
Fraud detection
Customer behavior modeling

Modern tree-based models often benefit from knowing that a value was missing, not just what replaced it.

Step 7: Compare multiple strategies in one reproducible workflow

Rather than arguing about imputation methods, benchmark them.

The workflow below compares:

Row deletion
Mean imputation
Median imputation
Model-based iterative imputation

import pandas as pd

from sklearn.experimental import enable_iterative_imputer  # noqa: F401
from sklearn.impute import IterativeImputer, SimpleImputer

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline

X = df.drop(columns=["target"])
y = df["target"]

mask = X.notna().all(axis=1)

scores = {}

# Drop incomplete rows
drop_model = RandomForestClassifier(random_state=42)

scores["drop"] = cross_val_score(
    drop_model,
    X.loc[mask],
    y.loc[mask],
    cv=5,
    scoring="accuracy"
).mean()

pipelines = {
    "mean": Pipeline([
        ("imputer", SimpleImputer(strategy="mean")),
        ("model", RandomForestClassifier(random_state=42))
    ]),
    "median": Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("model", RandomForestClassifier(random_state=42))
    ]),
    "iterative": Pipeline([
        ("imputer", IterativeImputer(random_state=42)),
        ("model", RandomForestClassifier(random_state=42))
    ])
}

for name, pipeline in pipelines.items():
    scores[name] = cross_val_score(
        pipeline,
        X,
        y,
        cv=5,
        scoring="accuracy"
    ).mean()

print(pd.Series(scores).sort_values(ascending=False))

The best strategy depends entirely on:

Missingness mechanism
Dataset size
Feature relationships
Model family

Measure first.

Assume nothing.

Keep imputation inside the pipeline

One of the easiest mistakes is fitting an imputer before splitting data.

That leaks information from the validation or test set.

Instead:

from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("model", RandomForestClassifier(random_state=42))
])

pipeline.fit(X_train, y_train)

Every cross-validation fold now learns imputation statistics independently, matching production behavior.

Practical decision guide

Situation	Recommended approach
Small amount of MCAR missingness with minimal impact from row removal	Consider dropping incomplete rows after verifying representativeness
Skewed numeric variables	Median imputation
Approximately symmetric numeric variables	Mean or median baseline
Categorical features	Most frequent value or explicit `"Unknown"`
Strong relationships among features	Iterative or other model-based imputation
Predictive missingness	Add missing-value indicators
Suspected MNAR	Investigate the data collection process before modeling

Common mistakes

Avoid these pitfalls:

Imputing before train/test splitting
Assuming all missing values are random
Ignoring missingness visualizations
Using mean imputation for heavily skewed variables
Evaluating only one imputation strategy
Reporting results without sensitivity analysis
Removing missing-value indicators that carry predictive information
Failing to document preprocessing decisions for reproducibility

Best-practice checklist

Before deploying a model:

✅ Measure missingness
✅ Visualize missingness patterns
✅ Classify the likely MCAR, MAR, or MNAR mechanism
✅ Split data before fitting imputers
✅ Keep preprocessing inside pipelines
✅ Compare row deletion, simple imputation, and model-based imputation
✅ Consider missing-value indicators
✅ Verify that conclusions remain stable across preprocessing choices
✅ Document every DataQuality decision for reproducibility

Final thoughts

Missing values are not empty cells waiting to be filled—they are evidence about how your data was generated.

Treating MissingData as part of the data-generating process leads to better Imputation decisions, more reliable models, and stronger DataQuality.

The next time you reach for fillna(), pause long enough to measure, visualize, understand the missingness mechanism, and benchmark multiple approaches. Your future analyses—and anyone relying on them—will be better for it.

Next step: apply the workflow in this article to one of your own datasets. Generate a missingness matrix, compare row deletion against at least three imputation strategies, and record how each one changes your model’s performance and your conclusions. That single exercise will teach you more than any rule of thumb ever could.

Why missing data deserves first-class attention

Step 1: Measure before you fix

Step 2: Visualize missingness

Step 3: Understand why values are missing

MCAR — Missing Completely At Random

MAR — Missing At Random

MNAR — Missing Not At Random

Step 4: Don’t treat row deletion as a default

Step 5: Build strong baselines

Cherry on the cake: when mean imputation flips the conclusion

Step 6: Preserve the fact that a value was missing

Step 7: Compare multiple strategies in one reproducible workflow

Keep imputation inside the pipeline

Practical decision guide

Common mistakes

Best-practice checklist

Final thoughts

Vulnerability Watch No15

ONNX: one model format to run everywhere, from server to browser

PySpark ML performance: partitions, data skew, caching, and the small-data trap