Outliers: detect, understand, and only then decide to delete

Outliers are observations, not mistakes

An outlier is a record that looks unusual relative to other observations. That definition says nothing about whether the record is wrong.

An unusual value may be:

• a sensor failure, duplicate transaction, or parsing error;

• a unit mismatch, such as pounds mixed with kilograms;

• a legitimate but rare customer, patient, machine state, or transaction;

• evidence that the underlying process has changed;

• the event your analysis is supposed to discover.

Automatically deleting outliers can make a dataset look cleaner while making the resulting model less truthful. Fraud, equipment failures, severe illnesses, cyberattacks, and high-value purchases are uncommon by definition.

Use a safer rule:

Detect outliers, understand why they were flagged, and only then decide whether to delete them.

An outlier detector produces an investigation lead—not a verdict.

Three detectors, three definitions of unusual

Different detectors disagree because they ask different statistical questions.

IQR: outside the middle spread

The interquartile range measures the width of the middle half of a distribution:

IQR = Q3 − Q1

The conventional 1.5 × IQR fences are:

Lower fence = Q1 − 1.5 × IQR

Upper fence = Q3 + 1.5 × IQR

Values outside those fences are flagged.

Because quartiles are less affected by extreme values than the mean and standard deviation, IQR is a useful first check for skewed or messy data. Its main limitation is that it examines one feature at a time.

A row can have an ordinary BMI and ordinary blood pressure while still having an unusual combination of BMI, blood pressure, age, and cholesterol measurements.

Z-score: far from the mean

A z-score expresses a value’s distance from the mean in standard-deviation units:

z = (x − mean) / standard deviation

A common heuristic flags values where |z| > 3.

Z-scores are easy to interpret when the data is approximately symmetric and represents one reasonably homogeneous population. They become less reliable when:

• the distribution is strongly skewed;

• the data has heavy tails;

• several subpopulations are mixed together;

• extreme observations distort the mean and standard deviation.

A z-score threshold is therefore a modeling assumption, not a universal law.

Isolation Forest: easy to separate

Isolation Forest is a multivariate anomaly detector. It builds randomized trees that repeatedly select a feature and a split point. Observations that require fewer splits to isolate receive more abnormal scores.

In current scikit-learn, predict and fit_predict return -1 for outliers and 1 for inliers. Lower decision_function values are more abnormal, and negative values fall on the outlier side of the fitted threshold. When contamination is a number rather than "auto", scikit-learn adjusts that threshold to produce approximately the requested number of training-set outliers.

That does not mean the requested percentage of records is corrupted. It means you instructed the estimator to classify roughly that proportion as anomalous.

A runnable comparison on real data

The following example compares all three methods using scikit-learn’s diabetes regression dataset. It contains 442 samples and ten numeric input features. We request scaled=False so the example uses the raw feature values rather than scikit-learn’s default scaled representation.

The target is retained only for later human review. It is deliberately excluded from anomaly detection because including an outcome in an unsupervised preprocessing rule would leak information unavailable when scoring new records.

As of July 30, 2026, the pinned environment below uses scikit-learn 1.9.0, pandas 3.0.5, and Matplotlib 3.11.1.

python -m pip install \
  "scikit-learn==1.9.0" \
  "pandas==3.0.5" \
  "matplotlib==3.11.1"

Save the following program as compare_outliers.py:

from __future__ import annotations

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.datasets import load_diabetes
from sklearn.ensemble import IsolationForest


def main() -> None:
    dataset = load_diabetes(as_frame=True, scaled=False)

    features = dataset.data.copy()
    results = features.assign(progression=dataset.target)
    results.index.name = "row_id"

    feature = "bmi"

    # 1. Univariate IQR rule
    q1, q3 = results[feature].quantile([0.25, 0.75])
    iqr = q3 - q1
    lower_fence = q1 - 1.5 * iqr
    upper_fence = q3 + 1.5 * iqr

    results["iqr_outlier"] = ~results[feature].between(
        lower_fence,
        upper_fence,
        inclusive="both",
    )

    # 2. Univariate z-score rule
    mean = results[feature].mean()
    standard_deviation = results[feature].std(ddof=0)

    if standard_deviation == 0:
        raise ValueError(f"{feature!r} has zero variance")

    z_scores = (results[feature] - mean) / standard_deviation
    results["z_outlier"] = z_scores.abs() > 3.0

    # 3. Multivariate Isolation Forest
    forest = IsolationForest(
        n_estimators=300,
        contamination=0.05,
        random_state=42,
        n_jobs=-1,
    )

    results["iforest_outlier"] = forest.fit_predict(features) == -1
    results["iforest_score"] = forest.decision_function(features)

    flags = [
        "iqr_outlier",
        "z_outlier",
        "iforest_outlier",
    ]

    summary = pd.DataFrame(
        {
            "flagged_rows": results[flags].sum(),
            "flagged_percent": 100 * results[flags].mean(),
        }
    ).round(2)

    summary.index = [
        "IQR on bmi",
        "Z-score on bmi",
        "Isolation Forest",
    ]

    print(summary.to_string())

    # Build an investigation queue rather than deleting records.
    results["detector_votes"] = results[flags].sum(axis=1)

    review_columns = [
        "age",
        "sex",
        "bmi",
        "bp",
        "s1",
        "s2",
        "progression",
        *flags,
        "detector_votes",
        "iforest_score",
    ]

    review_queue = (
        results.loc[results["detector_votes"] > 0, review_columns]
        .sort_values(
            ["detector_votes", "iforest_score"],
            ascending=[False, True],
        )
    )

    print("\nHighest-priority review candidates")
    print(review_queue.head(20).round(4).to_string())

    review_queue.to_csv("outlier_review.csv")

    # Visualize two dimensions of the multivariate result.
    normal = ~results["iforest_outlier"]
    anomalous = results["iforest_outlier"]

    figure, axis = plt.subplots(figsize=(9, 6))

    axis.scatter(
        results.loc[normal, "bmi"],
        results.loc[normal, "bp"],
        alpha=0.55,
        label="Isolation Forest inlier",
    )

    axis.scatter(
        results.loc[anomalous, "bmi"],
        results.loc[anomalous, "bp"],
        marker="x",
        s=80,
        label="Isolation Forest outlier",
    )

    axis.set(
        xlabel="BMI",
        ylabel="Blood pressure",
        title="Multivariate anomalies may not be univariate extremes",
    )
    axis.legend()

    figure.tight_layout()
    figure.savefig("outlier_comparison.png", dpi=160)


if __name__ == "__main__":
    main()

Run it:

python compare_outliers.py

With the pinned versions, fixed dataset, parameters, and random seed, the summary is:

                  flagged_rows  flagged_percent
IQR on bmi                   3             0.68
Z-score on bmi               2             0.45
Isolation Forest            23             5.20

The disagreement is the lesson:

• IQR asks whether BMI lies far beyond its quartiles.

• Z-score asks whether BMI is more than three standard deviations from its mean.

• Isolation Forest considers all ten input features together.

• contamination=0.05 converts continuous scores into labels using an assumed anomaly proportion.

Isolation Forest can therefore flag a row whose individual values look ordinary but whose combination of values is unusual.

Build a review queue, not an automatic filter

Detector agreement is useful for prioritization:

• IQR only: one marginal value is extreme.

• Isolation Forest only: the relationship between several values is unusual.

• Z-score but not IQR: mean-based and quantile-based definitions disagree.

• Multiple votes: review promptly, but still do not assume invalidity.

The valuable output is often outlier_review.csv, not a supposedly cleaned dataset.

A useful review record should include:

• the source row or event identifier;

• every detector flag;

• continuous anomaly scores;

• the features that contributed to the alert;

• links to source systems or logs;

• the reviewer’s decision and evidence;

• the treatment applied, if any.

Continuous scores are usually more informative than binary labels. A row immediately below a threshold is not fundamentally different from one immediately above it.

Understand the cause before choosing a treatment

For every flagged record, ask four questions.

Is the value possible?

Check:

• units and measurement scales;

• physical or contractual limits;

• signs and decimal separators;

• date parsing and time zones;

• schema constraints;

• impossible category combinations.

A body weight of 900 may be impossible in kilograms but plausible in pounds. The unusual value is real; the unit is wrong.

Can it be verified?

Compare the record with:

• original forms or transactions;

• device calibration logs;

• adjacent timestamps;

• duplicate events;

• upstream transformation code;

• independent measurements.

Correct a value only when you have evidence for the correction. Guessing a replacement can be worse than retaining a suspicious value.

Is it rare but legitimate?

Determine whether the row:

• belongs to an important subgroup;

• occurs elsewhere under similar conditions;

• represents a high-impact customer or event;

• is the phenomenon the model must detect;

• signals a new operating regime.

An anomaly detector can disproportionately flag members of smaller populations because the majority defines what appears normal. Compare flag rates across meaningful groups before using anomaly labels in rejection, eligibility, or access decisions.

Can it recur in production?

Deleting valid edge cases from training data can leave a deployed model unprepared for the exact conditions that matter most.

A rare equipment state, unusually large transaction, or extreme weather condition may be uncommon in historical data but entirely possible after deployment.

Choose the least destructive defensible treatment

Possible treatments include:

• correct a confirmed data error;

• keep the observation unchanged;

• add a feature such as was_reviewed_as_outlier;

• transform a heavily skewed feature;

• cap values at documented domain limits;

• model separate populations independently;

• quarantine the record for manual review;

• use robust estimators or loss functions;

• delete the row only after establishing that it is invalid.

Do not winsorize or cap data merely because a statistical threshold was crossed. A cap should have a domain, policy, measurement, or operational justification.

Document the original value before changing it so the decision remains reversible.

Prevent leakage: learn thresholds from training data

Quartiles, means, standard deviations, and Isolation Forest trees are learned parameters.

During model development:

1. split the data;

2. calculate thresholds using the training partition;

3. apply the unchanged thresholds to validation and test data;

4. fit the final rule again only when retraining the production model.

from sklearn.datasets import load_diabetes
from sklearn.ensemble import IsolationForest
from sklearn.model_selection import train_test_split


features = load_diabetes(
    as_frame=True,
    scaled=False,
).data

X_train, X_test = train_test_split(
    features,
    test_size=0.2,
    random_state=42,
)

# Learn the univariate rules from training data only.
q1, q3 = X_train["bmi"].quantile([0.25, 0.75])
iqr = q3 - q1
lower_fence = q1 - 1.5 * iqr
upper_fence = q3 + 1.5 * iqr

training_mean = X_train["bmi"].mean()
training_standard_deviation = X_train["bmi"].std(ddof=0)

test_iqr_flags = ~X_test["bmi"].between(
    lower_fence,
    upper_fence,
    inclusive="both",
)

test_z_scores = (
    X_test["bmi"] - training_mean
) / training_standard_deviation

test_z_flags = test_z_scores.abs() > 3.0

# Fit the multivariate detector on training data only.
forest = IsolationForest(
    n_estimators=300,
    contamination=0.05,
    random_state=42,
    n_jobs=-1,
)

forest.fit(X_train)

test_iforest_flags = forest.predict(X_test) == -1
test_iforest_scores = forest.decision_function(X_test)

print(f"Test IQR flags: {test_iqr_flags.sum()}")
print(f"Test z-score flags: {test_z_flags.sum()}")
print(f"Test Isolation Forest flags: {test_iforest_flags.sum()}")
print(f"Lowest test score: {test_iforest_scores.min():.4f}")

This also highlights the distinction between outlier detection and novelty detection.

In outlier detection, the training data may already contain anomalies. In novelty detection, the estimator is trained on data intended to represent normal behavior and is then applied to unseen observations. Estimator capabilities differ: Local Outlier Factor, for example, exposes prediction methods for new data only when configured with novelty=True.

Cherry on the cake: the ozone-hole outlier story

A famous retelling claims that NASA missed the Antarctic ozone hole because software discarded extremely low satellite readings as errors.

The real history is more instructive.

British Antarctic Survey scientists Joe Farman, Brian Gardiner, and Jon Shanklin repeatedly checked unusually low ground measurements before publishing the discovery in 1985. NASA satellite data subsequently demonstrated that the depletion extended across a large Antarctic region.

NASA scientist P. K. Bhartia later addressed the popular “deleted outlier” version directly. According to his historical account, the satellite anomaly was noticed within hours and analyzed within days. The delay in reporting was primarily caused by the need for independent validation and by limitations in retrieval assumptions—not because the values had vanished unnoticed.

That nuance strengthens the practical lesson:

An extreme observation should trigger verification, comparison, and investigation—not silent deletion.

The outlier may be a broken instrument. It may also be the first evidence that your definition of normal is broken.

A practical decision rule

Before removing any flagged row, answer:

• Which detector flagged it, and why?

• Is the value technically possible?

• Can the source record be verified?

• Does domain knowledge explain it?

• Is it rare but legitimate?

• Could it recur after deployment?

• How does each treatment affect important subgroups?

• Can another person reproduce the decision?

Start with interpretable methods such as IQR and z-scores. Add multivariate detection when relationships between features matter. Preserve continuous scores, review the most suspicious records, and maintain an audit trail for every correction or deletion.

Run the comparison on one of your datasets today. Export a review queue, investigate the highest-priority rows, and document the evidence before deleting a single observation.