From EDA to features: encoding, scaling, and the leakage traps in between

A machine learning model can achieve a perfect validation score and still be completely useless in production.

Sometimes the problem isn’t the algorithm, the hyperparameters, or the amount of training data. The problem is that information from the future, the validation set, or the target variable has quietly entered the feature engineering process.

This is data leakage, and some of its most dangerous forms happen between exploratory data analysis (EDA) and model training.

A simple preprocessing step can introduce it. So can encoding a categorical variable, standardizing numerical features, calculating group statistics, or selecting columns based on their relationship with the target.

The dangerous cases rarely produce errors. They produce excellent metrics.

In this tutorial, we’ll build a realistic feature engineering workflow using scikit-learn 1.9, deliberately introduce target-encoding leakage, verify the failure with assertions, and implement a corrected solution using Pipeline, ColumnTransformer, and cross-fitting.

We’ll also examine two advanced situations where a correctly constructed pipeline is not enough: temporal leakage and group leakage.

By the end, you’ll understand how to create features that are predictive, reproducible, statistically defensible, and available when your model actually runs.

1. The Real Objective of Feature Engineering

Feature engineering transforms raw data into representations that machine learning algorithms can use effectively.

Typical operations include:

• Imputing missing numerical values.

• Encoding categorical variables.

• Scaling continuous features.

• Creating ratios, differences, and interactions.

• Calculating historical group statistics.

• Reducing high-cardinality categorical variables.

• Selecting informative features.

• Removing features unavailable at prediction time.

However, feature engineering has two separate requirements.

Requirement 1: A feature should contain useful predictive information.

Requirement 2: That information must be legitimately available when the prediction is made.

A feature satisfying the first requirement but violating the second is not a valid predictive feature. It is leakage.

A concrete example: predicting retail returns

Imagine an e-commerce platform that wants to predict whether an order will eventually be returned.

The prediction happens immediately after checkout.

The last three features may be extremely predictive, but they are unusable for the intended prediction.

The historical merchant return rate is more interesting.

It can be legitimate if calculated exclusively from completed orders whose outcomes were known before the prediction timestamp.

It becomes leakage if calculated using outcomes that were not yet available.

This is why every serious feature engineering workflow needs an explicit understanding of prediction time, data provenance, and evaluation boundaries.

2. Three Leakage Mechanisms You Must Distinguish

Not all leakage originates from the same mistake.

2.1 Target leakage

Target leakage occurs when a feature directly or indirectly contains information about the outcome being predicted.

Examples include:

• Predicting delivery delays using actual delivery timestamps.

• Predicting returns using finalized refund amounts.

• Predicting subscription cancellation using subsequent account-closure records.

• Encoding a category using the target value of the same observation.

The final example is particularly subtle because the target can be hidden inside a numerical feature that looks perfectly ordinary.

2.2 Train-test contamination

Train-test contamination happens when information from validation or test observations influences model training.

Common examples include:

• Fitting a scaler before splitting the dataset.

• Learning imputation statistics from all available rows.

• Selecting features using the complete dataset’s labels.

• Fitting target encodings using validation targets.

• Calculating aggregate statistics across training and held-out records.

Even transformations that do not directly use labels can cross an evaluation boundary.

For example, standardizing data using the held-out set’s mean allows the test distribution to influence preprocessing.

This is generally less dramatic than directly leaking test labels, but it compromises a standard inductive evaluation.

Some explicitly transductive learning protocols permit access to unlabeled test features. Those protocols need to be declared and evaluated accordingly.

2.3 Validation-design leakage

A pipeline can be implemented correctly and still produce misleading scores if the validation strategy is inappropriate.

Examples include:

• Randomly splitting transactions when the task is forecasting future transactions.

• Allowing the same customer in training and validation when the model must generalize to unseen customers.

• Splitting overlapping sensor windows into different folds.

• Evaluating near-duplicate records that also appear in training.

The principle is straightforward:

Your validation process must reproduce the information boundaries of the real prediction problem.

A random split is not universally wrong. It is wrong when it permits information that the deployed model would not legitimately possess.

3. Start EDA With a Feature Availability Audit

Before calculating correlations or plotting distributions, establish the exact moment at which predictions will occur.

For our retail example, that moment is checkout.

We can organize candidate features into four categories.

Numerical features

Examples:

• Basket total.

• Discount percentage.

• Number of purchased items.

• Shipping cost quoted at checkout.

Useful EDA investigations include missingness, numerical ranges, skewness, outliers, units, and unexpected constant values.

Low-cardinality categorical features

Examples:

• Purchase channel.

• Shipping method.

• Payment method.

These often work well with one-hot encoding.

High-cardinality categorical features

Examples:

• Merchant identifier.

• Product identifier.

• Advertising campaign.

• Delivery zone.

Potential approaches include frequency encoding, hashing, one-hot encoding, and target encoding.

The correct choice depends on category frequency, available training history, the model family, and how frequently unfamiliar categories appear during inference.

Identifier-like features

Examples:

• Transaction identifier.

• Checkout session identifier.

• Unique coupon code.

These deserve additional scrutiny.

A unique identifier can become a perfect proxy for the target after unsafe encoding, even when it has no transferable predictive value.

What should training-only EDA measure?

Start with:

• Missing-value percentages.

• Unique category counts.

• Category frequency distributions.

• Numerical quantiles and unusual values.

• Duplicate and near-duplicate observations.

• Target prevalence within training data.

• Category stability across time.

• Differences between training periods.

• Relationships between training features and training targets.

Make the protected evaluation split before using target relationships to guide feature discovery.

There is an important distinction between examining raw schemas and using evaluation labels to make modeling decisions.

Checking that production data contains an expected column is reasonable. Repeatedly selecting transformations because they improve a supposedly untouched test score is not.

4. Choose Encodings Based on Their Statistical Assumptions

Categorical encoding is not merely converting strings into numbers.

Each encoding method introduces assumptions about the relationship between categories and outcomes.

4.1 One-hot encoding

One-hot encoding creates indicator columns for categorical values.

This is an effective starting point when:

• Categories have no natural order.

• Cardinality is modest.

• Separate category effects are useful.

• Interpretability matters.

For applications with rare categories, scikit-learn supports grouping infrequent values.

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(
    handle_unknown="infrequent_if_exist",
    min_frequency=10,
)

Categories appearing fewer than ten times during fitting are grouped into an infrequent category.

An unseen category maps to that bucket when the bucket exists; otherwise, the encoder handles it like an ignored category.

The frequencies must be learned within the training partition, not from the complete dataset.

4.2 Ordinal encoding

Ordinal encoding assigns integer values to categories.

It is appropriate for genuinely ordered categories such as low, medium, and high.

For unordered categories, arbitrary integers can introduce artificial relationships.

A linear model might interpret one category as numerically greater than another, even though the difference has no meaningful interpretation.

4.3 Frequency encoding

Frequency encoding replaces a category with its occurrence count or proportion in training data.

It does not directly use the target, making it less prone to label leakage.

However, category frequencies are still learned statistics.

They must respect the appropriate training boundary, just like means and standard deviations.

4.4 Target encoding

Target encoding replaces a category with a statistic derived from its associated target values.

For binary classification, the encoding often estimates the positive-class probability for each category.

Consider historical merchant returns:

Merchant C appears to have a 100% return rate.

But that estimate is based on only two observations.

It is much less reliable than estimates based on hundreds of completed orders.

This motivates smoothing.

5. Target Encoding: Smoothing and Cross-Fitting Solve Different Problems

An additive smoothing rule combines a category’s observed target statistic with the overall target average.

For a binary target:

Encoded rate = (category positive count + m × global positive rate) / (category count + m)

Here, m controls the influence of the global average.

Suppose the global return rate is 20%, and a merchant has two returned orders out of two completed orders.

With a smoothing strength of 20:

Smoothed rate = (2 + 20 × 0.20) / (2 + 20) = 0.273

Instead of assigning a return probability of 100%, we obtain approximately 27.3%.

The estimate still reflects the merchant’s observed outcomes, but recognizes that two observations provide limited evidence.

In scikit-learn’s TargetEncoder:

• smooth=0.0 applies no additive smoothing.

• smooth=20.0 uses an additive smoothing strength of 20.

• smooth="auto" uses empirical-Bayes smoothing estimates.

The numeric smoothing formula applies to each category’s statistics within the relevant fitting data. During cross-fitting, those statistics and the corresponding global mean are calculated separately from each fold’s fitting partition.

The current behavior is documented in the scikit-learn TargetEncoder API.

Why smoothing is insufficient

Suppose every order has a unique coupon code.

If we target-encode the complete labeled dataset with no smoothing, every coupon appears exactly once.

Its category target mean is therefore the observation’s own label.

A returned order receives 1.

A non-returned order receives 0.

The feature has become a copy of the answer.

There are two related but distinct failure modes.

Failure A: Held-out label contamination. If the encoder learns from the entire dataset before the train-test split, validation labels can appear inside the encoded validation features. Evaluation itself becomes invalid.

Failure B: Training-row self-leakage. Even when the split is correct, computing ordinary in-sample target means can let each training row encode its own target. The classifier may memorize this relationship, then fail on genuinely unseen observations.

Smoothing can reduce memorization in rare categories, but it cannot establish valid data boundaries.

For that, we need cross-fitting.

The critical scikit-learn behavior

In scikit-learn, these operations intentionally behave differently:

• fit(X, y).transform(X) computes in-sample encodings using statistics learned from all fitted rows.

• fit_transform(X, y) uses internal cross-fitting to construct the training representation.

During internal cross-fitting, the encoder divides training observations into folds.

Each fold receives encodings calculated using target statistics estimated from the other folds.

Afterward, the encoder also retains mappings fitted on the complete training partition for transforming future observations.

This is the essential distinction between protecting individual training rows and protecting an outer evaluation set.

Starting in scikit-learn 1.9, TargetEncoder accepts a cross-validation splitter directly through its cv parameter. The older shuffle and random_state parameters are deprecated in that version, so we’ll configure randomness through StratifiedKFold instead. scikit-learn 1.9 release notes.

6. Hands-On Lab: Reproduce Leakage and Fix It

We’ll now build a fully runnable experiment using synthetic e-commerce transactions.

The dataset contains:

• 6,000 orders.

• 80 merchants.

• Three purchase channels.

• Basket values and discounts.

• Missing numerical observations.

• A unique coupon code for every order.

• Repeated customer identifiers for a later group-validation exercise.

• A binary outcome indicating whether an order was returned.

Some merchants genuinely have different return probabilities.

This creates a legitimate predictive relationship that target encoding can learn without needing held-out labels.

Step 1: Create a reproducible environment

As of October 2026, the relevant stable releases include scikit-learn 1.9.1, pandas 3.0.6, NumPy 2.5.4, and SciPy 1.18.1.

The example targets Python 3.12 and uses exact top-level dependency versions.

On Linux or macOS:

python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip

python -m pip install \
    "numpy==2.5.4" \
    "pandas==3.0.6" \
    "scipy==1.18.1" \
    "scikit-learn==1.9.1"

python -m pip check

For Windows PowerShell, create the environment with Python 3.12 and activate it using the corresponding Windows activation script.

These version choices reflect the published Python and dependency compatibility requirements. The environment check verifies the packages resolved by your installer.

Record the complete installed dependency graph:

python -m pip freeze > requirements-lock.txt

The resulting file contains the direct and transitive versions installed in your environment.

To recreate that dependency selection in a compatible Python environment:

python -m pip install -r requirements-lock.txt
python -m pip check

For production-grade reproducibility across machines, also record the Python runtime, operating-system or container image, package hashes, and relevant execution configuration.

Check the installed versions:

python -c "import numpy, pandas, scipy, sklearn; print(numpy.__version__, pandas.__version__, scipy.__version__, sklearn.__version__)"

The version-specific APIs below require scikit-learn 1.9 or newer.

Step 2: Create the complete experiment

Save the following program as eda_leakage_lab.py.

It generates synthetic data, deliberately applies two unsafe encodings, verifies why they fail, builds a corrected preprocessing pipeline, and performs cross-validation.

The important assertions are part of the experiment, not optional diagnostics.

import numpy as np
import pandas as pd
import sklearn

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import (
    OneHotEncoder,
    StandardScaler,
    TargetEncoder,
)


# --------------------------------------------------
# 1. Generate synthetic retail transactions
# --------------------------------------------------

rng = np.random.default_rng(2026)

n = 6_000

merchant_number = rng.integers(
    0,
    80,
    size=n,
)

channels = rng.choice(
    ["web", "mobile", "store"],
    size=n,
    p=[0.50, 0.35, 0.15],
)

basket = rng.lognormal(
    mean=3.7,
    sigma=0.55,
    size=n,
)

discount = rng.uniform(
    0.0,
    0.45,
    size=n,
)

merchant_effect = rng.normal(
    0.0,
    0.8,
    size=80,
)

log_odds = (
    -2.1
    + 0.014 * (basket - 40.0)
    + 1.5 * discount
    + 0.35 * (channels == "mobile")
    + merchant_effect[merchant_number]
)

probability = 1.0 / (
    1.0 + np.exp(-log_odds)
)

y = pd.Series(
    rng.binomial(1, probability),
    name="returned",
)

X = pd.DataFrame(
    {
        "basket_total": basket,
        "discount_pct": discount,
        "channel": channels,
        "merchant_id": [
            f"merchant_{i:03d}"
            for i in merchant_number
        ],
        "coupon_code": [
            f"coupon_{i:05d}"
            for i in range(n)
        ],
        "customer_id": [
            f"customer_{i % 300:03d}"
            for i in range(n)
        ],
    }
)

# Simulate missing checkout telemetry.
X.loc[
    rng.choice(
        n,
        size=240,
        replace=False,
    ),
    "basket_total",
] = np.nan


# --------------------------------------------------
# 2. Split before learned preprocessing
# --------------------------------------------------

train_idx, test_idx = train_test_split(
    np.arange(n),
    test_size=0.25,
    stratify=y,
    random_state=42,
)

X_train = X.iloc[train_idx]
X_test = X.iloc[test_idx]

y_train = y.iloc[train_idx]
y_test = y.iloc[test_idx]


# --------------------------------------------------
# 3. Training-only EDA
# --------------------------------------------------

print("Training missing fractions:")

print(
    X_train.isna()
    .mean()
    .round(3)
    .to_string()
)

print("\nTraining cardinalities:")

print(
    X_train.nunique()
    .to_string()
)


# --------------------------------------------------
# 4. BAD: Fit target encoding using all labels
# --------------------------------------------------

# Zero smoothing is intentional here.
# It exposes direct label memorization.

leaked_encoder = TargetEncoder(
    target_type="binary",
    smooth=0.0,
)

# WRONG: includes held-out target labels.
leaked_encoder.fit(
    X[["coupon_code"]],
    y,
)

leaked_all = leaked_encoder.transform(
    X[["coupon_code"]]
)

# Verify the exact leakage mechanism.
# Every coupon appears once, so the encoded
# held-out value equals its true target.

np.testing.assert_array_equal(
    leaked_all[test_idx, 0],
    y_test.to_numpy(dtype=float),
)

leaked_model = LogisticRegression()

leaked_model.fit(
    leaked_all[train_idx],
    y_train,
)

leaked_auc = roc_auc_score(
    y_test,
    leaked_model.predict_proba(
        leaked_all[test_idx]
    )[:, 1],
)


# --------------------------------------------------
# 5. BAD: In-sample training target encoding
# --------------------------------------------------

in_sample_encoder = TargetEncoder(
    target_type="binary",
    smooth=0.0,
)

in_sample_encoder.fit(
    X_train[["coupon_code"]],
    y_train,
)

in_sample_train = (
    in_sample_encoder.transform(
        X_train[["coupon_code"]]
    )
)

in_sample_test = (
    in_sample_encoder.transform(
        X_test[["coupon_code"]]
    )
)

# Training encodings reproduce their labels.
np.testing.assert_array_equal(
    in_sample_train[:, 0],
    y_train.to_numpy(dtype=float),
)

# Every test coupon is unseen.
# Its encoding must be the training target mean.
np.testing.assert_allclose(
    in_sample_test[:, 0],
    y_train.mean(),
)

in_sample_model = LogisticRegression()

in_sample_model.fit(
    in_sample_train,
    y_train,
)

in_sample_train_auc = roc_auc_score(
    y_train,
    in_sample_model.predict_proba(
        in_sample_train
    )[:, 1],
)

in_sample_test_auc = roc_auc_score(
    y_test,
    in_sample_model.predict_proba(
        in_sample_test
    )[:, 1],
)


# --------------------------------------------------
# 6. Inspect legitimate cross-fitted coupon encoding
# --------------------------------------------------

inner_cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

cross_fitted_coupon_encoder = TargetEncoder(
    target_type="binary",
    smooth=0.0,
    cv=inner_cv,
)

cross_fitted_coupons = (
    cross_fitted_coupon_encoder.fit_transform(
        X_train[["coupon_code"]],
        y_train,
    )
)

# Each unique coupon is absent from the fold used
# to calculate its training encoding.
# Its value becomes that fold's target mean.

assert np.unique(
    cross_fitted_coupons[:, 0]
).size <= 5

assert np.all(
    (cross_fitted_coupons > 0.0)
    & (cross_fitted_coupons < 1.0)
)


# --------------------------------------------------
# 7. GOOD: Leakage-aware feature engineering
# --------------------------------------------------

features = ColumnTransformer(
    transformers=[
        (
            "numeric",
            Pipeline(
                steps=[
                    (
                        "impute",
                        SimpleImputer(
                            strategy="median",
                            add_indicator=True,
                        ),
                    ),
                    (
                        "scale",
                        StandardScaler(),
                    ),
                ]
            ),
            [
                "basket_total",
                "discount_pct",
            ],
        ),
        (
            "channel",
            OneHotEncoder(
                handle_unknown="ignore",
            ),
            ["channel"],
        ),
        (
            "merchant",
            TargetEncoder(
                target_type="binary",
                smooth=20.0,
                cv=StratifiedKFold(
                    n_splits=5,
                    shuffle=True,
                    random_state=42,
                ),
            ),
            ["merchant_id"],
        ),
    ],
    # Exclude unique coupons and customer IDs.
    remainder="drop",
)

model = Pipeline(
    steps=[
        ("features", features),
        (
            "classifier",
            LogisticRegression(
                max_iter=1_000,
            ),
        ),
    ]
)


# --------------------------------------------------
# 8. Cross-validate the entire pipeline
# --------------------------------------------------

outer_cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

cv_auc = cross_val_score(
    model,
    X_train,
    y_train,
    scoring="roc_auc",
    cv=outer_cv,
)


# --------------------------------------------------
# 9. Fit on training data and evaluate once
# --------------------------------------------------

model.fit(
    X_train,
    y_train,
)

clean_auc = roc_auc_score(
    y_test,
    model.predict_proba(
        X_test
    )[:, 1],
)


# --------------------------------------------------
# 10. Report and verify results
# --------------------------------------------------

print(
    f"\nscikit-learn: {sklearn.__version__}"
)

print(
    f"NumPy: {np.__version__}"
)

print(
    f"pandas: {pd.__version__}"
)

print(
    f"\nLeaked full-data test AUC: "
    f"{leaked_auc:.3f}"
)

print(
    f"In-sample training AUC: "
    f"{in_sample_train_auc:.3f}"
)

print(
    f"In-sample held-out AUC: "
    f"{in_sample_test_auc:.3f}"
)

print(
    f"Cross-fitted coupon distinct priors: "
    f"{np.unique(cross_fitted_coupons[:, 0]).size}"
)

print(
    f"Clean outer CV AUC: "
    f"{cv_auc.mean():.3f} "
    f"± {cv_auc.std():.3f}"
)

print(
    f"Clean held-out AUC: "
    f"{clean_auc:.3f}"
)

np.testing.assert_allclose(
    [
        leaked_auc,
        in_sample_train_auc,
        in_sample_test_auc,
    ],
    [1.0, 1.0, 0.5],
    atol=1e-12,
)

print("\nLeakage assertions passed.")

Step 3: Run the program

python eda_leakage_lab.py

Step 4: Understand the verified results

Our run (Python 3.13, scikit-learn 1.9.1, NumPy 2.5.4, pandas 3.0.6) printed:

Leaked full-data test AUC: 1.000
In-sample training AUC: 1.000
In-sample held-out AUC: 0.500
Cross-fitted coupon distinct priors: 2
Clean outer CV AUC: 0.706 ± 0.015
Clean held-out AUC: 0.693

Leakage assertions passed.

The first three lines are deterministic: the script asserts the conditions that produce them.

These values are not merely illustrative predictions.

The program verifies the underlying conditions that produce them.

The exact clean cross-validation and held-out scores are calculated at runtime instead of being hard-coded, so the results reflect the installed numerical libraries and execution environment.

The clean scores are computed at runtime; with the pinned versions above we got 0.706 ± 0.015 mean cross-validation AUC and 0.693 on the held-out set. Expect small differences on other library versions.

The experiment establishes the following contrast:

The contaminated AUC of 1.000 is not a valid generalization estimate.

The first experiment appears perfect because the encoded evaluation features contain their own targets.

The second experiment memorizes training labels but cannot transfer that relationship to unseen coupon codes.

Every test coupon receives the overall training target mean. Consequently, the coupon-only classifier produces identical scores for every held-out observation.

Its ROC AUC is exactly 0.5.

The corrected model achieves a less spectacular but meaningful score because it uses legitimate merchant history alongside numerical and categorical checkout information.

A credible AUC around 0.7 is far more useful than an invalid perfect score.

7. Why the Correct Pipeline Works

The corrected approach is not safe simply because it uses a class named Pipeline.

It works because each learned transformation respects the relevant training boundary.

7.1 Imputation uses training-only statistics

The numerical preprocessing branch first applies median imputation.

During cross-validation, the imputer learns medians from the current outer training fold.

It then applies those medians to the outer validation fold.

The held-out data never contributes to estimating the medians used for its transformation.

We also use add_indicator=True, allowing the model to learn whether values were missing during preprocessing.

An important detail: missingness indicators are created for features that had missing values during fitting.

If every numerical feature requires an indicator regardless of training missingness, configure a separate MissingIndicator(features="all") branch.

7.2 Scaling happens after imputation

The next transformation is StandardScaler.

Conceptually:

Scaled value = (value − training mean) / training standard deviation

The mean and standard deviation are estimated using the transformed training partition.

During inference, the scaler reuses those stored statistics.

It does not recalculate them from the new observations.

Scaling matters especially for:

• Regularized linear models.

• Nearest-neighbor methods.

• Distance-based learning.

• Algorithms sensitive to feature magnitude.

Ordinary axis-aligned decision trees are generally less sensitive to linear feature rescaling.

Our numerical branch also scales the missingness indicator created by the imputer.

That is valid for logistic regression, although a separate transformer branch can preserve indicators as binary values if desired.

7.3 One-hot encoding handles unfamiliar categories

The purchase-channel encoder uses handle_unknown="ignore".

Previously unseen channels therefore produce zero values across that categorical feature’s one-hot columns.

This prevents inference errors caused by unfamiliar categories.

It does not guarantee good performance under distribution shift, but it establishes well-defined transformation behavior.

7.4 Merchant target encoding uses internal cross-fitting

The merchant encoder uses five-fold stratified cross-fitting and additive smoothing with strength 20.

During model training:

1. The current outer training data enters ColumnTransformer.

2. Its merchant target encoder performs fit_transform.

3. Internal folds are constructed from the available training observations.

4. Each fold is encoded using statistics from its complementary fitting folds.

5. The downstream classifier learns from those cross-fitted features.

The internal encodings are deliberately different from the mappings subsequently used to transform new observations.

At prediction time, the encoder uses mappings learned from all of the relevant fitting partition.

A merchant absent from those mappings receives the encoder’s learned global target mean.

The target encoder’s internal cross-fitting example demonstrates why this distinction matters.

7.5 Unique identifiers are excluded

Our corrected pipeline drops the unique coupon code.

It also excludes the customer identifier, which we preserve only for a later validation example.

A unique coupon identifier is not inherently useful simply because target encoding can convert it into a number.

When every coupon is new, no historical coupon-specific outcome exists for inference.

Better features may include:

• Campaign family.

• Coupon discount type.

• Coupon expiration window.

• Minimum basket requirement.

• Historical campaign performance available at checkout.

These describe meaningful properties rather than merely identifying individual records.

7.6 There are two levels of cross-validation

The experiment uses two distinct cross-validation layers.

The internal encoder folds are not a substitute for outer validation.

Likewise, outer cross-validation cannot repair features contaminated before the pipeline begins.

This is one of the most important distinctions in leakage-aware modeling.

8. Advanced Extension: Respecting Group Boundaries

Consider an application with multiple orders from the same customer.

Suppose deployment requires making predictions for customers the model has never encountered.

A random split may place the same customer in both training and validation.

That does not reproduce the intended deployment setting.

Group-aware validation prevents observations from the same entity appearing on both sides of a split.

Scikit-learn provides GroupKFold and StratifiedGroupKFold.

However, using a group-aware outer splitter is only half the solution when target encoding is involved.

The target encoder’s internal cross-fitting must respect the same group restrictions.

Otherwise, records from one customer might help determine the encoding assigned to another record from that same customer during model training.

Group-aware target encoding in scikit-learn 1.9

Scikit-learn 1.9 introduced the ability to supply a compatible cross-validation splitter to TargetEncoder and route group metadata into its internal cross-fitting process.

This requires:

• scikit-learn 1.9 or newer.

• A group-aware splitter such as StratifiedGroupKFold.

• Metadata routing enabled.

• Group identifiers passed to the modeling workflow.

The following code uses the synthetic customer identifiers from our experiment.

Append it to the original script, or run it in a Python session where the earlier variables are available.

from sklearn import config_context
from sklearn.base import clone
from sklearn.model_selection import StratifiedGroupKFold

groups_train = (
    X_train["customer_id"].to_numpy()
)

grouped_model = clone(model)

# Replace the merchant encoder's internal
# cross-fitting splitter with a group-aware one.

grouped_model.set_params(
    features__merchant__cv=StratifiedGroupKFold(
        n_splits=5,
        shuffle=True,
        random_state=42,
    )
)

# Enable metadata routing for both the pipeline
# and the outer cross-validation operation.

with config_context(
    enable_metadata_routing=True
):
    grouped_scores = cross_val_score(
        grouped_model,
        X_train,
        y_train,
        scoring="roc_auc",
        cv=StratifiedGroupKFold(
            n_splits=5,
            shuffle=True,
            random_state=42,
        ),
        params={
            "groups": groups_train,
        },
    )

print(
    "Group-aware CV AUC:",
    grouped_scores.mean().round(3),
)

print(
    "Group-aware fold scores:",
    grouped_scores.round(3),
)

This configuration establishes group separation at both levels.

The outer splitter keeps customers disjoint between its training and validation folds.

Metadata routing delivers the relevant training-group identifiers through the pipeline and column transformer to the merchant target encoder’s internal splitter.

That splitter then keeps each customer’s training observations together when constructing out-of-fold merchant encodings.

The grouped evaluation represents a different generalization question from our original random-split evaluation.

Its scores should not be interpreted as directly equivalent without considering the intended production population.

Also, the original random held-out test set is not customer-disjoint. If the actual objective is generalization to new customers, create a separate protected group-disjoint final test partition from the outset.

Metadata routing remains an evolving scikit-learn capability, so nested workflows using it should have integration tests and pinned versions.

For details, consult the scikit-learn metadata routing guide.

9. Advanced Extension: Temporal Leakage Is Different

Group-aware encoding solves one class of dependency problem.

It does not automatically solve temporal leakage.

Imagine an application that predicts order returns for future transactions.

A random split might allow an encoder to learn merchant statistics from later completed orders and use them when predicting earlier orders.

For production forecasting, that may be unrealistic.

Chronological validation

Scikit-learn provides TimeSeriesSplit for time-ordered observations.

from sklearn.model_selection import TimeSeriesSplit

time_cv = TimeSeriesSplit(
    n_splits=5,
    gap=48,
)

The gap parameter excludes 48 samples between the training and validation portions.

It does not mean 48 hours unless the observations represent a regular hourly sequence.

For irregular transaction timestamps, explicit time-based boundaries may be more appropriate.

The subtle target-encoding problem

Even when outer validation respects chronology, ordinary internal target-encoder cross-fitting is not necessarily chronological.

An internal fold may use later training observations to encode earlier training observations.

For strict point-in-time feature construction, that can violate the information available at each historical prediction timestamp.

A chronological historical return-rate feature must consider at least two timestamps:

• When an order was created.

• When its return outcome became known.

Suppose an order was placed on Monday and returned on Friday.

That outcome cannot legitimately contribute to a merchant return-rate feature calculated on Wednesday.

A safer production approach is to maintain historical aggregates using outcome-availability timestamps.

For example, the feature store might provide the merchant’s return rate calculated from all eligible outcomes known before checkout.

When generating training data, the same point-in-time lookup logic must be applied to each historical observation.

This preserves consistency between historical training features and live inference.

Why ordinary expanding-window cross-validation is not enough

The target encoder’s internal cross-fitting expects every training observation to appear in exactly one validation fold.

Standard expanding-window schemes often leave the earliest training block without an out-of-fold representation.

They therefore do not automatically satisfy the target encoder’s internal cross-fitting requirements.

For strictly causal encoding, implement a dedicated chronological out-of-fold transformation or point-in-time aggregate generator.

The important distinction is between excluding a target from its own encoding and excluding all information that was unavailable at the row’s prediction time.

Those are different guarantees.

10. Supervised Feature Selection Can Leak Too

Target encoding receives considerable attention because it uses labels directly.

But supervised feature selection can introduce a similar evaluation failure.

Imagine a dataset containing thousands of random numerical features and an unrelated binary target.

If you calculate correlations between every feature and every target using the complete dataset, some features will appear strongly related to the target purely by chance.

Selecting those features before cross-validation allows held-out labels to influence feature selection.

The downstream model can then achieve artificially strong scores even though no genuine predictive relationship exists.

The correct feature-selection structure

Supervised feature selection belongs inside the cross-validated pipeline.

For example, this numerical-only model uses imputation, scaling, and univariate feature selection.

It can run using the X_train, y_train, and outer_cv variables created earlier.

from sklearn.feature_selection import (
    SelectKBest,
    f_classif,
)

numeric_model = Pipeline(
    steps=[
        (
            "impute",
            SimpleImputer(
                strategy="median",
            ),
        ),
        (
            "scale",
            StandardScaler(),
        ),
        (
            "select",
            SelectKBest(
                score_func=f_classif,
                k=1,
            ),
        ),
        (
            "classifier",
            LogisticRegression(
                max_iter=1_000,
            ),
        ),
    ]
)

numeric_columns = [
    "basket_total",
    "discount_pct",
]

numeric_scores = cross_val_score(
    numeric_model,
    X_train[numeric_columns],
    y_train,
    scoring="roc_auc",
    cv=outer_cv,
)

print(
    "Numeric-only CV AUC:",
    numeric_scores.mean().round(3),
)

During every fold:

1. Missing values are imputed using that fold’s training data.

2. Scaling statistics are estimated using that fold’s training data.

3. Feature scores are calculated using only that fold’s training targets.

4. The selected feature is passed to the classifier.

5. The entire fitted workflow predicts on the untouched validation fold.

The same rule applies to recursive feature elimination, mutual-information selection, model-based importance filtering, and other supervised feature-selection procedures.

Even unsupervised transformations such as PCA should generally be fitted within the training boundary for standard inductive evaluation.

11. Diagnose Leakage Before Trusting Model Performance

Not every unexpectedly strong model is contaminated.

But unexplained improvements should trigger an investigation.

Several diagnostic approaches are particularly useful.

11.1 Compare training and validation performance

Large discrepancies can indicate overfitting, memorization, leakage, or distribution shift.

Our deliberately unsafe coupon model demonstrates an extreme case:

• Training AUC: 1.000.

• Held-out AUC: 0.500.

That gap immediately tells us the learned representation does not transfer.

A small training-validation gap, however, does not prove leakage is absent. If the evaluation set itself is contaminated, both scores can look excellent.

11.2 Run a shuffled-target negative control

Randomizing training labels should destroy genuine predictive relationships.

A properly isolated workflow should generally produce near-chance cross-validated discrimination when trained on labels unrelated to its features.

Append this experiment to the main script:

from sklearn.base import clone

shuffled_y = (
    y_train.sample(
        frac=1.0,
        random_state=17,
    )
    .to_numpy()
)

null_scores = cross_val_score(
    clone(model),
    X_train,
    shuffled_y,
    scoring="roc_auc",
    cv=outer_cv,
)

print(
    "Shuffled-label CV AUC:",
    null_scores.mean().round(3),
)

For a binary ROC AUC metric, chance ranking corresponds to approximately 0.5.

Individual results can differ from 0.5 because of finite-sample randomness.

If a supposedly rigorous pipeline consistently produces excellent scores on randomized targets, investigate whether labels or target-derived information are crossing evaluation boundaries.

This test is a diagnostic rather than a formal proof that all leakage has been eliminated.

Also distinguish this experiment from permutation tests with repeated randomizations: a single shuffled run is useful for a quick sanity check, not for drawing precise statistical conclusions.

11.3 Investigate suspiciously powerful features

Any feature with extraordinary predictive power deserves a provenance review.

Ask:

• Was the feature created before or after the predicted event?

• Does it encode part of the outcome?

• Is it derived from a unique identifier?

• Did validation labels influence its calculation?

• Does it represent a future aggregate?

• Would the feature be available for a completely new observation?

Sometimes powerful features are legitimate.

Sometimes they are the answer wearing a different name.

11.4 Test unseen categories

A fitted preprocessing pipeline should have deliberate behavior for unfamiliar categories.

In our experiment:

• The one-hot encoder maps unfamiliar channels to its configured unknown-category representation.

• The target encoder maps unfamiliar merchants to its learned overall target mean.

These behaviors avoid transformation failures.

They do not guarantee that the model will perform well if unfamiliar merchants become a substantial fraction of production traffic.

Monitor unseen-category rates and evaluate relevant cold-start scenarios.

11.5 Compare evaluation strategies

Performance can change considerably between random, group-aware, and chronological validation.

Such differences may reveal:

• Temporal drift.

• Repeated-entity memorization.

• Duplicate contamination.

• Unstable category distributions.

• Incorrect feature-availability assumptions.

• An evaluation strategy inconsistent with deployment.

A performance decrease under stricter validation is not automatically evidence of previous leakage.

It may simply mean the stricter split asks a genuinely harder question.

The goal is to select the question that matches production.

12. Cherry on the Cake: A Real Kaggle Disqualification Involving Leaked Answers

A striking real-world example occurred in Kaggle’s PetFinder.my Adoption Prediction competition.

The competition asked participants to predict adoption speed from information about animals listed for adoption.

In a competition-host announcement published in early 2020, the organizer reported that the first-place team, Bestpetting, had been disqualified for cheating.

According to the organizer’s account, the team obtained private-test adoption-speed answers and concealed them in an external dataset.

The answers were reportedly encoded, obfuscated, and hashed into an identifier field disguised as part of an external image-related dataset.

The announcement also stated that the implicated participant was permanently banned and the competition leaderboard was revised.

The organizer suggested that the answers might have been obtained by scraping its website, but did not establish that acquisition method as a certainty.

Source: PetFinder.my contest: First-place winner disqualified — official competition-host announcement; also reported by The Register, January 2020.

This was deliberate misconduct, not an accidental preprocessing mistake.

A normal scikit-learn pipeline cannot prevent someone from intentionally embedding known answers in an input feature.

Nevertheless, the case illustrates the same fundamental evaluation failure as our target-encoding experiment.

When the prediction process can recover hidden outcomes, the resulting metric no longer measures predictive generalization.

Three lessons stand out:

• Excellent leaderboard performance does not establish that a model learned a legitimate predictive relationship.

• Obfuscated identifiers and auxiliary datasets can conceal target information.

• Independent validation, data provenance, and reproducibility matter as much as model accuracy.

There is also a practical lesson for ordinary machine learning development.

Most leakage is not malicious.

A data scientist may accidentally calculate category averages before splitting the data or include a field that becomes available only after the outcome is known.

The resulting leaderboard or validation score can still be misleading.

Our retail example achieves a perfect contaminated AUC with only a few lines of incorrect preprocessing.

That is what makes leakage so dangerous: it often looks like successful feature engineering.

13. A Practical Checklist for Production Feature Engineering

Before deploying a model, review the workflow across five areas.

Prediction-time availability

• Is the prediction timestamp explicitly defined?

• Are all input features available at that moment?

• Are historical aggregates based on outcomes known by then?

• Are label delays accounted for?

• Are future events excluded from historical feature calculations?

Data partitioning

• Was the protected evaluation partition established before supervised feature discovery?

• Does validation reproduce the intended production scenario?

• Are related entities kept together when generalization requires it?

• Are duplicate and overlapping observations handled appropriately?

• Are chronological boundaries enforced when necessary?

Preprocessing and encoding

• Are imputers and scalers fitted only on permitted training observations?

• Are categorical vocabularies learned within training partitions?

• Are rare and unseen categories handled deliberately?

• Does target encoding prevent training-row self-leakage?

• Are internal encoder folds compatible with relevant group or temporal constraints?

• Are unique identifiers evaluated for genuinely transferable information?

Model selection and evaluation

• Is the entire preprocessing workflow inside cross-validation?

• Is supervised feature selection fitted independently within each training fold?

• Are hyperparameters chosen without consulting final-test labels?

• Are training, cross-validation, and test scores compared?

• Have negative controls been considered?

• Are final evaluation results reported without repeatedly tuning against them?

Reproducibility and operations

• Are package versions and the Python runtime recorded?

• Are preprocessing definitions versioned with the estimator?

• Are training-data boundaries documented?

• Can the workflow be rebuilt without hidden notebook state?

• Are category frequencies and missingness monitored after deployment?

• Are point-in-time feature definitions consistent between training and inference?

A pipeline is one layer of protection.

The validation design, feature definitions, and data-generation process must satisfy the same information boundaries.

14. The Mental Model That Prevents Most Leakage

Imagine that every validation observation represents a future event your training procedure is forbidden to inspect.

Ask what the model would legitimately know if that observation had not yet happened.

It may know:

• Historical training feature distributions.

• Previously observed target outcomes.

• Training-derived category statistics.

• Fitted means, medians, and standard deviations.

• Model parameters estimated from permitted training data.

It must not know:

• The validation observation’s target.

• Target statistics incorporating that target.

• Future outcomes unavailable at prediction time.

• Test-set-derived preprocessing statistics under a standard inductive protocol.

• Data from entities or periods deliberately excluded by the evaluation design.

This clarifies why simply removing the target column from the final feature matrix is insufficient.

A target may already be embedded in a category encoding.

A test observation may already have influenced a scaler.

An identifier may function as a hidden lookup key.

A random split may allow future information into a historical prediction problem.

The reliable workflow is therefore:

Define prediction-time availability → establish evaluation boundaries → explore training data → build learned transformations inside a pipeline → cross-validate the complete process → evaluate once on protected data.

This ordering should become the default architecture of feature engineering rather than an additional precaution applied after suspicious results appear.

Further Reading

The following references document the APIs and implementation behaviors used throughout the tutorial.

• scikit-learn — Common pitfalls and recommended practices: Preprocessing leakage, cross-validation pitfalls, and reproducibility.

• scikit-learn — TargetEncoder API reference: Smoothing, cross-fitting, unseen categories, and custom splitters.

• scikit-learn — Target Encoder’s Internal Cross-Fitting: A focused demonstration of the difference between in-sample and cross-fitted encodings.

• scikit-learn — OneHotEncoder API reference: Rare-category grouping and unfamiliar-category behavior.

• scikit-learn — TimeSeriesSplit API reference: Chronological validation and sample-based gaps.

• scikit-learn — Metadata Routing: Passing metadata such as group identifiers through composite estimators.

• scikit-learn — Version 1.9 release notes: The current target-encoder cross-validation API changes.

• pandas — Version 3.0 release notes: Relevant changes in the current pandas major version, including default string-data handling.

• Kaggle — PetFinder.my competition disqualification announcement: A real incident involving concealed evaluation targets.

15. Your Next Step: Turn Leakage Detection Into a Habit

The most valuable exercise is not simply running the corrected pipeline.

It is deliberately reproducing the failure, verifying why it happens, and observing which safeguards prevent it.

Run the complete experiment, then try these five modifications:

1. Change smoothing: Compare smooth=0.0, smooth=20.0, and smooth="auto" on merchant target encoding. Observe the cross-validation scores and how rare-category estimates behave.

2. Compare encoder training methods: Examine in-sample fit followed by transform against cross-fitted fit_transform. Check which observations contribute to each encoded value.

3. Replace merchant target encoding: Try one-hot encoding for merchant identifiers and evaluate whether target encoding provides a meaningful advantage.

4. Introduce unfamiliar categories: Modify held-out merchant values to include previously unseen identifiers, then verify that inference remains defined and inspect the resulting predictions.

5. Change validation assumptions: Compare random and group-aware cross-validation, then consider what would change under a true chronological, point-in-time evaluation.

Record your results in a simple experiment table containing the preprocessing configuration, evaluation design, training score, cross-validation score, and held-out score.

Whenever performance improves dramatically, ask which information became available to the model and whether that information will exist at inference time.

For your next real-world machine learning project, define the prediction timestamp and validation strategy before building derived features. Keep learned transformations inside the modeling workflow, and test at least one intentionally contaminated alternative so you understand how leakage manifests in your own data.

Start by running eda_leakage_lab.py, change the target encoder’s smoothing configuration, and compare the contaminated, cross-fitted, and properly held-out results.

Then carry one question into every future feature engineering review:

Did the model learn a pattern it can use tomorrow, or did the features quietly tell it what happens tomorrow?

That distinction separates an impressive notebook from a dependable machine learning system.