Cross-validation done right: k-fold, stratification, and time-series traps

A machine learning model can achieve 97% cross-validation accuracy and still fail spectacularly when deployed. The culprit is often not the algorithm, hyperparameters, or training data. It’s the way the data was split.

Cross-validation is one of the most important techniques in machine learning because it helps estimate how a model will perform on unseen data. But choosing the wrong cross-validation strategy can produce misleadingly optimistic results, hide data leakage, and encourage you to select a model that performs poorly in production.

The central principle is simple:

Your validation strategy must reproduce the conditions under which your model will actually make predictions.

That means ordinary k-fold cross-validation is useful for independent observations, stratified k-fold is often preferable for imbalanced classification, and time-aware validation is essential when predictions concern the future.

In this practical guide, you’ll learn:

• How k-fold cross-validation works and when to use it.

• Why stratification matters for imbalanced classification datasets.

• How random splitting can distort results on chronological data.

• How to implement rolling and expanding time-series validation with scikit-learn.

• How to prevent preprocessing, feature engineering, and model-selection leakage.

• How to design a trustworthy evaluation workflow for production machine learning.

You’ll also run a reproducible experiment in which random cross-validation reports approximately 97% accuracy, while a chronological validation fold drops to approximately 26%. We’ll examine why both measurements can be technically correct, yet only one exposes a serious deployment risk.

1. What cross-validation actually measures

Imagine you’re building a model that predicts whether an e-commerce order will arrive late.

You’ve collected thousands of historical orders, including shipping distance, order characteristics, and whether each delivery met its promised date.

A basic evaluation approach divides the dataset into two subsets:

• Training set: Used to fit the model.

• Test set: Used to estimate performance on unseen observations.

The problem is that one train/test split may give an unusually favorable or unfavorable result.

Perhaps the test set happens to contain mostly easy predictions. Perhaps a few unusual orders dominate the error. Or perhaps the training period does not represent the operating conditions the model will encounter.

Cross-validation reduces dependence on one arbitrary split by repeating the train-and-evaluate process over multiple partitions.

For k-fold cross-validation, the procedure is:

1. Divide the available development data into k folds.

2. Hold out one fold for validation.

3. Train a fresh model using the remaining k − 1 folds.

4. Evaluate the model on the held-out fold.

5. Repeat until every fold has served as validation data.

6. Aggregate the validation metrics.

For example, with five folds, each iteration trains on approximately 80% of the development data and validates on approximately 20%.

The average score is:

CV score = (score₁ + score₂ + … + scoreₖ) / k

But there’s an important limitation: cross-validation does not automatically guarantee a reliable estimate of future performance.

It estimates performance under the assumptions imposed by its splitting strategy.

If those assumptions do not match deployment, even perfectly implemented cross-validation can produce the wrong conclusion.

2. Ordinary k-fold cross-validation

The standard KFold splitter in scikit-learn divides a dataset into k partitions. Each partition becomes the validation set once, and the remaining partitions form the training set.

By default, KFold does not shuffle observations. Enabling shuffling randomizes fold membership before the partitions are formed.

scikit-learn 1.9.1 documentation

When ordinary k-fold works well

K-fold is a reasonable choice when:

• Observations are approximately independent.

• There is no important temporal ordering.

• The same underlying data-generating process applies to training and future observations.

• No related entities are distributed across folds in a way that creates leakage.

• The evaluation goal is to generalize to new observations drawn from a similar population.

Examples include predicting prices for independently sampled products or estimating a property value using a suitably constructed housing dataset.

However, ordinary k-fold has two major limitations.

First, its folds may contain very different class proportions. Second, when observations have meaningful chronological or group relationships, ordinary k-fold does not inherently preserve those relationships.

Both problems require more deliberate splitting strategies.

3. Prepare the Python environment

We’ll use scikit-learn and NumPy throughout the tutorial. The examples generate their own synthetic data, so there are no external datasets to download.

Create a virtual environment:

python -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

Install the required libraries:

python -m pip install --upgrade numpy scikit-learn

All examples use the standard scikit-learn model-selection APIs, including KFold, StratifiedKFold, TimeSeriesSplit, cross_validate, and GridSearchCV.

4. Stratified k-fold: protecting class proportions

Suppose 90% of your e-commerce deliveries arrive on time and 10% arrive late.

Ordinary k-fold cross-validation doesn’t guarantee that each validation fold will contain approximately 10% late deliveries.

Some folds might contain very few positive examples. In extreme circumstances, a validation fold could contain only one class.

This makes evaluation difficult, especially for metrics that depend on observing both classes.

Stratified k-fold addresses this by approximately preserving the target-class proportions in every fold.

A reproducible example

Consider a small dataset containing 200 orders:

• 180 on-time deliveries.

• 20 late deliveries.

For demonstration, suppose these records happen to be sorted by their target label.

Compare ordinary k-fold without shuffling against stratified k-fold:

import numpy as np

from sklearn.model_selection import KFold, StratifiedKFold

# 200 orders: 180 on time, 20 late.
y = np.array([0] * 180 + [1] * 20)

# Placeholder features; only fold composition matters here.
X = np.arange(len(y)).reshape(-1, 1)

strategies = {
    "Ordinary KFold": KFold(n_splits=5),
    "StratifiedKFold": StratifiedKFold(
        n_splits=5,
        shuffle=True,
        random_state=42,
    ),
}

for name, cv in strategies.items():
    print(f"\n{name}")

    for fold, (_, test_idx) in enumerate(
        cv.split(X, y), start=1
    ):
        positives = y[test_idx].sum()
        total = len(test_idx)

        print(
            f"Fold {fold}: "
            f"{positives}/{total} late deliveries"
        )

The ordinary k-fold result is:

Ordinary KFold
Fold 1: 0/40 late deliveries
Fold 2: 0/40 late deliveries
Fold 3: 0/40 late deliveries
Fold 4: 0/40 late deliveries
Fold 5: 20/40 late deliveries

The stratified result is:

StratifiedKFold
Fold 1: 4/40 late deliveries
Fold 2: 4/40 late deliveries
Fold 3: 4/40 late deliveries
Fold 4: 4/40 late deliveries
Fold 5: 4/40 late deliveries

This example deliberately uses class-sorted data to expose the danger of non-shuffled k-fold. Shuffling ordinary k-fold would generally make its folds more representative, but it still would not guarantee balanced class proportions.

Stratification deliberately constructs validation sets containing approximately the same percentage of each class.

That makes it especially useful when evaluating classifiers on imbalanced datasets.

Stratification improves evaluation stability, not everything else

There is an important subtlety here.

Scikit-learn explicitly notes that stratification primarily solves an engineering problem: avoiding problematic fold compositions. It does not automatically solve statistical uncertainty, distribution shift, or data leakage.

scikit-learn 1.9.1 documentation

Stratification can actually reduce the observed variability of scores between folds because it makes their class distributions more similar.

And if the class distribution changes in production, preserving the historical ratio can hide that shift.

Stratification is appropriate for many independent classification datasets. It is not a replacement for time-aware validation.

5. The temporal trap: when random cross-validation lies

Now for the most important experiment.

Imagine an e-commerce business predicting whether an order will arrive late.

Initially, longer-distance deliveries are more likely to be late. Later, a new logistics policy changes the relationship: long-distance orders receive priority shipping, while short-distance orders are handled by a slower service.

The model needs to operate under a changing business process.

We will generate 1,200 consecutive daily observations and simulate this change.

The dataset has two features:

• order_day: The chronological day number.

• distance: A normalized shipping-distance measurement.

The target is binary:

• 0: Delivery on time.

• 1: Delivery late.

For the first 850 days, distance above 0.5 predicts late delivery. Starting on day 850, the relationship reverses.

We also randomly flip 3% of labels to introduce noise.

This is an intentionally simplified and abrupt distribution shift. It isolates the danger we want to study: training on outcomes from a future business regime that would not yet be known at the prediction date.

Step 1: Generate the dataset

import numpy as np

from sklearn.model_selection import (
    KFold,
    StratifiedKFold,
    TimeSeriesSplit,
    cross_validate,
)
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(21)

n = 1200

# One observation per consecutive day.
order_day = np.arange(n)

# Distance is available when an order is placed.
distance = rng.uniform(0.0, 1.0, size=n)

# A logistics-policy change begins on day 850.
after_change = order_day >= 850

# Before the change, longer distance means late.
# After the change, the relationship reverses.
y = ((distance > 0.50) ^ after_change).astype(int)

# Introduce 3% random label noise.
flips = rng.random(n) < 0.03
y = np.where(flips, 1 - y, y)

# Features available at prediction time.
X = np.column_stack([order_day, distance])

print("Observations:", len(X))
print("Overall late-delivery rate:", round(y.mean(), 3))

The model receives the day and the distance, but it isn’t given the hidden policy-change flag.

It can learn that the relationship changed only if its training data includes labeled examples from the newer regime.

This distinction matters: the day number itself is not a leaked feature. An order’s date is legitimately known when making the prediction.

The problem arises when a validation procedure allows the model to learn from future outcomes.

Step 2: Define three validation strategies

We’ll compare:

1. Ordinary shuffled k-fold.

2. Shuffled stratified k-fold.

3. Chronological time-series cross-validation.

All three use five validation folds.

strategies = {
    "KFold (shuffled)": KFold(
        n_splits=5,
        shuffle=True,
        random_state=7,
    ),
    "StratifiedKFold (shuffled)": StratifiedKFold(
        n_splits=5,
        shuffle=True,
        random_state=7,
    ),
    "TimeSeriesSplit": TimeSeriesSplit(
        n_splits=5,
    ),
}

The first two strategies randomly distribute observations across training and validation partitions.

The third keeps the observations chronological and trains only on data preceding each validation period.

Step 3: Train the same model under every strategy

We will use a decision tree so the results remain easy to understand.

The classifier, features, labels, and evaluation metrics will remain identical. Only the cross-validation strategy changes.

model = DecisionTreeClassifier(
    max_depth=5,
    min_samples_leaf=12,
    random_state=7,
)

for name, cv in strategies.items():
    results = cross_validate(
        estimator=model,
        X=X,
        y=y,
        cv=cv,
        scoring={
            "accuracy": "accuracy",
            "balanced_accuracy": "balanced_accuracy",
        },
        error_score="raise",
    )

    accuracy = results["test_accuracy"]
    balanced = results["test_balanced_accuracy"]

    print(f"\n{name}")
    print("Fold accuracies:", accuracy.round(3))
    print(f"Mean accuracy: {accuracy.mean():.3f}")
    print(f"Worst fold: {accuracy.min():.3f}")
    print(f"Balanced accuracy: {balanced.mean():.3f}")

The program fits a fresh estimator for each fold.

It calculates both standard accuracy and balanced accuracy, which averages recall across the two classes.

Step 4: Examine the results

With the seeded synthetic data, the experiment produces the following comparison.

Figures are rounded from the reproducible experiment. Small differences are possible across software versions.

Mean cross-validation accuracy

Shuffled KFold

96.8%

Stratified KFold

96.8%

TimeSeriesSplit

82.0%

The random strategies report nearly 97% accuracy. That sounds excellent.

But chronological validation discovers a fold where the classifier correctly predicts only about one-quarter of orders.

The reason isn’t that the tree is inherently bad. The difference arises because the two validation approaches answer different questions.

Random cross-validation asks:

Can I predict held-out records when my training data may contain observations from both before and after the policy change?

Chronological validation asks:

Can a model trained on information available before a particular date predict what happens afterward?

For a model deployed to predict future deliveries, the second question is the one that matters.

6. Understanding the time-series folds

Let’s inspect how TimeSeriesSplit organizes the experiment.

With 1,200 observations and five default splits, the test sets each contain 200 consecutive observations.

The folds look like this:

Fold 1
Train: days    0–199
Test:  days  200–399

Fold 2
Train: days    0–399
Test:  days  400–599

Fold 3
Train: days    0–599
Test:  days  600–799

Fold 4
Train: days    0–799
Test:  days  800–999

Fold 5
Train: days    0–999
Test:  days 1000–1199

The training window expands as time progresses.

Importantly, no fold uses observations occurring after its validation period to train the model.

The fifth fold can use days 850–999 because those outcomes already belong to its historical training period.

Now examine the individual accuracies.

Accuracy across chronological foldsThe policy changes on day 850, within fold 4’s validation window.

The fourth fold is the critical one.

Its training data ends on day 799, so the model has never seen the new logistics policy. Its validation period covers days 800–999, including 150 days under the changed relationship.

The model continues applying the old relationship and performs poorly.

By the fifth fold, the training set includes 150 observations from the new policy, allowing the tree to learn the updated behavior. Performance recovers.

This teaches three important lessons.

• A mean score can hide a disastrous period. An 82% mean does not adequately describe a fold that achieves only 25.5%.

• Model performance depends on historical context. A model trained before a change may fail, while one retrained after observing that change may recover.

• Validation design defines what can be learned. Random splitting permits learning from a regime that did not yet exist at the simulated prediction date.

There is another limitation: chronological validation cannot predict every unexpected future change. It can only reveal how the chosen training procedure would have performed across the historical transitions represented in the data.

If the available history contains no example of a particular future disruption, no validation technique can guarantee robustness to it.

7. TimeSeriesSplit: expanding versus rolling windows

Scikit-learn’s TimeSeriesSplit supports several useful controls:

• n_splits: Number of chronological validation splits.

• test_size: Number of observations in each test window.

• gap: Number of observations excluded between the training and test windows.

• max_train_size: Maximum number of observations allowed in the training window.

These parameters are documented in the current scikit-learn API.

scikit-learn 1.9.1 documentation

Expanding-window validation

With an expanding window, each new training set contains all available observations from earlier folds.

Fold 1: [ TRAIN ][ TEST  ]
Fold 2: [   TRAIN   ][ TEST  ]
Fold 3: [     TRAIN     ][ TEST  ]
Fold 4: [       TRAIN       ][ TEST  ]

This matches a system that regularly retrains using its entire available historical dataset.

It works well when older observations remain useful and long-term patterns are stable enough to learn from.

However, expanding windows can be a poor match for rapidly changing environments.

A model might give too much influence to outdated customer behavior, pricing policies, or operational processes.

Rolling-window validation

A rolling or sliding window limits how much historical data is used.

Instead of training on the entire past, it trains on a fixed number of recent observations.

Fold 1: [ TRAIN ][ TEST  ]
Fold 2:     [ TRAIN ][ TEST  ]
Fold 3:         [ TRAIN ][ TEST  ]
Fold 4:             [ TRAIN ][ TEST  ]

This can better represent a production system that retrains using only the latest 90 days or 12 months.

In scikit-learn, you can implement this with max_train_size.

Adding a gap

Imagine you’re predicting whether an order will arrive late, but the final delivery outcome may not be available until several days after the order is placed.

A training row immediately preceding the validation period might not have had its label available at the simulated training cutoff.

A gap can help model that operational delay.

Here is a rolling-window configuration that uses 500 days of training history, excludes seven observations, and validates on the next 100 days:

from sklearn.model_selection import TimeSeriesSplit

rolling_cv = TimeSeriesSplit(
    n_splits=3,
    max_train_size=500,
    test_size=100,
    gap=7,
)

for fold, (train_idx, test_idx) in enumerate(
    rolling_cv.split(X), start=1
):
    print(f"Fold {fold}")

    print(
        "Training period:",
        train_idx[0],
        "to",
        train_idx[-1],
    )

    print(
        "Validation period:",
        test_idx[0],
        "to",
        test_idx[-1],
    )

    print(
        "Excluded observations:",
        test_idx[0] - train_idx[-1] - 1,
    )

Because the synthetic dataset contains exactly one observation per day, a seven-row gap corresponds to seven days.

For datasets containing many orders per day, seven rows would not represent seven calendar days.

That’s an important distinction.

The gap is measured in observations, not calendar units.

For irregularly spaced data, you may need custom calendar-based splits rather than relying directly on row counts.

Also, choose the gap based on actual label-availability delays, forecast horizons, and any other leakage risks. Arbitrarily adding a gap does not guarantee that every feature or target is safe.

8. The hidden problem: preprocessing leakage

Even with the correct cross-validation splitter, model evaluation can remain invalid if preprocessing learns from observations outside the training fold.

Common examples include:

• Fitting a scaler on the entire dataset before cross-validation.

• Calculating missing-value imputation statistics using validation rows.

• Selecting features based on correlations with the target across all observations.

• Encoding categories using target statistics learned from validation data.

• Creating aggregate features using information that would not have existed at prediction time.

Consider a standardization procedure.

If you compute the mean and standard deviation of a feature using the complete dataset, validation observations influence those statistics.

For a temporal prediction system, those statistics may include information about future periods.

The correct approach is to fit the transformation only on the training portion of each fold.

Pipelines solve fold-local preprocessing

Scikit-learn’s Pipeline combines preprocessing and modeling so that fitting a pipeline inside cross-validation fits its transformers on the appropriate training subset.

scikit-learn 1.9.1 documentation

For example:

from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier

pipeline = Pipeline(
    steps=[
        (
            "impute",
            SimpleImputer(strategy="median"),
        ),
        (
            "model",
            DecisionTreeClassifier(
                random_state=7,
            ),
        ),
    ]
)

Our synthetic dataset contains no missing values, so imputation is not needed for this particular experiment. It is included to demonstrate safe handling of learned preprocessing.

A pipeline prevents the imputer from learning medians from its held-out validation fold.

It does not, however, fix features that were fundamentally constructed using unavailable future information before entering the pipeline.

A leakage-safe pipeline and a leakage-safe feature-generation process are separate requirements.

9. Model selection with chronological cross-validation

Cross-validation is not just for reporting a score. It is frequently used to choose hyperparameters.

For example, a decision tree can vary in:

• Maximum depth.

• Minimum samples per leaf.

• Minimum samples required for splitting.

The problem is that tuning hyperparameters using an unsuitable validation scheme will optimize for the wrong deployment conditions.

A randomly tuned model might perform exceptionally well when both old and new business regimes appear in its training data, but fail when forecasting a genuinely unseen future period.

A safer GridSearchCV workflow

We’ll reserve the final 200 observations as an untouched chronological test set.

Hyperparameter selection will use only the first 1,000 observations and a time-series splitter.

This snippet continues from the previously generated X, y, and pipeline variables.

from sklearn.metrics import balanced_accuracy_score
from sklearn.model_selection import GridSearchCV, TimeSeriesSplit

# Chronological development and final test periods.
X_dev = X[:1000]
y_dev = y[:1000]

X_test = X[1000:]
y_test = y[1000:]

cv = TimeSeriesSplit(
    n_splits=4,
    gap=7,
)

param_grid = {
    "model__max_depth": [3, 5, 8],
    "model__min_samples_leaf": [5, 12, 30],
}

search = GridSearchCV(
    estimator=pipeline,
    param_grid=param_grid,
    cv=cv,
    scoring="balanced_accuracy",
    n_jobs=-1,
    refit=True,
)

search.fit(X_dev, y_dev)

predictions = search.predict(X_test)

final_score = balanced_accuracy_score(
    y_test,
    predictions,
)

print("Best parameters:", search.best_params_)
print(f"Selection CV score: {search.best_score_:.3f}")
print(f"Final holdout score: {final_score:.3f}")

With the generated dataset, this should produce a selection CV score of approximately 0.79 and a final holdout balanced accuracy of approximately 0.96.

The final period performs well because, by day 1000, the development data already contains observations from the changed logistics policy.

This doesn’t contradict the catastrophic fourth chronological fold. Each fold simulates a different moment in history with a different amount of available knowledge.

Why the final test set matters

The workflow separates three different activities:

1. Model fitting: Learning parameters from each training fold.

2. Model selection: Comparing hyperparameter settings using the validation folds.

3. Final evaluation: Testing the selected procedure on an untouched future period.

The best cross-validation score reported by a hyperparameter search is not a completely unbiased estimate of its generalization performance.

Why?

Because you explicitly selected the hyperparameters that performed best on those folds. Some apparent improvement may reflect chance variation in the validation results.

The final holdout gives you a separate evaluation after the selection process has finished.

A crucial rule follows:

Once you examine your final test results and make decisions based on them, that test set is no longer an untouched source of evidence for those decisions.

If you need to compare many model families and experimental ideas, keep the final holdout separate until the selection process is complete, or use a carefully designed nested validation procedure.

10. Cherry on the cake: a real bike-sharing forecast exposes the same trap

This isn’t only a synthetic problem.

The official scikit-learn documentation contains a concrete forecasting example using real bike-sharing demand observations.

The task is to predict the number of bike rentals in the next hour using historical demand information.

The example builds lagged features, fits a histogram-based gradient boosting regressor, and compares random splitting with chronological evaluation.

The surprising part is how much the evaluation depends on the chosen split.

Random holdout

Mean absolute percentage errorFirst chronological fold

Mean absolute percentage errorResults published in scikit-learn’s lagged-feature forecasting example. Lower is better. The two evaluations use different train/test periods and sizes, so these figures are illustrative rather than a controlled same-test-set comparison. scikit-learn 1.9.1 documentationThe random holdout reports MAPE of approximately 0.396, while the first chronological fold reports approximately 0.458.

The example also evaluates three chronological folds:

• First fold: approximately 45.8% MAPE.

• Second fold: approximately 28.3% MAPE.

• Third fold: approximately 36.4% MAPE.

These variations demonstrate another important point: one temporal validation period may be substantially easier or harder than another.

The mean of those three chronological scores is approximately 36.8%, so it would be incorrect to claim that the chronological average is worse than the random holdout in that particular example.

Instead, the essential finding is that a shuffled holdout does not reproduce forward forecasting conditions, and the individual chronological scores reveal substantial period-to-period variation that a single random holdout cannot show.

The example is a documented failure of an evaluation assumption, not a reported production outage. It is valuable precisely because it demonstrates the risk on a real forecasting task rather than only on a contrived dataset.

You can explore the complete example in the official scikit-learn lagged-feature forecasting tutorial.

scikit-learn 1.9.1 documentation

11. Five mistakes that invalidate otherwise good cross-validation

Mistake 1: Using a random split because the data is already in a table

A timestamped dataset does not become independent merely because it’s represented as rows and columns.

Orders, energy usage, website traffic, and server latency measurements can all have temporal relationships.

Before selecting a splitter, ask whether future observations would genuinely be available during model training.

For a forecasting model, the answer is generally no.

Mistake 2: Confusing stratification with leakage prevention

Stratification protects class proportions, not time ordering.

A stratified fold can still contain validation observations from March while its training partition contains labeled observations from June.

That may be perfectly acceptable when evaluating exchangeable observations from a stable population.

It is not appropriate for evaluating a forecasting system as it would have operated in March.

If you’re working with rare events in chronological data, consider metrics and temporal windows carefully rather than automatically applying randomized stratification.

Mistake 3: Creating features using future information

Suppose you predict next-day electricity consumption.

Useful features might include:

• Electricity consumption yesterday.

• Consumption over the last seven completed days.

• Day of the week.

• Historical weather observations available at prediction time.

Dangerous features include:

• Tomorrow’s measured consumption.

• A centered rolling average that includes future observations.

• A future day’s actual temperature when only a forecast would have been available.

• Aggregate statistics calculated using the entire evaluation period.

A correct lagged feature must reflect the information available at the actual prediction cutoff.

If you produce a seven-day-ahead forecast, yesterday’s actual consumption may not be available for every predicted day in the horizon. The feature-generation logic must reflect whether you’re using recursive forecasts, directly predicting the horizon, or updating predictions as actual observations arrive.

This is why feature availability is part of the validation design, not simply a preprocessing detail.

Mistake 4: Ignoring repeated entities

Consider predicting a customer’s future purchasing behavior.

If the same customer appears in both training and validation folds, the model may benefit from customer-specific patterns.

Whether that is a problem depends on the deployment objective.

If you’re predicting future orders from existing customers, historical observations from those customers may be legitimate training data.

If you’re trying to generalize to entirely new customers, allowing the same customers in both partitions produces an inappropriate evaluation.

Scikit-learn provides GroupKFold and StratifiedGroupKFold for group-aware validation.

StratifiedGroupKFold attempts to preserve class proportions while keeping groups separated, although perfect stratification may not be achievable.

scikit-learn 1.9.1 documentation

For datasets containing both repeated entities and time dependence, neither ordinary group splitting nor ordinary chronological splitting is automatically sufficient. You may need custom folds that enforce both chronological cutoffs and the required group separation.

Mistake 5: Reporting only the average

An average cross-validation score hides important details.

Imagine two systems with these five validation accuracies:

Model A performs well in four folds but fails dramatically in one.

Model B performs more consistently and also has a higher mean.

In applications where failure during unusual operating periods is costly, fold-level performance can be more important than a single summary number.

When reporting cross-validation, consider including:

• Mean and standard deviation of fold scores.

• Individual fold scores, especially for chronological evaluation.

• Worst-fold performance.

• Class counts or class prevalence in each validation period.

• Performance by relevant time period, customer segment, or operating regime.

Be careful about treating the standard deviation of correlated cross-validation folds as a formal uncertainty interval. In particular, expanding-window training sets overlap, and their fold scores are not independent measurements.

12. Choosing the right cross-validation strategy

The best cross-validation method depends on the generalization problem you’re trying to solve.

This table is a starting point, not a mechanical decision rule.

For example, consider predicting whether an order will arrive late.

If your model is trained once using a randomly sampled historical dataset and evaluated on similar independent observations from the same period, stratified k-fold may be appropriate.

If that model will be retrained each week and used to predict orders arriving in subsequent weeks, chronological evaluation better matches the operational question.

If you’re simultaneously evaluating generalization to new merchants, the validation design may also need to separate merchant groups.

The correct choice depends on what information is available, what entities recur, and what kind of future uncertainty matters.

13. A practical production evaluation checklist

Before trusting a cross-validation result, go through the following checklist.

• Define the prediction moment. Know exactly when the prediction is made and which information exists at that time.

• Identify the generalization target. Are you predicting new rows, new customers, new locations, or future periods?

• Preserve meaningful boundaries. Keep time ordering and entity-group relationships intact when deployment requires it.

• Account for label availability. Exclude training records whose labels would not have been known at the historical training cutoff.

• Fit preprocessing inside each fold. Use pipelines for learned transformations, feature selection, and model fitting.

• Choose an appropriate metric. Accuracy may conceal minority-class failures; forecasting tasks may be better evaluated using MAE, RMSE, or another domain-appropriate loss.

• Use realistic validation horizons. If the business cares about the next 30 days, evaluate on comparable 30-day periods.

• Inspect individual folds. Look for failures around promotions, seasonal peaks, policy changes, and other operational transitions.

• Keep a final holdout. Avoid repeatedly adapting your experiments to the same supposedly untouched test period.

• Monitor production drift. Cross-validation provides historical evidence; it does not guarantee future stability.

One especially important operational detail is the relationship between retraining frequency and evaluation frequency.

If a model is retrained monthly, evaluating it with daily updates to its training set may give an unrealistic advantage.

Similarly, if the production system uses a one-year rolling training window, evaluating with an unlimited expanding window simulates a different learning procedure.

The validation design should imitate the complete workflow, including the timing of retraining and the delay before new labels become available.

14. Key takeaways

Cross-validation is powerful because it evaluates model behavior across multiple partitions. But the quality of the result depends fundamentally on how those partitions are constructed.

Ordinary k-fold is useful for approximately independent observations. Stratified k-fold helps preserve class proportions and makes evaluation more practical when classes are imbalanced. Neither automatically makes chronological forecasting evaluation valid.

For future-facing predictions, chronological cross-validation is usually the stronger starting point because it respects the direction in which information becomes available.

The synthetic shipping experiment demonstrated the consequences:

• Random k-fold achieved approximately 96.8% mean accuracy.

• Stratified k-fold achieved approximately 96.8% mean accuracy.

• Time-series cross-validation achieved approximately 82.0% mean accuracy and exposed a disastrous 25.5% fold.

The difference was not caused by changing the estimator or the target. It came from changing which observations the estimator was allowed to learn from before making each validation prediction.

That is the critical lesson: cross-validation is not just a scoring technique. It is a simulation of the model’s operating environment.

If the simulation allows information that would not be available in production, the score can be convincing and still be misleading.

Your next step: audit one of your own models

Take an existing machine learning project and rerun its evaluation using a splitter that more closely matches deployment.

Compare the average scores, the worst validation fold, the class distributions, and the actual information available at each prediction cutoff.

If your data has a temporal component, try TimeSeriesSplit with a realistic validation horizon and retraining window. If your data has repeated entities, investigate group-aware validation. Keep all learned preprocessing within the folds.

Make one change today: stop choosing your cross-validation strategy by habit. Choose it by the prediction conditions your model will face in the real world.