Detecting data drift in production with the K–S test and PSI

Why drift detection belongs in production monitoring

A machine-learning model can be perfectly healthy from an infrastructure perspective and still become increasingly wrong.

Its API responds in milliseconds. CPU and memory remain within their expected ranges. No exceptions appear in the logs. The feature pipeline produces the correct columns, and every prediction request returns a valid response. Yet business performance slowly deteriorates because the world that generates production data no longer resembles the world represented by the training data.

This is the problem that data drift detection helps solve. Rather than waiting for delayed ground-truth labels or a visible drop in business metrics, a monitoring system compares incoming feature distributions with a reference distribution and raises an alert when the differences become significant enough to investigate.

Two techniques are especially useful for tabular machine-learning systems:

• The two-sample Kolmogorov–Smirnov (K–S) test: A statistical hypothesis test that measures differences between two empirical distributions of a continuous variable.

• The Population Stability Index (PSI): A distribution-comparison metric that measures how much population proportions have changed across predefined bins or categories.

They complement each other. The K–S test provides a statistical significance measure and a distribution-distance statistic. PSI provides a practical, interpretable score that monitoring teams can use to classify changes as relatively small or large.

In this tutorial, we will build a runnable Python monitoring example using synthetic e-commerce and service-telemetry data, establish practical alert thresholds, and discuss how to turn the results into an operational MLOps workflow.

1. Understand what is actually drifting

Before writing a monitoring job, distinguish three related problems.

Data drift

Data drift occurs when the distribution of model inputs changes over time.

For example, an e-commerce order-value model might have been trained when most orders were between €20 and €80. Following a successful premium-product campaign, the production population contains many more orders above €100.

The distribution of a feature has changed, even if the feature’s name, type, and schema remain identical.

In statistical notation, the input distribution P(X) has changed.

Concept drift

Concept drift occurs when the relationship between inputs and the target changes.

Imagine a delivery-time model trained on historical traffic patterns. A new warehouse-routing policy changes the relationship between distance, congestion, and actual delivery time. Even if the feature distributions remain similar, the model’s learned relationships may no longer be reliable.

Concept drift is particularly important because univariate feature-distribution tests may not detect it.

Changes in the target distribution

The distribution of the outcome itself can also change. For example, the proportion of orders that are returned may rise because of a new product category or a change in customer behavior.

When ground-truth labels are available, monitoring the target distribution and actual predictive performance adds information that input-only monitoring cannot provide.

The practical distinction is simple:

• Data drift monitoring asks whether incoming inputs look different.

• Performance monitoring asks whether the model is still doing its job.

• Concept-drift investigation asks whether the relationship the model learned remains valid.

A drift alert is evidence that something changed, not proof that model quality has deteriorated. Some changes are harmless, some are beneficial, and some are operationally critical.

2. Establish a trustworthy reference dataset

Both K–S and PSI need two populations to compare: a reference population and a current population.

Choosing those populations correctly is one of the most consequential design decisions in a production monitoring system.

Option A: Compare live data with training data

This answers a useful question: how different is today’s input population from the population the model originally learned from?

It is valuable when the training distribution represents an important operating assumption.

However, a permanent training baseline can produce recurring alerts for expected seasonality. A retail model trained in January may legitimately encounter different customer behavior during a holiday promotion.

Option B: Compare live data with a stable production period

A second approach is to select a known-good production window, such as the first four weeks after a successful deployment.

This helps distinguish changes from the distribution the deployed model has actually been operating under.

The risk is that a production baseline can itself become stale or incorporate an already degraded state. It must be versioned, reviewed, and updated deliberately.

Option C: Use seasonal or segmented references

For systems with strong seasonal patterns, compare the current period with a comparable historical period. For example, compare this week’s weekday traffic with similar weekdays rather than with a holiday weekend.

You can also establish separate baselines for important segments, such as traffic source, product category, geographic region, or device type.

A good reference dataset should have:

• A known model version and time range.

• The same feature definitions and preprocessing conventions used by the deployed model.

• Documented handling of missing values, outliers, and invalid records.

• Sufficient observations to detect the changes that matter operationally.

• A clear explanation of why the reference period represents acceptable behavior.

Do not silently rebuild the reference distribution every time an alert fires. Doing so can normalize gradual deterioration until the monitoring system no longer recognizes it as unusual.

3. The Kolmogorov–Smirnov test, explained

The two-sample K–S test compares two samples drawn from continuous distributions. SciPy’s current implementation, scipy.stats.ks_2samp, returns a K–S statistic and a p-value, and supports automatic selection between exact and asymptotic p-value calculations. The SciPy release page lists version 1.18.1, released in August 2026, as the current stable release at the time of writing (ks_2samp documentation).

What the test measures

Imagine plotting the cumulative distribution of training-time order values and the cumulative distribution of live order values.

At every possible value, each cumulative distribution tells you the proportion of observations less than or equal to that value.

The K–S statistic, D, is the maximum absolute vertical distance between the two empirical cumulative distribution functions.

In plain language:

• A D value near zero means the empirical distributions are similar.

• A larger D value means there is a more substantial difference somewhere along the distributions.

• The p-value measures how surprising the observed statistic would be under the null hypothesis that the two underlying distributions are identical.

The statistic is useful because it captures differences in location, spread, and distribution shape without requiring you to assume that the data follows a normal distribution.

For example, the average order value might change only slightly while the proportion of exceptionally large orders rises sharply. A distribution-level test can reveal differences that a comparison of means alone would miss.

What the p-value does not tell you

A p-value is not the probability that the null hypothesis is true. Nor does a small p-value automatically mean the difference is important to the business.

This matters because the K–S test is sensitive to sample size. Given enough observations, even a relatively small distributional difference may become statistically significant.

Consider a live window containing hundreds of thousands of requests. A tiny shift in request latency could produce a very small p-value without meaningfully affecting model predictions.

For this reason, a production alert should not rely exclusively on a significance threshold.

A stronger policy combines:

1. A p-value threshold.

2. A minimum K–S effect-size threshold.

3. Minimum sample-size requirements.

4. Operational context and, where possible, observed model-performance changes.

Important assumptions and limitations

The standard two-sample K–S test is intended for independent samples from continuous distributions.

That qualification matters. Applying the ordinary test indiscriminately to categorical values or heavily discrete variables, such as a count that takes only a handful of integer values, can make its p-values inappropriate.

For categorical features, use a suitable categorical test or PSI. For discrete numerical features, select a method appropriate to the variable’s distribution rather than assuming that the standard continuous K–S test is always valid.

Also consider temporal dependence. Consecutive requests from the same campaign, customer, or machine may not be independent. When observations are strongly correlated, naive significance calculations can overstate the evidence provided by a sample.

4. Population Stability Index: measuring the size of a shift

The Population Stability Index takes a different approach. Instead of directly comparing empirical cumulative distributions, it divides a feature into bins and compares the proportion of observations in each bin.

For each bin, calculate:

• The reference proportion.

• The current proportion.

• The logarithm of the ratio between those proportions.

Then sum the contributions across all bins.

The formula is:

PSI = Σᵢ (Current proportionᵢ − Reference proportionᵢ) × ln(Current proportionᵢ / Reference proportionᵢ)

A PSI value of zero means the binned proportions are identical. Larger values indicate greater differences between the reference and current populations.

Choosing bins correctly

For a continuous feature such as order value, one practical strategy is to calculate quantile-based bin boundaries from the reference dataset.

With ten quantile bins, each bin initially contains approximately 10% of the reference observations. You then retain those boundaries for every subsequent monitoring window.

This fixed-bin requirement is essential.

If you recompute the quantile boundaries independently for each live window, the bins will adapt to the new distribution and conceal part of the change you intended to measure.

For categorical features, compare the proportions of each category. Include an explicit missing-value category and account for previously unseen categories rather than silently dropping them.

Handling zero proportions

Suppose the reference population contains no observations in a particular bin, but the live population contains several.

The logarithm in the PSI formula becomes undefined when a proportion is zero. A common practical solution is to replace zero or extremely small proportions with a small positive value, then renormalize the proportions before calculating the score.

This is a numerical safeguard, not a substitute for thoughtful binning. An excessive number of sparse bins can make PSI unstable, particularly when the live window is small.

Practical PSI alert thresholds

The following values are widely used as initial monitoring heuristics:

These are starting thresholds, not universal statistical guarantees. They do not establish whether model performance has changed, and they should not be treated as equally appropriate for every feature.

For a highly sensitive fraud-detection feature, even a small change might warrant immediate investigation. For a noisy operational metric that naturally fluctuates, a PSI of 0.10 might be routine.

Calibrate the thresholds against historical windows, known harmless changes, and actual performance outcomes. Use different policies for features with different business importance.

5. Set up the Python environment

We will compare synthetic training and live data representing an e-commerce platform. The dataset includes order value, delivery distance, request latency, and customer region.

No external dataset is required. The example deliberately introduces shifts in some features while leaving another feature largely unchanged.

Create an isolated Python environment and install NumPy, pandas, and SciPy:

python -m venv .venv

Activate the environment using the command appropriate for your operating system, then install the dependencies:

python -m pip install --upgrade pip
python -m pip install numpy==2.5.4 pandas==3.0.6 scipy==1.18.1

The implementation uses SciPy’s current K–S test and its false-discovery-control function to adjust for multiple feature comparisons. The latter implements procedures including Benjamini–Hochberg, which helps control the false discovery rate when testing many hypotheses (false_discovery_control documentation).

6. Build a runnable train-versus-live drift detector

The following script performs four tasks:

1. Generates a reference dataset and a live dataset with known distribution changes.

2. Calculates the K–S statistic and p-value for continuous numerical features.

3. Calculates PSI for both numerical and categorical features.

4. Applies practical alert thresholds and prints a monitoring report.

The example uses an adjusted K–S p-value below 0.01 together with a K–S statistic of at least 0.05 to trigger a K–S alert. For PSI, it uses 0.10 as a warning threshold and 0.20 as an alert threshold.

Those values are illustrative operational defaults, not universally optimal settings.

import numpy as np
import pandas as pd
from scipy.stats import false_discovery_control, ks_2samp

# ---------------------------------------------------------
# 1. Generate reproducible reference and live populations
# ---------------------------------------------------------

rng = np.random.default_rng(42)

n_reference = 5000
n_live = 1200

reference = pd.DataFrame({
    "basket_value": rng.lognormal(
        mean=3.5, sigma=0.45, size=n_reference
    ),
    "delivery_distance_km": rng.gamma(
        shape=2.5, scale=3.0, size=n_reference
    ),
    "request_latency_ms": rng.lognormal(
        mean=np.log(120), sigma=0.25, size=n_reference
    ),
    "region": rng.choice(
        ["north", "south", "east", "west"],
        p=[0.25, 0.25, 0.25, 0.25],
        size=n_reference,
    ),
})

# Introduce drift in basket value, latency, and region mix.
# Delivery distance retains approximately the same distribution.
live = pd.DataFrame({
    "basket_value": rng.lognormal(
        mean=3.72, sigma=0.50, size=n_live
    ),
    "delivery_distance_km": rng.gamma(
        shape=2.5, scale=3.0, size=n_live
    ),
    "request_latency_ms": rng.lognormal(
        mean=np.log(155), sigma=0.35, size=n_live
    ),
    "region": rng.choice(
        ["north", "south", "east", "west"],
        p=[0.18, 0.22, 0.25, 0.35],
        size=n_live,
    ),
})


# ---------------------------------------------------------
# 2. PSI for continuous numerical features
# ---------------------------------------------------------

def psi_numeric(reference_values, current_values, bins=10, epsilon=1e-6):
    reference_values = np.asarray(reference_values, dtype=float)
    current_values = np.asarray(current_values, dtype=float)

    # Invalid and missing numeric values must be monitored separately.
    reference_values = reference_values[np.isfinite(reference_values)]
    current_values = current_values[np.isfinite(current_values)]

    if reference_values.size == 0 or current_values.size == 0:
        raise ValueError("PSI requires non-empty finite samples.")

    # Fit quantile boundaries on the reference population only.
    quantiles = np.quantile(
        reference_values, np.linspace(0, 1, bins + 1)
    )
    inner_edges = np.unique(quantiles[1:-1])

    # Infinite outer edges include values outside the reference range.
    edges = np.concatenate(([-np.inf], inner_edges, [np.inf]))

    reference_counts, _ = np.histogram(reference_values, bins=edges)
    current_counts, _ = np.histogram(current_values, bins=edges)

    expected = reference_counts / reference_counts.sum()
    actual = current_counts / current_counts.sum()

    # Smooth zero proportions and renormalize.
    expected = np.clip(expected, epsilon, None)
    actual = np.clip(actual, epsilon, None)

    expected /= expected.sum()
    actual /= actual.sum()

    return float(
        np.sum((actual - expected) * np.log(actual / expected))
    )


# ---------------------------------------------------------
# 3. PSI for categorical features
# ---------------------------------------------------------

def psi_categorical(reference_values, current_values, epsilon=1e-6):
    reference_values = (
        pd.Series(reference_values, dtype="string")
        .fillna("__MISSING__")
    )
    current_values = (
        pd.Series(current_values, dtype="string")
        .fillna("__MISSING__")
    )

    # Include categories found in either population.
    categories = sorted(
        set(reference_values.unique()) | set(current_values.unique())
    )

    reference_counts = (
        reference_values.value_counts()
        .reindex(categories, fill_value=0)
        .to_numpy(dtype=float)
    )
    current_counts = (
        current_values.value_counts()
        .reindex(categories, fill_value=0)
        .to_numpy(dtype=float)
    )

    expected = reference_counts / reference_counts.sum()
    actual = current_counts / current_counts.sum()

    expected = np.clip(expected, epsilon, None)
    actual = np.clip(actual, epsilon, None)

    expected /= expected.sum()
    actual /= actual.sum()

    return float(
        np.sum((actual - expected) * np.log(actual / expected))
    )


# ---------------------------------------------------------
# 4. Run K-S and PSI checks
# ---------------------------------------------------------

numeric_features = [
    "basket_value",
    "delivery_distance_km",
    "request_latency_ms",
]

rows = []
p_values = []

for feature in numeric_features:
    ref = reference[feature].to_numpy(dtype=float)
    cur = live[feature].to_numpy(dtype=float)

    # Exclude missing and non-finite observations from these tests.
    # Their rates should be monitored as separate data-quality metrics.
    ref = ref[np.isfinite(ref)]
    cur = cur[np.isfinite(cur)]

    if len(ref) < 100 or len(cur) < 100:
        raise ValueError(
            f"Not enough valid observations to compare {feature}."
        )

    ks_result = ks_2samp(
        ref,
        cur,
        alternative="two-sided",
        method="auto",
    )

    psi_value = psi_numeric(ref, cur)

    p_values.append(float(ks_result.pvalue))
    rows.append({
        "feature": feature,
        "ks_D": float(ks_result.statistic),
        "ks_p": float(ks_result.pvalue),
        "PSI": psi_value,
    })

# Adjust for multiple K-S comparisons.
adjusted_p_values = false_discovery_control(
    p_values,
    method="bh",
)

for row, p_adjusted in zip(rows, adjusted_p_values):
    row["ks_p_adj"] = float(p_adjusted)

    row["ks_alert"] = bool(
        p_adjusted < 0.01 and row["ks_D"] >= 0.05
    )

    row["psi_status"] = (
        "ALERT" if row["PSI"] >= 0.20
        else "WARN" if row["PSI"] >= 0.10
        else "OK"
    )

# Categorical variables use PSI rather than the continuous K-S test.
region_psi = psi_categorical(
    reference["region"],
    live["region"],
)

rows.append({
    "feature": "region",
    "ks_D": np.nan,
    "ks_p": np.nan,
    "ks_p_adj": np.nan,
    "PSI": region_psi,
    "ks_alert": False,
    "psi_status": (
        "ALERT" if region_psi >= 0.20
        else "WARN" if region_psi >= 0.10
        else "OK"
    ),
})

# ---------------------------------------------------------
# 5. Print a monitoring report
# ---------------------------------------------------------

report = pd.DataFrame(
    rows,
    columns=[
        "feature",
        "ks_D",
        "ks_p",
        "ks_p_adj",
        "PSI",
        "ks_alert",
        "psi_status",
    ],
)

print(
    report.to_string(
        index=False,
        float_format=lambda value: f"{value:.4g}",
    )
)

What the results tell you

With the seeded synthetic data, the report should resemble the following:

             feature   ks_D      ks_p   ks_p_adj      PSI  ks_alert psi_status
        basket_value 0.2154 7.655e-40  1.148e-39   0.2546      True      ALERT
delivery_distance_km 0.0254    0.5509     0.5509 0.005422     False         OK
  request_latency_ms 0.3458 2.08e-103 6.239e-103   0.6633      True      ALERT
              region    NaN       NaN        NaN 0.07053     False         OK

The exact interpretation is more important than the printed numbers.

• Basket value: Both metrics indicate a substantial change. The K–S statistic is 0.2154, and PSI exceeds 0.20. This feature warrants investigation.

• Delivery distance: The K–S statistic is small, its p-value is not significant, and PSI is close to zero. The synthetic data provides no strong evidence of a distribution shift in this feature.

• Request latency: Both metrics detect a substantial change. Latency is an operational metric rather than necessarily a model input, so the next step is to determine whether the change affects the feature pipeline, the model, or the service itself.

• Region: PSI indicates some change in the regional mix, but the score remains below the initial warning threshold. A team might still investigate if region is especially important to model performance.

Notice that the K–S p-values are extremely small for the shifted continuous features. This is expected with the deliberately introduced changes and the sample sizes used in the example.

Also notice that the regional proportions have changed, but PSI remains below 0.10. That is a useful reminder that a changed distribution does not automatically imply a severe population shift.

The script is a working demonstration, not a complete production service. A real implementation also needs schema checks, missingness monitoring, logging, alert routing, baseline versioning, and a strategy for handling repeated windows.

7. Combine K–S and PSI into an actionable alert policy

Running two metrics is useful only if you have a clear rule for what to do with their results.

A practical initial policy could look like this:

This table is a starting point, not a universal decision matrix.

Disagreement between the metrics is not necessarily a defect. The K–S test and PSI summarize distributions differently. A difference in their outputs can help reveal the role of binning, sample size, and distribution shape.

Why the K–S effect-size threshold matters

In the example, the K–S alert requires both an adjusted p-value below 0.01 and D of at least 0.05.

The p-value requirement controls statistical evidence according to a chosen significance policy. The D threshold filters out some changes that are statistically detectable but small in magnitude.

Neither threshold is universally optimal. Choose the effect-size threshold based on the feature’s historical variation, business importance, and the size of the change you need to detect.

Why adjust for multiple testing?

Suppose you monitor 150 numerical features and run a K–S test for each one every hour.

Even when no meaningful drift is occurring, testing many hypotheses increases the likelihood of seeing some small p-values by chance.

The Benjamini–Hochberg procedure adjusts p-values to control the false discovery rate under its applicable assumptions. SciPy exposes this through its current false-discovery-control API.

This does not eliminate false alerts, but it is a more defensible approach than treating hundreds of unadjusted p-values as independent evidence of a problem.

For a monitoring system with many related features or complex dependencies between tests, consider the statistical assumptions behind the adjustment and evaluate the alert rate against historical data.

8. Turn the script into a production monitoring service

The transition from a notebook to production involves more than scheduling the code to run.

Choose a meaningful monitoring window

Compare sufficiently large, well-defined windows.

A five-minute window may contain too few observations for a reliable test on a low-volume service. A monthly window might detect the problem but delay the alert for weeks.

Select the window size according to traffic volume, the speed at which drift matters, and the statistical power required for the changes you care about.

If you monitor hourly, retain hourly results so you can identify whether a shift began suddenly or developed gradually.

Add persistence and severity rules

One noisy window should not necessarily page an on-call engineer.

For moderate drift, require repeated threshold breaches across consecutive windows before escalating. For a severe shift in a critical feature, an immediate alert may be appropriate.

Avoid persistence rules that are so strict they suppress a genuine incident. Calibrate them using replayed historical data and known incidents.

Monitor data quality separately

A distribution comparison is not a replacement for input validation.

Track at least:

• Missing-value rates.

• Infinite or invalid numerical values.

• Unexpected categories.

• Feature-schema changes.

• Value ranges and impossible values.

• Abrupt changes in ingestion volume.

• Delays and failures in feature computation.

These checks are important because an upstream pipeline bug can look like ordinary drift—or cause a model to receive corrupted inputs without producing an obvious distribution alert.

The monitoring code above removes missing and non-finite numerical values before statistical comparison. That is appropriate for this demonstration, but a production system must also alert on the removed values. Otherwise, a sudden increase in missingness could be hidden by the comparison itself.

Segment the results

An aggregate distribution can conceal an important local change.

Imagine that overall delivery distance remains stable because short-distance orders dominate the population, while long-distance orders experience a significant shift.

Compare important segments separately when there is enough data to support reliable monitoring. Record segment sample sizes alongside every metric so that a high score based on a tiny sample is not interpreted in the same way as a high score based on thousands of observations.

Connect drift to model performance

Whenever labels become available, compare drift alerts with performance metrics such as precision, recall, calibration, ranking quality, or prediction error, depending on the model’s purpose.

This is how the monitoring system learns which types of drift are actually consequential.

A feature may drift regularly without affecting model quality. Another feature may change only slightly but have a major impact because the model relies heavily on it.

For delayed labels, record predictions and their corresponding model versions so that later outcomes can be joined to the correct predictions. This makes it possible to investigate whether a drift alert preceded a measurable decline in performance.

Current MLOps monitoring frameworks also use reference and analysis periods, chunk-level metrics, and configurable alert thresholds. For example, NannyML’s documentation describes calculating metrics for analysis chunks and comparing those results against fitted or configured thresholds (NannyML univariate drift tutorial).

Never retrain blindly on every drift alert

Automatic retraining is tempting, but it can turn a monitoring signal into a feedback-loop problem.

A drift alert might be caused by:

• A legitimate change in customer behavior.

• A temporary marketing campaign.

• A feature-pipeline defect.

• A broken upstream data source.

• An unexpected but valid operating condition.

• A shift that does not affect model performance.

Retraining on corrupted data or an unrepresentative short-term spike can make the model worse.

A safer workflow is to validate the input data, identify the source of the shift, check available performance evidence, and then decide whether to retrain, recalibrate, update features, or roll back an upstream change.

9. The cherry on the cake: when a model quietly decays

Consider an illustrative production incident involving an e-commerce ranking model.

The model has been deployed for several months. Its service dashboard is green, requests complete successfully, and the feature schema has not changed. However, a new advertising campaign attracts a different mix of customers, while a checkout migration also changes request latency.

Over several weeks, ranking quality deteriorates. The ground-truth labels needed to calculate definitive performance metrics arrive with a delay, so the business team does not immediately see the full extent of the decline.

A scheduled K–S comparison finally raises an alert on basket value. The statistic is large enough to indicate a meaningful distribution difference, and PSI independently crosses the substantial-drift threshold.

The alert does not prove that the ranking model is failing. It tells the team where to start looking.

The investigation reveals that the traffic mix has changed and that request latency has shifted at the same time. The team separates the affected traffic source, checks feature computation, compares prediction behavior, and later evaluates ranking quality when the corresponding labels arrive. Only after identifying the cause can it decide whether the right intervention is a feature correction, a model update, or a change to the deployment configuration.

This is an illustrative scenario, not a claim about a named company. Its lesson is that the drift alarm is an early-warning mechanism, not an automatic diagnosis.

10. Common mistakes to avoid

Before deploying a drift detector, check for these failure modes.

• Using K–S on every feature: The ordinary two-sample K–S test is designed for continuous distributions. Select appropriate methods for categorical and discrete features.

• Treating p-values as severity scores: A tiny p-value is not a business-impact measurement. Consider D, PSI, feature criticality, and performance evidence.

• Recomputing PSI bins on every live window: This makes comparisons inconsistent. Fit numerical bin boundaries on the reference population and preserve them.

• Ignoring zero proportions: PSI requires careful handling of empty bins and unseen categories. Use small-value smoothing and monitor category novelty explicitly.

• Comparing incompatible populations: Schema changes, preprocessing changes, seasonal patterns, and model-version differences can invalidate an otherwise correct comparison.

• Ignoring missingness: Dropping invalid values without monitoring their rate can hide a serious pipeline defect.

• Alerting on every isolated fluctuation: Use historical calibration, minimum sample sizes, and appropriate persistence rules.

• Treating drift as proof of degradation: Investigate the change and connect it to observed performance whenever possible.

• Automatically retraining without diagnosis: Establish data-quality and governance checks before allowing a drift alert to trigger a new model release.

11. A practical implementation checklist

Use this checklist when taking the example into a real monitoring pipeline.

• Select and document a stable reference dataset for each model version.
• Define current monitoring windows and minimum sample sizes.
• Apply the K–S test only to appropriate continuous features.
• Calculate PSI with stable reference-based bins or consistent categorical definitions.
• Configure initial warning and alert thresholds, then calibrate them against historical windows.
• Adjust for multiple K–S comparisons.
• Track missingness, invalid values, unseen categories, and schema changes separately.
• Store metrics, sample sizes, timestamps, model versions, and reference versions.
• Add alert persistence rules and feature-specific severity policies.
• Investigate drift alongside prediction behavior and delayed ground-truth performance metrics.
• Document the actions that each alert level should trigger.
• Review alert quality regularly and update thresholds only through a controlled process.

Conclusion: make drift detection part of the model lifecycle

The K–S test and PSI solve complementary problems. K–S measures the maximum difference between empirical cumulative distributions and provides a hypothesis-test p-value. PSI summarizes population changes across stable bins or categories in a score that is practical for monitoring dashboards.

Used together, they provide a useful first line of defense against silent changes in production data. Used without appropriate baselines, thresholds, data-quality checks, and performance monitoring, they can create noise or miss important problems.

Your next step: choose one production model and three high-impact input features. Establish a reference population, implement the K–S and PSI checks, replay historical data to calibrate thresholds, and connect the resulting alerts to an investigation workflow. Then measure whether those alerts provide useful warning before model performance degrades.

A model is not production-ready simply because it works at deployment time. It needs a monitoring process that helps you recognize when the conditions it depends on have changed.