Reproducible XGBoost: seeds, pinned versions, and a validation protocol you can defend

A reproducible training run and a trustworthy validation score are related, but they are not the same guarantee.

A run is repeatable when the same inputs, code, configuration, and execution environment produce the same artifact or predictions. A score is defensible when the evaluation procedure prevents leakage, repeated peeking, split shopping, and post-hoc metric selection.

You need both. Bit-identical predictions do not rescue a contaminated holdout, and a statistically sound validation design does not help when the production model cannot be rebuilt.

This guide builds a narrow, practical contract for current XGBoost 3.x:

• Pin the software environment.

• Freeze every source of randomness you control.

• Treat thread count, tree method, and device as model parameters.

• Diff predictions instead of trusting a seed.

• Save the model beside a complete feature schema.

• Commit the split and metric protocol before comparing models.

• Touch the final holdout once.

• Run identical transformation code during training and serving.

The examples use XGBoost 3.3.0, released in June 2026, with current NumPy, pandas, and scikit-learn versions.

The three contracts behind a reproducible model

A strong training system has three explicit contracts.

The execution contract

This controls how the algorithm runs:

• Python version

• XGBoost version

• NumPy, pandas, and scikit-learn versions

• Operating system and CPU architecture

• CPU versus CUDA execution

• Number of worker threads

• Tree construction method

• Random seeds

• Input row order

• Input column order

A seed controls pseudo-random choices. It does not define the entire numerical execution environment.

The data contract

This controls what the model believes each input column means:

• Feature names

• Feature order

• Numeric dtypes

• Categorical columns

• Allowed category levels

• Missing-value behavior

• Transformation logic

A model that receives the right values under the wrong column order is not reproducible. It is simply wrong in a consistent-looking way.

The evaluation contract

This controls what claims you may make about the score:

• How the development and holdout sets were created

• Which row identifiers belong to each split

• Which metric is primary

• Whether higher or lower is better

• Which candidate configurations may be compared

• When the holdout may be opened

• What happens after the holdout is evaluated

Write these contracts down before the experiment. Do not reconstruct them later from notebook history.

Pin the environment before debugging randomness

XGBoost 3.3.0 is the current stable release at the time of writing. Its standard Python package is available from PyPI, while scikit-learn 1.9.0, NumPy 2.5.1, and pandas 3.0.5 are also current releases.

Create a clean virtual environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Create requirements.txt:

numpy==2.5.1
pandas==3.0.5
scikit-learn==1.9.0
xgboost==3.3.0

Install and verify the environment:

python -m pip install -r requirements.txt
python -m pip check
python -VV
python -m pip freeze --all

The four direct pins make the example readable, but a production build should also preserve the complete resolved environment. After installing in a clean environment, capture it:

mkdir -p artifacts
python -m pip freeze --all > artifacts/requirements.lock.txt
python -VV > artifacts/python-version.txt

A lock file is necessary but not sufficient. Two environments with the same Python packages can still differ in:

• Operating-system libraries

• OpenMP runtime

• CPU instruction support

• Compiler flags used to build a native dependency

• CUDA, driver, and GPU architecture

• Environment variables that alter thread behavior

For high-assurance rebuilding, run training in an immutable container image and record the image digest. A mutable tag such as latest is not a reproducibility boundary.

Why random_state is not the whole story

For tree models, XGBoost exposes parameters that affect randomness and execution topology. Its documentation defines seed, while the scikit-learn wrapper exposes the familiar random_state parameter. XGBoost also uses parallel threads by default when no thread count is supplied.

Three parameters deserve to be treated as part of the model definition:

• random_state

• n_jobs

• tree_method

Pin the device as well:

• device="cpu" for CPU training

• device="cuda" for GPU training

Do not leave tree_method="auto" in a strict reproducibility contract. In XGBoost 3.3.0, auto currently selects hist, but spelling out hist prevents a future change in automatic selection from silently altering the training path.

A reasonable CPU contract looks like this:

MODEL_CONTRACT = {
    "random_state": 20260803,
    "n_jobs": 1,
    "tree_method": "hist",
    "device": "cpu",
}

Using one thread is the most conservative choice when exact repeatability matters more than training speed. It is not a universal requirement: modern XGBoost CPU histogram training is often bit-identical across thread counts. The important practice is to test the exact environment you approve rather than assuming that either determinism or nondeterminism is universal.

Build a reproducibility probe, not a belief

The correct question is not, “Did I set a seed?”

It is, “When I refit the approved configuration, are the predictions exactly equal?”

The following project performs that test and then builds the rest of the validation protocol around the same model contract.

repro-xgb/
├── artifacts/
├── protocol.json
├── reproducible_xgb.py
└── requirements.txt

Create protocol.json before running an experiment:

{
  "protocol_version": 1,
  "dataset_seed": 20260801,
  "split_seed": 20260802,
  "model_contract": {
    "random_state": 20260803,
    "n_jobs": 1,
    "tree_method": "hist",
    "device": "cpu",
    "training_metric": "logloss"
  },
  "primary_metric": {
    "name": "roc_auc",
    "direction": "maximize"
  },
  "split_policy": {
    "fit_fraction": 0.6,
    "validation_fraction": 0.2,
    "holdout_fraction": 0.2
  },
  "candidates": [
    {
      "name": "depth_5",
      "params": {
        "n_estimators": 220,
        "learning_rate": 0.05,
        "max_depth": 5,
        "min_child_weight": 4,
        "subsample": 0.85,
        "colsample_bytree": 0.8,
        "reg_lambda": 2.0
      }
    },
    {
      "name": "depth_7",
      "params": {
        "n_estimators": 220,
        "learning_rate": 0.05,
        "max_depth": 7,
        "min_child_weight": 6,
        "subsample": 0.85,
        "colsample_bytree": 0.8,
        "reg_lambda": 3.0
      }
    }
  ]
}

Several decisions are now frozen before scores exist:

• ROC AUC is the primary metric.

• Higher is better.

• The split seed is fixed.

• The model seed is fixed.

• The thread count is fixed.

• The tree method is fixed.

• The candidate set is finite and declared.

• The holdout fraction is fixed.

ROC AUC consumes prediction scores rather than hard class labels and measures ranking performance across thresholds.

A complete runnable implementation

Create reproducible_xgb.py:

from __future__ import annotations

import argparse
import hashlib
import importlib.metadata
import json
import os
import platform
import shutil
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split


ROOT = Path(__file__).resolve().parent
ARTIFACTS = ROOT / "artifacts"
PROTOCOL_PATH = ROOT / "protocol.json"

MODEL_PATH = ARTIFACTS / "model.json"
SCHEMA_PATH = ARTIFACTS / "feature_schema.json"
SPLITS_PATH = ARTIFACTS / "splits.json"
HOLDOUT_MARKER_PATH = ARTIFACTS / "HOLDOUT_EVALUATED.json"


def read_json(path: Path) -> dict[str, Any]:
    with path.open("r", encoding="utf-8") as handle:
        return json.load(handle)


def write_json(path: Path, payload: Any) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    temporary_path = path.with_suffix(path.suffix + ".tmp")

    with temporary_path.open("w", encoding="utf-8") as handle:
        json.dump(payload, handle, indent=2, sort_keys=True)
        handle.write("\n")

    temporary_path.replace(path)


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()

    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)

    return digest.hexdigest()


def load_protocol() -> dict[str, Any]:
    if not PROTOCOL_PATH.exists():
        raise FileNotFoundError(
            f"Missing {PROTOCOL_PATH}. Commit the protocol before training."
        )

    protocol = read_json(PROTOCOL_PATH)

    if protocol["primary_metric"]["name"] != "roc_auc":
        raise ValueError("This example implements roc_auc as its primary metric.")

    if protocol["primary_metric"]["direction"] != "maximize":
        raise ValueError("ROC AUC must use direction='maximize'.")

    return protocol


def feature_schema() -> dict[str, Any]:
    return {
        "schema_version": 1,
        "features": [
            {
                "name": "order_value",
                "dtype": "float32",
                "categories": None,
            },
            {
                "name": "item_count",
                "dtype": "int16",
                "categories": None,
            },
            {
                "name": "distance_km",
                "dtype": "float32",
                "categories": None,
            },
            {
                "name": "warehouse_load",
                "dtype": "float32",
                "categories": None,
            },
            {
                "name": "promised_days",
                "dtype": "int8",
                "categories": None,
            },
            {
                "name": "shipping_mode",
                "dtype": "category",
                "categories": ["standard", "express", "pickup"],
                "ordered": False,
            },
            {
                "name": "region",
                "dtype": "category",
                "categories": ["north", "south", "east", "west"],
                "ordered": False,
            },
        ],
    }


def make_orders(
    seed: int,
    n_rows: int = 12_000,
) -> tuple[pd.DataFrame, pd.Series]:
    rng = np.random.default_rng(seed)

    order_value = rng.lognormal(
        mean=4.0,
        sigma=0.65,
        size=n_rows,
    ).clip(5.0, 900.0)

    item_count = rng.integers(
        low=1,
        high=12,
        size=n_rows,
    )

    distance_km = rng.gamma(
        shape=2.2,
        scale=12.0,
        size=n_rows,
    ).clip(0.5, 180.0)

    warehouse_load = rng.beta(
        a=2.5,
        b=2.0,
        size=n_rows,
    )

    promised_days = rng.integers(
        low=1,
        high=8,
        size=n_rows,
    )

    shipping_mode = rng.choice(
        ["standard", "express", "pickup"],
        size=n_rows,
        p=[0.60, 0.28, 0.12],
    )

    region = rng.choice(
        ["north", "south", "east", "west"],
        size=n_rows,
        p=[0.26, 0.24, 0.27, 0.23],
    )

    raw = pd.DataFrame(
        {
            "order_value": order_value,
            "item_count": item_count,
            "distance_km": distance_km,
            "warehouse_load": warehouse_load,
            "promised_days": promised_days,
            "shipping_mode": shipping_mode,
            "region": region,
        },
        index=pd.Index(
            [f"ORD-{number:06d}" for number in range(n_rows)],
            name="order_id",
        ),
    )

    region_effect = pd.Series(region).map(
        {
            "north": 0.12,
            "south": 0.28,
            "east": -0.08,
            "west": 0.05,
        }
    ).to_numpy()

    shipping_effect = pd.Series(shipping_mode).map(
        {
            "standard": 0.42,
            "express": -0.38,
            "pickup": -0.75,
        }
    ).to_numpy()

    log_odds = (
        -3.0
        + 0.014 * order_value
        + 0.025 * distance_km
        + 1.25 * warehouse_load
        - 0.20 * promised_days
        + 0.055 * item_count
        + shipping_effect
        + region_effect
        + rng.normal(0.0, 0.6, size=n_rows)
    )

    probability = 1.0 / (1.0 + np.exp(-log_odds))
    target = rng.binomial(1, probability).astype(np.int8)

    y = pd.Series(
        target,
        index=raw.index,
        name="late_delivery",
        dtype="int8",
    )

    return raw, y


def prepare_features(
    raw: pd.DataFrame,
    schema: dict[str, Any],
) -> pd.DataFrame:
    required = [spec["name"] for spec in schema["features"]]
    missing = [name for name in required if name not in raw.columns]

    if missing:
        raise ValueError(f"Missing required features: {missing}")

    transformed = pd.DataFrame(index=raw.index)

    for spec in schema["features"]:
        name = spec["name"]
        categories = spec.get("categories")

        if categories is None:
            transformed[name] = pd.to_numeric(
                raw[name],
                errors="raise",
            ).astype(spec["dtype"])
            continue

        observed = set(
            raw[name]
            .dropna()
            .astype("string")
            .unique()
            .tolist()
        )
        allowed = set(categories)
        unknown = sorted(observed - allowed)

        if unknown:
            raise ValueError(
                f"Feature {name!r} contains unknown categories: {unknown}"
            )

        category_dtype = pd.CategoricalDtype(
            categories=categories,
            ordered=bool(spec.get("ordered", False)),
        )

        transformed[name] = raw[name].astype(category_dtype)

    if transformed.columns.tolist() != required:
        raise AssertionError("Feature order does not match the schema.")

    return transformed


def validate_splits(
    splits: dict[str, list[str]],
    all_ids: pd.Index,
) -> None:
    fit_ids = splits["fit"]
    validation_ids = splits["validation"]
    holdout_ids = splits["holdout"]

    combined = fit_ids + validation_ids + holdout_ids

    if len(combined) != len(set(combined)):
        raise ValueError("Split identifiers overlap.")

    if set(combined) != set(all_ids.astype(str)):
        raise ValueError("Splits do not cover the dataset exactly.")


def build_or_load_splits(
    all_ids: pd.Index,
    y: pd.Series,
    split_seed: int,
) -> dict[str, list[str]]:
    if SPLITS_PATH.exists():
        splits = read_json(SPLITS_PATH)
        validate_splits(splits, all_ids)
        return splits

    development_ids, holdout_ids = train_test_split(
        all_ids.to_numpy(),
        test_size=0.20,
        random_state=split_seed,
        shuffle=True,
        stratify=y.loc[all_ids].to_numpy(),
    )

    fit_ids, validation_ids = train_test_split(
        development_ids,
        test_size=0.25,
        random_state=split_seed + 1,
        shuffle=True,
        stratify=y.loc[development_ids].to_numpy(),
    )

    splits = {
        "fit": fit_ids.tolist(),
        "validation": validation_ids.tolist(),
        "holdout": holdout_ids.tolist(),
    }

    validate_splits(splits, all_ids)
    write_json(SPLITS_PATH, splits)
    return splits


def make_model(
    protocol: dict[str, Any],
    candidate: dict[str, Any],
    *,
    n_jobs: int | None = None,
    tree_method: str | None = None,
) -> xgb.XGBClassifier:
    contract = protocol["model_contract"]
    params = dict(candidate["params"])

    return xgb.XGBClassifier(
        objective="binary:logistic",
        eval_metric=contract["training_metric"],
        random_state=int(contract["random_state"]),
        n_jobs=int(
            contract["n_jobs"]
            if n_jobs is None
            else n_jobs
        ),
        tree_method=str(
            contract["tree_method"]
            if tree_method is None
            else tree_method
        ),
        device=str(contract["device"]),
        enable_categorical=True,
        validate_parameters=True,
        **params,
    )


def prediction_diff(
    left: np.ndarray,
    right: np.ndarray,
) -> dict[str, Any]:
    absolute = np.abs(left - right)

    return {
        "array_equal": bool(np.array_equal(left, right)),
        "different_values": int(np.count_nonzero(left != right)),
        "maximum_absolute_difference": float(absolute.max(initial=0.0)),
        "mean_absolute_difference": float(absolute.mean()),
    }


def environment_record() -> dict[str, Any]:
    package_names = [
        "numpy",
        "pandas",
        "scikit-learn",
        "xgboost",
    ]

    return {
        "python": platform.python_version(),
        "python_implementation": platform.python_implementation(),
        "platform": platform.platform(),
        "machine": platform.machine(),
        "processor": platform.processor(),
        "logical_cpu_count": os.cpu_count(),
        "packages": {
            name: importlib.metadata.version(name)
            for name in package_names
        },
    }


def command_probe() -> None:
    ARTIFACTS.mkdir(parents=True, exist_ok=True)

    protocol = load_protocol()
    schema = feature_schema()
    raw, y = make_orders(int(protocol["dataset_seed"]))

    splits = build_or_load_splits(
        raw.index,
        y,
        int(protocol["split_seed"]),
    )

    fit_ids = splits["fit"]
    validation_ids = splits["validation"]
    candidate = protocol["candidates"][0]

    X_fit = prepare_features(raw.loc[fit_ids], schema)
    X_validation = prepare_features(
        raw.loc[validation_ids],
        schema,
    )
    y_fit = y.loc[fit_ids]

    def fit_predict(n_jobs: int) -> np.ndarray:
        model = make_model(
            protocol,
            candidate,
            n_jobs=n_jobs,
            tree_method="hist",
        )
        model.fit(X_fit, y_fit, verbose=False)
        return model.predict_proba(X_validation)[:, 1]

    alternate_threads = max(
        2,
        min(8, os.cpu_count() or 2),
    )

    one_thread_predictions = fit_predict(1)
    alternate_predictions = fit_predict(alternate_threads)

    cross_thread_result = prediction_diff(
        one_thread_predictions,
        alternate_predictions,
    )

    pinned_threads = int(
        protocol["model_contract"]["n_jobs"]
    )

    pinned_first = fit_predict(pinned_threads)
    pinned_second = fit_predict(pinned_threads)

    pinned_result = prediction_diff(
        pinned_first,
        pinned_second,
    )

    np.testing.assert_array_equal(
        pinned_first,
        pinned_second,
        err_msg=(
            "The pinned training contract did not produce "
            "bit-identical predictions."
        ),
    )

    result = {
        "comparison": {
            "left_n_jobs": 1,
            "right_n_jobs": alternate_threads,
            "tree_method": "hist",
            "result": cross_thread_result,
        },
        "pinned_repeat": {
            "n_jobs": pinned_threads,
            "tree_method": "hist",
            "result": pinned_result,
        },
        "interpretation": (
            "A non-zero cross-thread diff exposes thread-count "
            "sensitivity in this environment. A zero diff means "
            "this environment passed the cross-thread test; it "
            "does not remove n_jobs from the model contract."
        ),
    }

    write_json(ARTIFACTS / "repro_probe.json", result)
    print(json.dumps(result, indent=2, sort_keys=True))


def command_train() -> None:
    ARTIFACTS.mkdir(parents=True, exist_ok=True)

    protocol = load_protocol()
    schema = feature_schema()
    raw, y = make_orders(int(protocol["dataset_seed"]))

    splits = build_or_load_splits(
        raw.index,
        y,
        int(protocol["split_seed"]),
    )

    fit_ids = splits["fit"]
    validation_ids = splits["validation"]
    development_ids = fit_ids + validation_ids

    X_fit = prepare_features(raw.loc[fit_ids], schema)
    X_validation = prepare_features(
        raw.loc[validation_ids],
        schema,
    )
    y_fit = y.loc[fit_ids]
    y_validation = y.loc[validation_ids]

    validation_results: list[dict[str, Any]] = []

    for index, candidate in enumerate(protocol["candidates"]):
        model = make_model(protocol, candidate)
        model.fit(X_fit, y_fit, verbose=False)

        probability = model.predict_proba(
            X_validation
        )[:, 1]

        score = roc_auc_score(
            y_validation,
            probability,
        )

        validation_results.append(
            {
                "candidate_index": index,
                "candidate_name": candidate["name"],
                "primary_metric": "roc_auc",
                "validation_score": float(score),
            }
        )

    selected = max(
        validation_results,
        key=lambda result: (
            result["validation_score"],
            -result["candidate_index"],
        ),
    )

    selected_candidate = protocol["candidates"][
        selected["candidate_index"]
    ]

    X_development = prepare_features(
        raw.loc[development_ids],
        schema,
    )
    y_development = y.loc[development_ids]

    final_model = make_model(
        protocol,
        selected_candidate,
    )
    final_model.fit(
        X_development,
        y_development,
        verbose=False,
    )

    final_model.save_model(MODEL_PATH)
    write_json(SCHEMA_PATH, schema)
    write_json(
        ARTIFACTS / "validation_results.json",
        validation_results,
    )
    write_json(
        ARTIFACTS / "selected_model.json",
        {
            "candidate": selected_candidate,
            "selection_result": selected,
            "model_contract": protocol["model_contract"],
        },
    )
    write_json(
        ARTIFACTS / "environment.json",
        environment_record(),
    )

    shutil.copyfile(
        PROTOCOL_PATH,
        ARTIFACTS / "protocol.json",
    )

    manifest_files = [
        MODEL_PATH,
        SCHEMA_PATH,
        SPLITS_PATH,
        ARTIFACTS / "protocol.json",
        ARTIFACTS / "selected_model.json",
        ARTIFACTS / "validation_results.json",
        ARTIFACTS / "environment.json",
    ]

    manifest = {
        "created_at_utc": datetime.now(
            timezone.utc
        ).isoformat(),
        "files": {
            path.name: {
                "sha256": sha256_file(path),
                "bytes": path.stat().st_size,
            }
            for path in manifest_files
        },
    }

    write_json(
        ARTIFACTS / "manifest.json",
        manifest,
    )

    print(
        json.dumps(
            {
                "selected": selected,
                "model_path": str(MODEL_PATH),
                "schema_path": str(SCHEMA_PATH),
            },
            indent=2,
            sort_keys=True,
        )
    )


def load_trained_model() -> xgb.XGBClassifier:
    if not MODEL_PATH.exists():
        raise FileNotFoundError(
            "Run the train command before loading the model."
        )

    model = xgb.XGBClassifier()
    model.load_model(MODEL_PATH)
    return model


def command_parity() -> None:
    protocol = load_protocol()
    raw, _ = make_orders(int(protocol["dataset_seed"]))

    if not SCHEMA_PATH.exists():
        raise FileNotFoundError(
            "Run the train command before the parity check."
        )

    schema = read_json(SCHEMA_PATH)
    model = load_trained_model()

    training_sample_raw = raw.iloc[:32].copy()
    training_features = prepare_features(
        training_sample_raw,
        schema,
    )

    service_payload = (
        training_sample_raw
        .reset_index()
        .to_dict(orient="records")
    )

    service_raw = (
        pd.DataFrame.from_records(service_payload)
        .set_index("order_id")
    )

    serving_features = prepare_features(
        service_raw,
        schema,
    )

    pd.testing.assert_frame_equal(
        training_features,
        serving_features,
        check_dtype=True,
        check_categorical=True,
    )

    training_predictions = model.predict_proba(
        training_features
    )[:, 1]

    serving_predictions = model.predict_proba(
        serving_features
    )[:, 1]

    np.testing.assert_array_equal(
        training_predictions,
        serving_predictions,
        err_msg=(
            "Training and serving paths produced "
            "different predictions."
        ),
    )

    result = {
        "rows_checked": len(training_features),
        "feature_frames_equal": True,
        "predictions_bit_identical": True,
    }

    write_json(
        ARTIFACTS / "train_serve_parity.json",
        result,
    )
    print(json.dumps(result, indent=2, sort_keys=True))


def command_evaluate_holdout() -> None:
    if HOLDOUT_MARKER_PATH.exists():
        previous = read_json(HOLDOUT_MARKER_PATH)
        raise RuntimeError(
            "The holdout has already been evaluated. "
            f"Recorded result: {previous}"
        )

    protocol = load_protocol()
    raw, y = make_orders(int(protocol["dataset_seed"]))

    if not SPLITS_PATH.exists():
        raise FileNotFoundError(
            "Run the train command before evaluating holdout."
        )

    schema = read_json(SCHEMA_PATH)
    splits = read_json(SPLITS_PATH)
    validate_splits(splits, raw.index)

    holdout_ids = splits["holdout"]
    X_holdout = prepare_features(
        raw.loc[holdout_ids],
        schema,
    )
    y_holdout = y.loc[holdout_ids]

    model = load_trained_model()
    probability = model.predict_proba(X_holdout)[:, 1]

    score = roc_auc_score(
        y_holdout,
        probability,
    )

    result = {
        "evaluated_at_utc": datetime.now(
            timezone.utc
        ).isoformat(),
        "primary_metric": "roc_auc",
        "holdout_score": float(score),
        "holdout_rows": len(holdout_ids),
        "model_sha256": sha256_file(MODEL_PATH),
        "splits_sha256": sha256_file(SPLITS_PATH),
        "protocol_sha256": sha256_file(
            ARTIFACTS / "protocol.json"
        ),
    }

    write_json(HOLDOUT_MARKER_PATH, result)
    print(json.dumps(result, indent=2, sort_keys=True))


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "command",
        choices=[
            "probe",
            "train",
            "parity",
            "evaluate-holdout",
        ],
    )
    args = parser.parse_args()

    commands = {
        "probe": command_probe,
        "train": command_train,
        "parity": command_parity,
        "evaluate-holdout": command_evaluate_holdout,
    }

    commands[args.command]()


if __name__ == "__main__":
    main()

Run the reproducibility probe first:

python reproducible_xgb.py probe

Its output contains two comparisons.

The first deliberately changes only n_jobs:

{
  "comparison": {
    "left_n_jobs": 1,
    "right_n_jobs": 8,
    "tree_method": "hist",
    "result": {
      "array_equal": true,
      "different_values": 0,
      "maximum_absolute_difference": 0.0,
      "mean_absolute_difference": 0.0
    }
  }
}

On a current CPU histogram build, zero may be the expected result. On an environment with thread-count-sensitive reductions, a non-zero result exposes the difference immediately.

This distinction matters: the probe does not manufacture nondeterminism to make a dramatic demonstration. It tests whether your real wheel, OpenMP runtime, machine, and configuration exhibit it.

The second comparison refits the fully pinned contract twice:

{
  "pinned_repeat": {
    "n_jobs": 1,
    "tree_method": "hist",
    "result": {
      "array_equal": true,
      "different_values": 0,
      "maximum_absolute_difference": 0.0,
      "mean_absolute_difference": 0.0
    }
  }
}

Unlike np.allclose, np.testing.assert_array_equal requires exact equality. One changed floating-point value fails the probe.

The cherry on the cake: a real thread-count failure

XGBoost has a memorable historical reproducibility report in which setting a seed did not produce identical results. The explanation given in the project issue was that different threads executed in a different order; limiting execution to one thread disabled the behavior.

That report is historical, not evidence that XGBoost 3.3.0 CPU histogram training has the same bug.

Its value is the engineering lesson:

A parameter that looks operational can become part of the statistical result.

Teams frequently treat thread count as a deployment concern:

• A laptop uses four threads.

• CI uses two.

• A training server uses all available cores.

• A container platform changes its CPU quota.

• A retraining job moves to a different instance class.

If thread topology affects accumulation order, split selection, sampling, or another numerical operation, those “equivalent” jobs may not be equivalent.

The defensive response is not to claim that multithreading is always unsafe. It is to:

1. Pin n_jobs.

2. Test repeated fits.

3. Test the approved deployment image.

4. Store the result of that probe.

5. Fail promotion when the equality contract is broken.

Save more than model.json

Train and save the artifact:

python reproducible_xgb.py train

XGBoost supports JSON and UBJSON model serialization. JSON or UBJSON preserves auxiliary information such as feature names, and XGBoost 3.1 and later can store category encodings used by its Python dataframe integration.

That does not mean one model file is a complete production contract.

XGBoost’s documentation explicitly notes that parameters which are not part of the fitted model, including items such as metrics and some training parameters, are not all saved as part of the model artifact.

The example therefore writes:

artifacts/
├── environment.json
├── feature_schema.json
├── manifest.json
├── model.json
├── protocol.json
├── requirements.lock.txt
├── selected_model.json
├── splits.json
├── validation_results.json
└── python-version.txt

Each file answers a different question.

model.json

What trees and objective were fitted?

feature_schema.json

What input names, order, dtypes, and category levels are legal?

An abbreviated schema looks like this:

{
  "schema_version": 1,
  "features": [
    {
      "name": "order_value",
      "dtype": "float32",
      "categories": null
    },
    {
      "name": "shipping_mode",
      "dtype": "category",
      "categories": [
        "standard",
        "express",
        "pickup"
      ],
      "ordered": false
    }
  ]
}

selected_model.json

Which declared candidate won validation, and what execution contract was used?

splits.json

Which stable row identifiers belong to fit, validation, and holdout?

protocol.json

Which decisions were made before results were observed?

environment.json

Which interpreter, platform, CPU count, and package versions created the artifact?

manifest.json

Which exact bytes belong to this release?

A manifest turns accidental replacement into a detectable event. If model.json changes without a corresponding manifest update, the artifact is no longer the reviewed artifact.

Use stable identifiers, not row positions

The split file stores order IDs such as ORD-004821, not integer positions such as row 4821.

This matters because positional splits silently change when:

• A data extract is sorted differently.

• An upstream job inserts records.

• Duplicate handling changes.

• Partition loading order changes.

• A dataframe index is reset.

• Two files are concatenated in a different order.

A fixed random seed does not rescue a split if the input order has changed.

The example verifies three properties whenever it loads splits.json:

• No identifier appears in more than one split.

• Every dataset identifier appears exactly once.

• No unknown identifier is present.

For a real dataset, also record a snapshot identifier, table version, object-store generation, or content hash. Stable split IDs cannot reproduce records that have been edited in place.

Freeze the holdout and make peeking visible

Scikit-learn’s evaluation guidance recommends keeping a test set separate from model fitting and model selection. Using the same data to tune parameters and report performance creates an optimistically biased estimate; nested validation is the standard alternative when selection and evaluation must both happen through resampling.

The example uses three partitions:

• Fit set: train each candidate.

• Validation set: choose among the candidates declared in protocol.json.

• Holdout set: evaluate the selected procedure once.

After candidate selection, the selected configuration is refitted on the combined fit and validation data. Only then is the holdout opened.

Run the parity test before touching the holdout:

python reproducible_xgb.py parity

Then perform the final evaluation:

python reproducible_xgb.py evaluate-holdout

The command creates HOLDOUT_EVALUATED.json. A second invocation fails.

This file is a guardrail, not an access-control system. Anyone can delete it. In a real workflow, strengthen the boundary:

• Keep holdout labels in a restricted storage location.

• Grant access only to a release job.

• Permit one evaluation per model version.

• Write results to an append-only store.

• Require a new holdout or nested evaluation after substantial post-holdout tuning.

• Review all experiments conducted after the score was revealed.

“Touched once” means the score influences one final decision. Copying the holdout into several notebooks still counts as repeated use even when each notebook runs only once.

Choose the metric before seeing candidate results

Metric shopping is a quieter form of holdout leakage.

Suppose two models produce these validation results:

Model A:
ROC AUC = 0.842
Log loss = 0.391
Accuracy = 0.814

Model B:
ROC AUC = 0.839
Log loss = 0.378
Accuracy = 0.821

There is no universally correct winner. The answer depends on the decision you made before the comparison:

• Choose Model A when ranking quality measured by ROC AUC is primary.

• Choose Model B when probability quality measured by log loss is primary.

• Accuracy is meaningful only after the classification threshold and error costs are justified.

The invalid move is to inspect all three metrics and announce whichever one favors the model you already prefer.

A defensible protocol states:

• The primary metric.

• Its direction.

• The unit of analysis.

• Any sample weights.

• Any grouping or temporal constraints.

• Secondary diagnostic metrics.

• The practical improvement required to justify extra complexity.

• The tie-breaking rule.

In the example, the candidate with the highest validation ROC AUC wins. Candidate order breaks an exact tie, so even the tie-breaking behavior is deterministic.

Fixed splits versus cross-validation

A single fixed validation split is easy to audit and reproduce. It can also be noisy.

For larger tabular datasets with stable class proportions, a fixed split may be adequate. For smaller datasets, cross-validation gives a more complete view of variation across partitions.

For binary classification, a reproducible stratified splitter can be declared explicitly:

from sklearn.model_selection import StratifiedKFold

cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=20260804,
)

With shuffle=True, scikit-learn documents that an integer random_state controls fold ordering and produces repeatable splits across calls. Stratification preserves class proportions as closely as possible, although the documentation correctly describes this primarily as an engineering solution rather than a cure for every statistical problem.

Do not regenerate folds independently for every candidate. Materialize them once or use the same splitter object and row ordering.

For grouped, temporal, geographic, or customer-level data, ordinary stratification may be invalid. Examples include:

• Multiple orders from the same customer

• Repeated telemetry from the same server

• Transactions from the same shop

• Forecasting future periods

• Events connected by a session or household

In those cases, the independence boundary is the group or time period, not the row. A perfectly reproducible leaking split is still a bad split.

Train/serve parity is part of reproducibility

The parity command sends the same sample through two ingestion shapes:

• The dataframe used by training code

• A list of JSON-like records representing a service request

Both paths call the same prepare_features function. The test then asserts:

• Equal feature names

• Equal feature order

• Equal numeric dtypes

• Equal categorical dtypes

• Equal category levels

• Bit-identical predictions

python reproducible_xgb.py parity

The important design choice is not the assertion itself. It is that there is only one transformation implementation.

A fragile system often has:

• One pandas notebook for training

• One SQL expression for batch inference

• One Python web-service function for online inference

• One feature definition in a stream processor

Even when all four were originally equivalent, they evolve independently.

A safer architecture packages feature preparation as a versioned library used in every execution environment. The service may accept JSON while training reads Parquet, but both must call the same semantic transformation code after ingestion.

Scikit-learn similarly recommends pipeline-based composition to prevent inconsistent preprocessing and data leakage.

Category levels deserve explicit treatment

XGBoost 3.1 and later can remember dataframe category encodings and perform automatic recoding for supported Python dataframe inputs. It also requires compatible category value types and documents restrictions around how categorical inputs are represented.

The model’s category support is valuable, but an application-level schema is still necessary.

Consider a service request containing:

{
  "shipping_mode": "same_day"
}

There are several possible meanings:

• It is an invalid request.

• It is a new product that requires retraining.

• It should map to an explicit "other" category.

• It should be treated as missing.

• It should be handled by a fallback model.

Letting a dataframe silently convert an unknown value to missing makes that policy accidental.

The example raises an error when it sees an undeclared category. This is intentionally strict. A production system may choose a different policy, but the policy should be versioned and tested.

Row order is also part of the experiment

Even when a training algorithm is deterministic for a fixed matrix, changing row order can alter:

• How a split is generated

• How groups are batched

• Quantile construction

• Distributed partitioning

• Floating-point accumulation

• Tie resolution

• The meaning of saved positional indices

Before fitting, define a stable ordering such as:

training_frame = training_frame.sort_index()

Do not sort on a non-unique timestamp and assume the result is stable. Use a deterministic compound key:

training_frame = training_frame.sort_values(
    ["event_time", "order_id"],
    kind="stable",
)

When row order is semantically meaningful, as in time series, preserve the real chronology rather than sorting merely to obtain convenient determinism.

What bit-identical actually guarantees

The pinned probe establishes a narrow claim:

Within the tested environment, repeated fitting with the same data, row order, schema, parameters, thread count, tree method, and device produced exactly equal predictions.

It does not establish that:

• Every CPU architecture will produce the same model bytes.

• A future XGBoost version will preserve identical trees.

• A CUDA build will match a CPU build.

• A distributed run will match a single-node run.

• A retrained model on changed data should remain identical.

• The validation score is unbiased.

• The features are causally meaningful.

• The model is safe to deploy.

State the scope in release documentation. “Reproducible” without a boundary is too vague to audit.

A useful hierarchy is:

1. Repeated-fit equality: same process, same machine image.

2. Clean-rebuild equality: new environment created from the lock and image.

3. Cross-host equality: separate hosts using the same image and architecture.

4. Cross-platform equality: different operating systems or architectures.

5. Cross-version stability: different library versions.

Each level is a stronger requirement. Do not promise level five because level one passed.

Common failure modes

Pinning only XGBoost

NumPy, pandas, scikit-learn, Python, and native runtimes also participate in the run.

Setting a seed but leaving n_jobs=None

The effective thread count can change with the host, CPU quota, or scheduler.

Leaving tree_method="auto"

Current auto behavior is documented, but an explicit method is a clearer long-lived contract.

Saving a pickle as the only artifact

A runtime snapshot is not a durable model contract. XGBoost recommends its model serialization formats for portable model storage and distinguishes model serialization from memory snapshots.

Saving feature names but not category levels

Two dataframes can have the same column names while assigning different semantic codes to categories.

Recreating the split from a seed every time

A changed row order or changed dataset creates a different split despite the same seed.

Looking at the holdout during feature development

Once holdout performance changes your next experiment, the holdout has become development data.

Choosing the best-looking metric afterward

The reported metric must be selected before candidate scores are known.

Using approximate equality without recording the tolerance

A tolerance may be appropriate for cross-platform checks, but define it in advance:

• Absolute tolerance

• Relative tolerance

• Fraction of predictions allowed to differ

• Maximum acceptable metric drift

• Whether class decisions may change

Do not replace an exact assertion with allclose merely to make a failing test green.

A release checklist you can defend

Before approving a model, verify the following.

Execution

• Python and package versions are pinned.

• The container or machine image is identified.

• random_state is explicit.

• n_jobs is explicit.

• tree_method is explicit.

• device is explicit.

• Two pinned fits produce bit-identical predictions.

• The reproducibility probe result is stored.

Data

• The dataset snapshot is identifiable.

• Split membership is saved by stable identifier.

• Split overlap is checked.

• Feature names and order are saved.

• Numeric dtypes are saved.

• Category levels are saved.

• Unknown-category behavior is tested.

• Training and serving call the same transformation code.

Validation

• The primary metric was selected before comparison.

• Candidate configurations were declared or generated under a declared search policy.

• The holdout was excluded from selection.

• The selected configuration was refitted without using holdout labels.

• The final holdout result is immutable and traceable to a model hash.

• Further tuning requires a new evaluation design.

Artifact integrity

• The model uses JSON or UBJSON serialization.

• Training configuration is stored beside the model.

• Environment metadata is stored beside the model.

• Files are covered by a checksum manifest.

• A parity sample passes through both training and serving paths.

• Prediction equality is asserted in CI.

Put the contract into CI

The reproducibility and parity checks should not live only in documentation.

A minimal CI sequence is:

set -euo pipefail

python -m pip install -r requirements.txt
python -m pip check

python reproducible_xgb.py probe
python reproducible_xgb.py train
python reproducible_xgb.py parity

Do not include evaluate-holdout in ordinary pull-request CI. The holdout evaluation belongs in a controlled release workflow after model selection is complete.

For a production pipeline, add comparisons against an approved reference:

import numpy as np

expected = np.load(
    "reference/predictions.npy",
    allow_pickle=False,
)

actual = np.load(
    "artifacts/predictions.npy",
    allow_pickle=False,
)

np.testing.assert_array_equal(expected, actual)

When exact cross-host equality is not a supported guarantee, replace this with a predeclared tolerance policy and report the full difference summary. Never silently downgrade the test.

The standard to aim for

A defensible XGBoost result is not “I ran the notebook twice and got roughly the same AUC.”

It is a chain of evidence:

• The environment is identified.

• The data snapshot is identified.

• The split IDs are immutable.

• The metric was chosen before results.

• The holdout did not guide development.

• The execution parameters include seed, threads, tree method, and device.

• Repeated fits were compared at prediction level.

• The feature schema travels with the model.

• Training and serving transformations are the same code path.

• Every released file is covered by a manifest.

Run the probe on your real training image today. Commit the protocol before your next experiment, fail CI on prediction or schema drift, and refuse to publish a holdout score that cannot be traced back to an exact model, environment, split, and feature contract.