Tuning XGBoost without fooling yourself: early stopping, learning rate, and the parameters that matter

XGBoost tuning becomes dangerous when optimization and evaluation blur together. A workflow can try dozens of configurations, stop each run at its best validation round, select the lowest validation error, and then report that same number as proof of generalization.

That number is a selection result, not an unbiased estimate.

A defensible workflow separates three decisions:

• Couple learning_rate with the available boosting-round budget.

• Use early stopping on data that was not used to fit tree splits or leaf weights.

• Judge the selected workflow once on untouched data, or with outer cross-validation.

Reproducible setup with a three-way split

The snippets target XGBoost 3.3.0, Optuna 4.9.0, scikit-learn 1.9.0, NumPy 2.5.1, SciPy 1.18.0, pandas 3.0.5, and Matplotlib 3.11.1—the current releases used for this July 2026 configuration.

python -m pip install \
  "xgboost==3.3.0" \
  "optuna==4.9.0" \
  "scikit-learn==1.9.0" \
  "numpy==2.5.1" \
  "scipy==1.18.0" \
  "pandas==3.0.5" \
  "matplotlib==3.11.1"

Create a seeded retail-demand regression problem. The final split is 60% training, 20% tuning, and 20% testing.

from __future__ import annotations

import numpy as np
import optuna
import pandas as pd
import xgboost as xgb
from matplotlib import pyplot as plt
from sklearn.datasets import make_regression
from sklearn.metrics import root_mean_squared_error
from sklearn.model_selection import train_test_split

SEED = 42

X_raw, y_linear = make_regression(
    n_samples=4_500,
    n_features=12,
    n_informative=8,
    noise=18.0,
    random_state=SEED,
)

feature_names = [
    "price_index",
    "discount_rate",
    "store_traffic",
    "inventory_level",
    "competitor_price",
    "holiday_intensity",
    "local_income",
    "store_age",
] + [f"signal_{i}" for i in range(4)]

X = pd.DataFrame(X_raw, columns=feature_names)
y = (
    500
    + y_linear
    + 35 * np.sin(X["price_index"])
    + 20 * X["discount_rate"] * X["store_traffic"]
    + 12 * np.maximum(X["inventory_level"], 0) ** 2
)

X_train, X_holdout, y_train, y_holdout = train_test_split(
    X,
    y,
    test_size=0.40,
    random_state=SEED,
)
X_tune, X_test, y_tune, y_test = train_test_split(
    X_holdout,
    y_holdout,
    test_size=0.50,
    random_state=SEED,
)

BASE_PARAMS = {
    "objective": "reg:squarederror",
    "eval_metric": "rmse",
    "tree_method": "hist",
    "max_depth": 4,
    "min_child_weight": 5.0,
    "subsample": 0.8,
    "colsample_bytree": 0.8,
    "reg_lambda": 1.0,
    "random_state": SEED,
    "n_jobs": -1,
}

Keep the roles strict:

• train: fit splits and leaf values.

• tune: compare settings and choose stopping rounds.

• test: evaluate the locked workflow once.

Tune learning rate and tree count together

XGBoost defines eta, aliased as learning_rate, as shrinkage applied after each boosting step. Smaller updates usually need more boosting rounds, so comparing learning rates under the same small n_estimators cap is unfair.

Plot complete tuning curves instead of comparing arbitrary final rounds:

curve_specs = {
    "learning_rate=0.20, cap=400": {
        "learning_rate": 0.20,
        "n_estimators": 400,
    },
    "learning_rate=0.03, cap=2000": {
        "learning_rate": 0.03,
        "n_estimators": 2_000,
    },
}

plt.figure(figsize=(9, 5))

for label, extra_params in curve_specs.items():
    model = xgb.XGBRegressor(**BASE_PARAMS, **extra_params)
    model.fit(
        X_train,
        y_train,
        eval_set=[(X_tune, y_tune)],
        verbose=False,
    )

    validation_rmse = np.asarray(
        model.evals_result()["validation_0"]["rmse"],
        dtype=float,
    )
    rounds = np.arange(1, validation_rmse.size + 1)
    best_round = int(validation_rmse.argmin()) + 1

    plt.plot(
        rounds,
        validation_rmse,
        label=f"{label}; best={best_round}",
    )

plt.xlabel("Boosting round")
plt.ylabel("Tuning RMSE")
plt.title("Learning rate changes the useful tree budget")
plt.legend()
plt.tight_layout()
plt.show()

Read the graph in three ways:

• Which curve reaches the lowest minimum?

• How many trees does it need to become competitive?

• Does error flatten or rise before the configured cap?

The lesson is not that 0.03 always wins. It is that learning_rate and n_estimators describe one capacity schedule and should be tuned as a pair.

Let early stopping choose the effective tree count

Give training a deliberately generous ceiling, then stop when the tuning metric fails to improve for a patience window.

early_stopped = xgb.XGBRegressor(
    **BASE_PARAMS,
    learning_rate=0.03,
    n_estimators=3_000,
    early_stopping_rounds=75,
)

early_stopped.fit(
    X_train,
    y_train,
    eval_set=[(X_tune, y_tune)],
    verbose=False,
)

print("Best boosting rounds:", early_stopped.best_iteration + 1)
print("Best tuning RMSE:", float(early_stopped.best_score))

In the current scikit-learn interface, early_stopping_rounds belongs on the estimator. best_iteration is zero-based, and ordinary predict() calls automatically use the best iteration found during early stopping.

Early stopping is an implicit regularizer: it limits how many additive corrections the model can make. It does not replace controls on tree depth, child weight, row sampling, column sampling, or leaf-weight penalties.

Rank parameters by measured metric movement

Do not spend equal search budget everywhere. Run coarse one-parameter sweeps first, then rank parameters by the difference between their best and worst tuning RMSE.

sweeps = {
    "max_depth": [2, 4, 6],
    "min_child_weight": [1.0, 10.0, 40.0],
    "subsample": [0.65, 0.85, 1.0],
    "colsample_bytree": [0.65, 0.85, 1.0],
    "reg_lambda": [0.1, 1.0, 20.0],
}

rows: list[dict[str, float | int | str]] = []

for parameter, values in sweeps.items():
    for value in values:
        params = BASE_PARAMS | {parameter: value}
        model = xgb.XGBRegressor(
            **params,
            learning_rate=0.05,
            n_estimators=2_000,
            early_stopping_rounds=60,
        )
        model.fit(
            X_train,
            y_train,
            eval_set=[(X_tune, y_tune)],
            verbose=False,
        )
        rows.append(
            {
                "parameter": parameter,
                "value": value,
                "rmse": float(model.best_score),
                "best_trees": model.best_iteration + 1,
            }
        )

sweep_results = pd.DataFrame(rows)
ranking = (
    sweep_results.groupby("parameter", as_index=False)
    .agg(best_rmse=("rmse", "min"), worst_rmse=("rmse", "max"))
    .assign(
        metric_movement=lambda frame: (
            frame["worst_rmse"] - frame["best_rmse"]
        )
    )
    .sort_values("metric_movement", ascending=False)
)

print(ranking.to_string(index=False))

This prints the ranking produced by your environment instead of claiming fixed movements that may vary across hardware, library builds, or thread scheduling.

Interpret the controls compactly:

• max_depth raises tree capacity.

• min_child_weight blocks weakly supported child nodes.

• subsample and colsample_bytree add row and feature randomness.

• reg_lambda applies L2 regularization to leaf weights.

XGBoost documents larger depth as more complex, while larger child weight and reg_lambda make fitting more conservative.

Use these sweeps for screening, not final selection. Interactions matter: deeper trees may need larger child weights, and stronger sampling can tolerate somewhat greater tree capacity.

The validation-set trap

Early stopping selects the best round on X_tune. Hyperparameter search then selects the best configuration on that same X_tune. The winning score has survived two layers of selection.

Even when many configurations have similar true performance, finite-sample noise makes their tuning scores differ. Selecting the minimum preferentially selects a configuration whose noise was favorable. More trials can improve the model while making the reported best tuning score increasingly optimistic.

The minimum safe fix is the three-way split already in use:

1. Tune and early-stop on X_tune.

2. Freeze features, preprocessing, parameters, and decision rules.

3. Evaluate once on X_test.

For smaller datasets or publication-grade estimates, use nested cross-validation:

• Each outer fold holds out data for unbiased scoring.

• Hyperparameter search and early stopping happen only inside the outer training portion.

• Average the untouched outer-fold scores.

For time-dependent retail or telemetry data, replace random folds with time-ordered splits. For repeated customers, stores, or devices, use grouped splits so the same entity cannot leak across folds.

Finish with a small Optuna study

Optuna 4.9 supports suggest_float, suggest_int, create_study, and study.optimize; logarithmic ranges use log=True.

optuna.logging.set_verbosity(optuna.logging.WARNING)


def objective(trial: optuna.Trial) -> float:
    trial_params = {
        "learning_rate": trial.suggest_float(
            "learning_rate",
            0.01,
            0.20,
            log=True,
        ),
        "max_depth": trial.suggest_int("max_depth", 2, 7),
        "min_child_weight": trial.suggest_float(
            "min_child_weight",
            1.0,
            50.0,
            log=True,
        ),
        "subsample": trial.suggest_float("subsample", 0.6, 1.0),
        "colsample_bytree": trial.suggest_float(
            "colsample_bytree",
            0.6,
            1.0,
        ),
        "reg_lambda": trial.suggest_float(
            "reg_lambda",
            0.1,
            100.0,
            log=True,
        ),
    }

    model = xgb.XGBRegressor(
        **(BASE_PARAMS | trial_params),
        n_estimators=3_000,
        early_stopping_rounds=75,
    )
    model.fit(
        X_train,
        y_train,
        eval_set=[(X_tune, y_tune)],
        verbose=False,
    )

    trial.set_user_attr(
        "best_n_estimators",
        model.best_iteration + 1,
    )
    return float(model.best_score)


study = optuna.create_study(
    direction="minimize",
    sampler=optuna.samplers.TPESampler(seed=SEED),
)
study.optimize(objective, n_trials=20)

selected_model = xgb.XGBRegressor(
    **(BASE_PARAMS | study.best_params),
    n_estimators=3_000,
    early_stopping_rounds=75,
)
selected_model.fit(
    X_train,
    y_train,
    eval_set=[(X_tune, y_tune)],
    verbose=False,
)

locked_test_rmse = root_mean_squared_error(
    y_test,
    selected_model.predict(X_test),
)
selection_gap = locked_test_rmse - study.best_value

print("Best tuning RMSE:", study.best_value)
print("Locked test RMSE:", locked_test_rmse)
print("Selection gap:", selection_gap)
print("Best parameters:", study.best_params)
print("Best tree count:", selected_model.best_iteration + 1)

The selection_gap may be positive or negative in one experiment; its purpose is not to prove bias from a single run. Track it across projects. A persistently favorable tuning score is evidence that the selection process is overfitting its validation data.

After evaluation, a deployment model can be retrained on train + tune using the locked parameters and selected tree count. Do not score the test set again after changing the workflow.

X_development = pd.concat([X_train, X_tune], axis=0)
y_development = pd.concat([y_train, y_tune], axis=0)

production_model = xgb.XGBRegressor(
    **(BASE_PARAMS | study.best_params),
    n_estimators=selected_model.best_iteration + 1,
)
production_model.fit(X_development, y_development, verbose=False)

Cherry on the cake: a real leaderboard reality check

In the named ARC Prize 2025 write-up, the ARChitects reported that their combined estimate suggested roughly 26%, but the submission scored 21.67% on the public leaderboard and 16.53% on the private split. The official competition report lists their final private score as 16.53%.

That was not an XGBoost competition, but the model-selection lesson is identical: once public feedback repeatedly influences architecture, seeds, blends, and parameters, the public leaderboard becomes part of the training loop.

The tuning checklist

• Couple learning_rate with a generous boosting-round ceiling.

• Use early stopping to choose the effective number of trees.

• Screen parameter families by measured metric movement.

• Treat the best tuning score as a selection statistic, not a final estimate.

• Report one untouched test score or outer-fold average.

Run the study, record the tuning score, locked test score, and selection gap, then add those three numbers to every future model-selection report. That small habit makes it much harder for XGBoost tuning to fool you.