XGBoost vs LightGBM vs CatBoost: an honest benchmark on the same data and the same budget

LEARN · GRADIENT BOOSTING WITH XGBOOST

The result in one paragraph

Under the same 20-second CPU tuning budget, CatBoost produced the best test AUC with its first configuration, despite taking roughly 11 times longer to fit than the winning XGBoost trial. XGBoost delivered the best threshold accuracy, the most completed trials, and a strong speed–quality balance. LightGBM used the least memory but needed more careful regularization and still finished third on this mixed, high-cardinality retail dataset.

That is the central lesson: the fastest training library is not necessarily the fastest route to a good model.

What was benchmarked

The dataset is a seeded synthetic retail-conversion problem:

80,000 transactions
10 numeric features
7 categorical features
Cardinalities ranging from 5 to 350
A binary purchase-conversion target
Numeric nonlinearities, categorical effects, and cross-feature interactions
A fixed 60%/20%/20% train, validation, and test split

Each library received:

Eight CPU threads
Native categorical features
Histogram-based training where applicable
Early stopping
The same validation and test rows
A 20-second tuning budget
A predetermined configuration sequence
No GPU and no external hyperparameter optimizer

A new trial was skipped when the previous trial’s duration suggested that it would exceed the remaining budget. Prediction latency was measured after a warm-up, using repeated predictions over the complete 16,000-row test set.

Versions: current releases versus measured builds

As of July 31, 2026, the current stable releases are XGBoost 3.3.0, LightGBM 4.7.0, and CatBoost 1.2.10.

The measured benchmark image contained XGBoost 3.1.3, LightGBM 4.6.0, and CatBoost 1.2.8. The script below pins the current releases and uses their current APIs. Absolute timings can move between hardware and package builds, so rerun it in your deployment environment before making a production decision.

python -m pip install \
    xgboost==3.3.0 \
    lightgbm==4.7.0 \
    catboost==1.2.10 \
    scikit-learn \
    pandas \
    numpy \
    psutil

The benchmark results

Library	Measured version	Test AUC	Accuracy	Winning fit time	Prediction latency	Peak RSS	Tuning effort	Time to validation AUC ≥ 0.730
CatBoost	1.2.8	0.7402	0.6840	14.71 s	2.11 ms/1,000 rows	475 MB	1 trial	14.89 s
XGBoost	3.1.3	0.7378	0.6843	1.29 s	3.39 ms/1,000 rows	299 MB	14 trials	Not reached
LightGBM	4.6.0	0.7338	0.6826	3.97 s	5.18 ms/1,000 rows	255 MB	4 trials	Not reached

Do not overinterpret the third decimal place. This is one seeded split, not a confidence interval. CatBoost’s AUC advantage over XGBoost was only about 0.0025, so a serious model-selection exercise should repeat the benchmark across several seeds or time-based folds.

The operational differences were less ambiguous:

XGBoost iterated through configurations extremely quickly.
LightGBM had the smallest memory footprint.
CatBoost required almost no tuning but had an expensive first fit.
CatBoost’s symmetric trees also gave it the fastest batch prediction in this run.

Why the libraries behave differently

XGBoost: controlled, level-wise growth

XGBoost’s histogram tree method defaults to a depth-wise growth policy. In practical terms, it expands a tree level by level, producing a comparatively balanced structure. Its histogram implementation can also use a loss-guided policy, but that must be selected explicitly.

This structure makes XGBoost relatively predictable:

max_depth has an intuitive effect.
Regularization parameters behave consistently.
A bad configuration usually fails gracefully rather than growing one pathological branch.
Short trials make broad parameter exploration affordable.

That is exactly what happened here. XGBoost completed 14 trials in less than 20 seconds and gradually improved as min_child_weight and L2 regularization increased.

Where it bites:

High-cardinality categoricals still require deliberate validation.
Its categorical support remains marked experimental in the 3.3 documentation.
Fast trials can tempt you into an unnecessarily large search.
The best configuration is often task-specific rather than an obvious default.

XGBoost is the strongest default when you need a mature, controllable baseline and expect to tune systematically.

LightGBM: aggressive leaf-wise growth

LightGBM grows the leaf that offers the greatest immediate loss reduction, rather than expanding every node at the same depth. With the number of leaves held constant, this best-first strategy can reduce training loss faster than level-wise growth. It can also create highly asymmetric trees.

That aggressiveness is both its superpower and its trap.

On large datasets, it can find productive interactions quickly. On smaller or noisy datasets, one branch can become too specialized. LightGBM’s own documentation warns that leaf-wise growth can overfit when the dataset is small and recommends controlling depth.

The relevant controls are not just num_leaves:

Increase min_child_samples.
Add reg_lambda or reg_alpha.
Limit max_depth.
Reduce num_leaves.
Smooth high-cardinality categoricals with cat_smooth.

In this benchmark, increasing minimum leaf size and categorical smoothing helped considerably. The naive configuration was not competitive.

GOSS and EFB

LightGBM is also associated with two efficiency techniques:

Gradient-based One-Side Sampling, or GOSS, retains examples with large gradients and samples from examples with small gradients.
Exclusive Feature Bundling, or EFB, combines sparse features that rarely contain nonzero values simultaneously.

These techniques reduce the rows or effective features considered during split search. The original LightGBM paper introduced both approaches.

One current-API detail matters: GOSS is a selectable sampling strategy, not something you should assume is active in ordinary gradient boosting. Modern LightGBM exposes it through data_sample_strategy="goss". EFB is normally enabled and can be disabled with enable_bundle=false.

Our dense, 17-column dataset gave EFB little room to shine. It becomes more compelling for wide, sparse inputs such as encoded event, advertising, or text-derived features.

CatBoost: ordered statistics instead of naive encoding

CatBoost was designed around categorical variables. Rather than requiring global one-hot encoding, it constructs numeric statistics from categories and category combinations. Its documentation explicitly advises against preprocessing its categorical columns with one-hot encoding because doing so can hurt both speed and quality.

The important idea is ordered processing.

A naive target encoding can leak information from a row’s own label into its encoded feature. CatBoost calculates categorical statistics using ordered permutations so that an example’s representation is based on earlier examples rather than its own target. Ordered boosting applies a related principle while estimating residuals. These mechanisms were introduced to reduce prediction shift and target leakage.

That extra work explains CatBoost’s slow first fit:

It builds categorical statistics.
It may evaluate category combinations.
Its ordered machinery has more setup cost.
Symmetric trees can require evaluating splits across an entire level.

The payoff is a strong model without a separate encoding pipeline. In our benchmark, the first depth-6 configuration was already the overall AUC winner.

Reproduce the experiment

The following compact generator creates the same type of mixed retail dataset. It avoids external data downloads and keeps all categorical columns as pandas categories.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

SEED = 20260731
rng = np.random.default_rng(SEED)
n = 80_000

X = pd.DataFrame({
    "order_value": rng.lognormal(3.5, 0.75, n),
    "discount_pct": rng.beta(2, 8, n),
    "account_age_days": rng.gamma(2.2, 180, n),
    "page_views": rng.poisson(6, n),
    "shipping_days": rng.integers(1, 9, n),
    "prior_orders": rng.poisson(3, n),
    "returns_rate": rng.beta(1.5, 12, n),
    "ad_score": rng.normal(0, 1, n),
    "hour": rng.integers(0, 24, n),
    "stock_ratio": rng.beta(5, 2, n),
})

cardinalities = {
    "channel": 8,
    "region": 24,
    "device": 5,
    "campaign": 120,
    "category": 40,
    "segment": 7,
    "merchant": 350,
}

category_values = {}
category_effects = {}

for name, size in cardinalities.items():
    values = rng.integers(0, size, n)
    category_values[name] = values
    category_effects[name] = rng.normal(0, 0.35, size)
    X[name] = pd.Categorical(values.astype(str))

logit = (
    -2.0
    + 0.015 * np.minimum(X["order_value"], 180)
    + 2.2 * X["discount_pct"]
    + 0.06 * np.sqrt(X["account_age_days"])
    + 0.09 * X["page_views"]
    - 0.12 * X["shipping_days"]
    + 0.12 * np.log1p(X["prior_orders"])
    - 1.8 * X["returns_rate"]
    + 0.35 * X["ad_score"]
    + 0.8 * (X["stock_ratio"] > 0.8)
)

for name, values in category_values.items():
    logit += category_effects[name][values]

logit += 0.9 * (
    (category_values["channel"] == 2)
    & (category_values["device"] == 0)
)
logit += 0.8 * (
    category_values["campaign"] % 17
    == category_values["region"] % 17
)
logit += rng.normal(0, 1.25, n)

probability = 1.0 / (1.0 + np.exp(-logit))
y = rng.binomial(1, probability)

X_train, X_temp, y_train, y_temp = train_test_split(
    X,
    y,
    test_size=0.40,
    stratify=y,
    random_state=SEED,
)

X_valid, X_test, y_valid, y_test = train_test_split(
    X_temp,
    y_temp,
    test_size=0.50,
    stratify=y_temp,
    random_state=SEED,
)

These are representative winning model definitions using current APIs:

from catboost import CatBoostClassifier
from lightgbm import LGBMClassifier
from xgboost import XGBClassifier

xgboost_model = XGBClassifier(
    n_estimators=1200,
    learning_rate=0.05,
    max_depth=6,
    min_child_weight=20,
    reg_lambda=16,
    subsample=0.85,
    colsample_bytree=0.85,
    tree_method="hist",
    enable_categorical=True,
    max_cat_to_onehot=8,
    eval_metric="auc",
    early_stopping_rounds=40,
    n_jobs=8,
    random_state=SEED,
)

lightgbm_model = LGBMClassifier(
    n_estimators=1200,
    learning_rate=0.05,
    num_leaves=31,
    min_child_samples=120,
    reg_lambda=8,
    cat_smooth=40,
    subsample=0.85,
    subsample_freq=1,
    colsample_bytree=0.85,
    n_jobs=8,
    random_state=SEED,
    verbosity=-1,
)

catboost_model = CatBoostClassifier(
    iterations=1200,
    learning_rate=0.05,
    depth=6,
    l2_leaf_reg=5,
    random_strength=1.0,
    loss_function="Logloss",
    eval_metric="AUC",
    early_stopping_rounds=40,
    thread_count=8,
    random_seed=SEED,
    allow_writing_files=False,
    verbose=False,
)

The cherry on the cake

CatBoost was the slowest library per fit, yet it was the only one to cross validation AUC 0.730 within the budget.

XGBoost: 14 attempts, no crossing
LightGBM: 4 attempts, no crossing
CatBoost: 1 attempt, crossing after 14.89 seconds

This is the difference between fit time and time-to-a-good-model. Teams often benchmark only a single training call, then ignore the engineer-hours and compute spent finding acceptable parameters.

CatBoost lost the stopwatch race and won the development race.

A practical chooser rubric

Choose XGBoost when:

You need a dependable general-purpose baseline.
Training and tuning speed both matter.
Your team wants fine control over regularization.
Numeric features dominate or categories are already well managed.

Choose LightGBM when:

The dataset is very large, wide, or sparse.
Memory pressure is a primary constraint.
You can validate leaf-size and depth controls carefully.
You understand that its aggressive growth can overfit small data.

Choose CatBoost when:

Raw categorical columns carry important signal.
Cardinalities are moderate or high.
You want to avoid maintaining a target-encoding pipeline.
A slower initial fit is acceptable in exchange for less tuning.

For an unfamiliar dataset, do not choose from reputation. Give all three the same split, metric, thread count, and tuning budget.

Your next step

Run this benchmark on your own table using at least three seeds or time-based folds. Record validation quality, final test quality, fit time, batch latency, peak memory, and total tuning time—then choose the library that reaches your production target with the least total engineering cost.