Gradient boosting from the ground up: why XGBoost and LightGBM still rule tabular data in 2026

Tabular ML has an unfashionable truth: the best model is often not the newest architecture.

For structured rows and columns—orders, prices, telemetry, energy usage—gradient-boosted trees remain exceptionally hard to beat. In tabular data in 2026, XGBoost and LightGBM still offer a rare combination of accuracy, training speed, modest preprocessing, and predictable deployment.

The current stable releases are XGBoost 3.3.0 and LightGBM 4.7.0. Current XGBoost GPU code should use tree_method="hist" with device="cuda", not legacy GPU-specific tree methods.

Reproducible pins:

python -m pip install "xgboost==3.3.0" "lightgbm==4.7.0"

The reason these libraries remain strong is not mysterious. GradientBoosting repeatedly asks one question:

What small tree would most improve the predictions I already have?

Gradient boosting in one minute

Suppose we are predicting a continuous target with squared-error loss.

Start with a constant prediction:

prediction₀(x) = 0

Then repeat:

• Compute each row’s residual: actual − current prediction.

• Fit a small decision tree to those residuals.

• Add the tree’s output to the current prediction.

• Optionally shrink the update with a learning rate.

After two trees:

prediction₂(x) = prediction₀(x) + η·tree₁(x) + η·tree₂(x)

Here, η is the learning rate.

Each tree can be weak. A depth-one tree—a stump—makes only one split. The ensemble becomes powerful because later trees specialize in correcting earlier mistakes.

Two stumps by hand

Consider one feature and six targets:

x	y
0	0
1	0
2	1
3	2
4	4
5	5

Use a starting prediction of zero and a learning rate of one.

The best first stump splits at x < 3.5.

• Left leaf: (0 + 0 + 1 + 2) / 4 = 0.75

• Right leaf: (4 + 5) / 2 = 4.5

Its predictions are:

[0.75, 0.75, 0.75, 0.75, 4.50, 4.50]

The residuals become:

[-0.75, -0.75, 0.25, 1.25, -0.50, 0.50]

The second stump fits those residuals. Its best split is x < 1.5.

• Left update: −0.75

• Right update: 0.375

Adding both stumps gives:

[0.000, 0.000, 1.125, 1.125, 4.875, 4.875]

The second tree does not restart the problem. It concentrates on what the first tree failed to explain.

Build the booster in NumPy

This small implementation supports one numerical feature, squared-error loss, and regression stumps. That is enough to expose the core algorithm.

import numpy as np


def fit_stump(x: np.ndarray, target: np.ndarray) -> tuple[float, float, float]:
    values = np.unique(x)
    cuts = (values[:-1] + values[1:]) / 2
    best = None

    for cut in cuts:
        left = x < cut
        left_value = target[left].mean()
        right_value = target[~left].mean()

        prediction = np.where(left, left_value, right_value)
        squared_error = np.sum((target - prediction) ** 2)

        if best is None or squared_error < best[0]:
            best = (squared_error, cut, left_value, right_value)

    _, cut, left_value, right_value = best
    return cut, left_value, right_value


def train_booster(
    x: np.ndarray,
    y: np.ndarray,
    n_estimators: int = 2,
    learning_rate: float = 1.0,
) -> list[tuple[float, float, float]]:
    prediction = np.zeros_like(y, dtype=float)
    trees = []

    for _ in range(n_estimators):
        residual = y - prediction
        tree = fit_stump(x, residual)
        trees.append(tree)

        cut, left_value, right_value = tree
        update = np.where(x < cut, left_value, right_value)
        prediction += learning_rate * update

    return trees


def predict_booster(
    x: np.ndarray,
    trees: list[tuple[float, float, float]],
    learning_rate: float = 1.0,
) -> np.ndarray:
    prediction = np.zeros_like(x, dtype=float)

    for cut, left_value, right_value in trees:
        update = np.where(x < cut, left_value, right_value)
        prediction += learning_rate * update

    return prediction


x = np.array([0, 1, 2, 3, 4, 5], dtype=float)
y = np.array([0, 0, 1, 2, 4, 5], dtype=float)

tiny_trees = train_booster(x, y)
tiny_predictions = predict_booster(x, tiny_trees)

print(tiny_predictions)

[0.    0.    1.125 1.125 4.875 4.875]

Production libraries add multiple features, missing-value handling, histogram construction, parallelism, sampling, regularization, categorical support, and optimized inference. The sequential correction mechanism remains recognizable.

The XGBoost equivalent

Configure XGBoost to match our implementation:

• Two depth-one trees

• Squared-error loss

• Learning rate of one

• Starting prediction of zero

• No L1, L2, or split regularization

• Exact split enumeration

import numpy as np
from xgboost import XGBRegressor


xgb = XGBRegressor(
    n_estimators=2,
    max_depth=1,
    learning_rate=1.0,
    objective="reg:squarederror",
    base_score=0.0,
    reg_lambda=0.0,
    reg_alpha=0.0,
    gamma=0.0,
    min_child_weight=0.0,
    tree_method="exact",
    n_jobs=1,
).fit(x.reshape(-1, 1), y)

xgb_predictions = xgb.predict(x.reshape(-1, 1))

np.testing.assert_array_equal(
    tiny_predictions.astype(np.float32),
    xgb_predictions,
)

print(xgb_predictions)

[0.    0.    1.125 1.125 4.875 4.875]

The predictions are identical.

tree_method="exact" is appropriate for this microscopic demonstration because it enumerates every split candidate, like our NumPy search. XGBoost 3.3 supports exact, approx, and hist; for practical datasets, hist is normally the useful choice.

What XGBoost adds: gradients, Hessians, and regularization

“Fit the residuals” is exact intuition for squared error, but XGBoost supports many differentiable objectives.

For each row, it calculates:

• gᵢ: the first derivative of the loss with respect to the current prediction

• hᵢ: the second derivative, or Hessian

It approximates the next boosting step as:

objective ≈ Σ[gᵢ·f(xᵢ) + ½·hᵢ·f(xᵢ)²] + Ω(f)

A simplified tree penalty is:

Ω(f) = γ·T + ½·λ·Σwⱼ² + α·Σ|wⱼ|

Where:

• T is the number of leaves.

• gamma is exposed to users as the minimum loss reduction required to make a split.

• reg_lambda applies L2 regularization to leaf weights.

• reg_alpha applies L1 regularization.

• min_child_weight requires enough Hessian mass in a child.

The structural derivation uses γ·T, while the practical meaning of gamma is “do not create a split unless it improves the objective enough.”

For one leaf, define:

G = sum of gradients reaching the leaf
H = sum of Hessians reaching the leaf

Without L1 regularization, the optimal leaf weight is:

leaf weight = −G / (H + λ)

For squared error:

g = prediction − target
h = 1

When λ = 0, the optimal update becomes the mean residual—the exact operation used in our NumPy implementation.

For logistic classification, the Hessian changes with the predicted probability. XGBoost therefore uses both the direction of the error and the local curvature of the loss.

Why LightGBM behaves differently

XGBoost normally grows trees depth-wise: it splits nodes at similar depths before proceeding downward.

LightGBM grows trees leaf-wise, also called best-first growth. At each step, it expands the leaf that offers the largest reduction in loss, even when another branch remains shallow.

This can converge quickly, but it may create highly asymmetric, overly specific trees. The important controls are therefore:

• num_leaves

• min_child_samples

• min_sum_hessian_in_leaf

• max_depth

• min_split_gain

LightGBM’s documentation explicitly warns that leaf-wise growth can overfit when num_leaves and leaf-size constraints are too permissive.

A useful starting distinction is:

• Tune max_depth first in XGBoost.

• Tune num_leaves and minimum leaf size first in LightGBM.

Head-to-head: trees versus a small MLP

Here is a neutral TabularML benchmark using 50,000 synthetic server-telemetry rows and 24 numeric features.

The target is generated from threshold-heavy rules involving latency, queue pressure, resource saturation, and feature interactions. Two percent of labels are randomly flipped.

All three models receive:

• The same seeded train/test split

• The same standardized float32 features

• The same five CPU threads where supported

• Three training runs with different model seeds

from time import perf_counter

import numpy as np
from lightgbm import LGBMClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier


rng = np.random.default_rng(42)
X = rng.uniform(0, 1, size=(50_000, 24)).astype(np.float32)

score = (
    1.8 * ((X[:, 0] > 0.78) & (X[:, 1] > 0.62))
    + 1.5 * ((X[:, 2] > 0.90) | (X[:, 3] > 0.94))
    + 1.4 * ((X[:, 4] < 0.12) & (X[:, 5] > 0.72))
    + 1.2 * ((X[:, 6] > 0.55) & (X[:, 7] > 0.55) & (X[:, 8] > 0.55))
    + 1.1 * ((X[:, 9] > 0.82) ^ (X[:, 10] > 0.82))
    + 0.7 * (X[:, 13] > 0.85)
    + rng.normal(0, 0.35, size=X.shape[0])
)

y = (score > 1.65).astype(np.int32)
y[rng.random(y.size) < 0.02] ^= 1

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    stratify=y,
    random_state=42,
)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train).astype(np.float32)
X_test = scaler.transform(X_test).astype(np.float32)

models = {
    "XGBoost": XGBClassifier(
        n_estimators=250,
        max_depth=6,
        learning_rate=0.08,
        subsample=0.9,
        colsample_bytree=0.9,
        tree_method="hist",
        device="cpu",
        n_jobs=5,
        eval_metric="logloss",
        random_state=42,
    ),
    "LightGBM": LGBMClassifier(
        n_estimators=250,
        num_leaves=63,
        learning_rate=0.08,
        subsample=0.9,
        subsample_freq=1,
        colsample_bytree=0.9,
        n_jobs=5,
        verbosity=-1,
        random_state=42,
    ),
    "MLP": MLPClassifier(
        hidden_layer_sizes=(64, 32),
        batch_size=512,
        max_iter=80,
        early_stopping=True,
        n_iter_no_change=6,
        tol=1e-3,
        random_state=42,
    ),
}

for name, model in models.items():
    started = perf_counter()
    model.fit(X_train, y_train)
    elapsed = perf_counter() - started
    accuracy = accuracy_score(y_test, model.predict(X_test))
    print(f"{name:10s} accuracy={accuracy:.4f} train_seconds={elapsed:.2f}")

Representative five-core Xeon results, averaged for accuracy and using median training time across three seeded runs:

Model	Test accuracy	Training time
XGBoost	89.94%	0.96 s
LightGBM	89.67%	1.22 s
Small MLP	80.74%	2.04 s

These numbers are not a universal leaderboard. This dataset deliberately contains discontinuities, Boolean interactions, and narrow threshold regions—the structures trees represent naturally.

The lesson is methodological: begin a tabular project with a strong boosted-tree baseline before spending days tuning a neural architecture.

Cherry on the cake: does tabular deep learning still lose?

The popular claim that “deep learning loses on tabular data” came from benchmarks in which boosted trees often beat dataset-specific neural networks under realistic tuning budgets.

The updated story is more interesting.

The 2025 TabArena benchmark manually selected 51 real-world classification and regression datasets from 1,053 candidates. It compared 16 model classes, including XGBoost, LightGBM, CatBoost, conventional MLPs, specialized neural architectures, and tabular foundation models. Tunable models received 200 sampled configurations, eight-fold inner cross-validation, and up to one hour per configuration. Binary classification used ROC AUC, multiclass classification used log-loss, and regression used RMSE, with cross-dataset results aggregated through Elo ratings.

What did it find?

• Boosted trees remained strong practical contenders.

• Deep models caught up or won when given larger tuning budgets and post-hoc ensembling.

• Foundation models were especially strong on smaller datasets.

• Cross-model ensembles were stronger than treating one model family as universally best.

So “tabular deep learning still loses” is not a law. It is a result conditioned on dataset size, preprocessing, model class, tuning budget, validation design, ensembling, hardware, and metric.

That nuance strengthens the case for XGBoost and LightGBM rather than weakening it: they provide excellent performance before you pay the computational and engineering cost of specialized deep TabularML.

What to remember

GradientBoosting works because each new tree corrects the current ensemble.

XGBoost adds second-order optimization and explicit regularization. LightGBM adds aggressive histogram-based, leaf-wise growth. Both directly represent thresholds, missing-value routes, and conditional feature interactions—the grammar of real tabular data.

Run the benchmark, change max_depth and num_leaves, and observe where accuracy begins to fall. That experiment is the fastest route from understanding boosting to using it well.