Which model when: a practical decision playbook for tabular, text, images, and small data

Start with the data shape, not the leaderboard

Model selection gets easier when you stop asking, “What is the best machine-learning model?” and ask a more useful set of questions:

• What structure does the input naturally have?

• How many independent labeled examples do I really have?

• What kind of relationship must the model learn?

• How will I validate it without leaking information?

• What are the latency, hardware, interpretability, and retraining constraints?

• Am I choosing a strong engineering baseline, or chasing the current accuracy frontier?

That last distinction matters more in 2026 than it did even a year ago.

For ordinary tabular projects, gradient-boosted trees remain an excellent first serious benchmark. They are mature, fast, strong on nonlinear thresholds and interactions, and easy to deploy on conventional infrastructure.

But they are no longer a defensible answer to the separate question, “Which family currently holds the strongest tabular benchmark results?” The tabular foundation-model wave accelerated rapidly from late 2025 into 2026. TabPFN-2.5 reported beating heavily tuned tree models, TabICLv2 pushed the foundation-model frontier further, and the May 2026 TabPFN-3 technical report reported another substantial step on TabArena and larger tabular regimes.

So the practical rule is not:

Gradient boosting is the best tabular model.

It is:

Gradient boosting is usually one of the best places to begin a serious tabular benchmark, while current tabular foundation models deserve a place in the experiment when maximum predictive performance justifies their extra requirements.

That distinction—engineering default versus benchmark leader—is the foundation of this playbook.

The fast 2026 decision tree

Use this before launching a 30-model tournament.

If the input is ordinary tabular data

Think:

• transactions;

• account or customer attributes;

• retail orders;

• prices and counts;

• sensor aggregates;

• server telemetry;

• categorical business attributes;

• engineered date and time features;

• one row per event, product, device, property, session, or account.

Start with three questions, represented by three model families:

1. Regularized linear model: how much signal is approximately additive?

2. Histogram gradient boosting: how much do nonlinear thresholds and interactions buy?

3. Current tabular foundation model, when feasible: does pretrained tabular structure produce another meaningful gain?

Scikit-learn 1.9 continues to provide HistGradientBoostingClassifier and HistGradientBoostingRegressor; its documentation recommends the histogram-based estimators over the older gradient-boosting implementation for larger sample counts and documents native handling for missing values and categorical features.

The crucial 2026 update is that the third candidate is no longer an exotic research-only footnote. Recent foundation models have won important benchmark comparisons, so an accuracy-sensitive tabular project should not assume the competition ends with XGBoost, LightGBM, CatBoost, or scikit-learn boosting.

If the input is text

First ask what the output requires.

If you mostly need:

• semantic search;

• retrieval;

• clustering;

• near-duplicate detection;

• recommendations based on textual similarity;

• a lightweight classifier over semantic representations;

start with a current pretrained embedding model.

If you need:

• nuanced sequence classification;

• token-level labels;

• sequence labeling;

• task-specific language understanding;

• generation;

• distinctions that disappear when the document is compressed into one vector;

benchmark a pretrained transformer adapted or fine-tuned for the task.

Sentence Transformers remains designed around fixed-size embeddings for tasks such as semantic similarity, retrieval, clustering, and classification, while current Hugging Face Transformers APIs continue to support task-specific fine-tuning of pretrained models.

If the input is images

Start with transfer learning, not random initialization.

Put at least one mature pretrained convolutional network and one pretrained vision transformer or related modern vision backbone into the candidate set when your compute budget permits.

Torchvision 0.28 includes pretrained weights across convolutional families such as ResNet and ConvNeXt and transformer-based families including Vision Transformer and Swin.

The production question is not “CNN or ViT?” as an ideology.

It is:

Which pretrained backbone gives the best validated quality-versus-latency-versus-memory tradeoff on my images?

If labeled data is tiny

Begin by controlling variance.

For structured classification, that normally means:

• logistic regression with regularization;

• careful preprocessing;

• repeated or otherwise variance-aware validation;

• a conservative boosted-tree comparison.

For structured regression:

• ridge regression;

• elastic-net when sparse feature selection is useful;

• a carefully regularized tree model as the nonlinear challenger.

In 2026 there is an important amendment: tiny tabular data is also one of the regimes where tabular foundation models are particularly interesting. The original TabArena work already found foundation models strong on smaller datasets, and subsequent 2025–2026 models pushed that advantage further.

That does not make regularized linear models obsolete. It makes them even more valuable as low-cost reference points.

If interpretability is a hard requirement

Prefer intrinsically transparent models early:

• linear or generalized linear models;

• small decision trees;

• constrained tree ensembles where acceptable.

Do not wait until the end of the project to discover that the numerically strongest candidate cannot satisfy an audit, explanation, latency, or policy requirement.

If you have multiple modalities

Do not fuse everything on day one.

Establish a credible baseline for each modality first. If product metadata, descriptions, and images all predict the same target, learn how much each source contributes independently before building a multimodal system.

Fusion should have to demonstrate incremental value.

Why gradient boosting remains such a useful tabular starting point

Consider an e-commerce conversion model with features such as:

• order value;

• number of items;

• discount percentage;

• days since previous purchase;

• device type;

• traffic source;

• customer tenure.

Real business relationships often resemble conditional rules:

• a discount matters only above a particular basket value;

• mobile behavior differs for very small orders;

• a long gap since the previous purchase matters mainly for low-item-count baskets.

A tree represents those relationships naturally.

It can split on:

• discount > 0.22;

• then order_value > 60;

• then another feature.

A plain linear classifier instead begins with a structure resembling:

score = intercept + w₁x₁ + w₂x₂ + … + wₚxₚ

Without explicit feature crosses, that model does not directly represent an “A AND B” threshold interaction.

Gradient boosting builds an additive collection of trees, each improving the ensemble’s current fit. Histogram-based implementations make this practical on substantial tabular datasets, which is one reason they remain such strong engineering baselines. Scikit-learn’s current histogram estimator also supports missing values, categorical features, monotonic constraints, and interaction constraints, giving it a broader 2026 feature set than the “numeric-only basic baseline” description sometimes associated with older versions.

This is an inductive-bias argument, not a leaderboard claim.

Trees are structurally well suited to thresholded tabular relationships. That remains true even when a foundation model ultimately achieves a better test score.

A runnable mini-benchmark: linear structure versus threshold interactions

A small experiment makes the distinction concrete.

We will create synthetic e-commerce behavior. There is no medical, patient, disease, or other sensitive sample dataset involved.

The target will depend partly on threshold interactions that a tree can discover without manual feature crosses.

Create a reproducible environment

The examples below target current August 2026 releases: scikit-learn 1.9.0, NumPy 2.5.2, and pandas 3.0.5. Scikit-learn 1.9.0 was released in June 2026; the pinned NumPy and pandas releases are also current 2026 versions.

python -m venv .venv

On macOS or Linux:

source .venv/bin/activate

On Windows PowerShell:

.venv\Scripts\Activate.ps1

Install the exact versions used by this walkthrough:

python -m pip install "numpy==2.5.2" "pandas==3.0.5" "scikit-learn==1.9.0"

Create benchmark.py:

import numpy as np
import pandas as pd

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n_samples = 8000

order_value = rng.lognormal(
    mean=3.7,
    sigma=0.7,
    size=n_samples,
)

days_since_last = rng.integers(
    0,
    180,
    size=n_samples,
)

items = rng.poisson(
    3.0,
    size=n_samples,
) + 1

discount = rng.uniform(
    0.0,
    0.5,
    size=n_samples,
)

mobile = rng.integers(
    0,
    2,
    size=n_samples,
)

logit = (
    -2.2
    + 1.7 * ((order_value > 60) & (discount > 0.22))
    + 1.4 * ((days_since_last > 70) & (items <= 2))
    - 1.1 * ((order_value < 25) & (mobile == 1))
    + 0.018 * np.minimum(days_since_last, 100)
    + rng.normal(0.0, 0.65, size=n_samples)
)

probability = 1.0 / (1.0 + np.exp(-logit))
converted = rng.binomial(1, probability)

X = pd.DataFrame(
    {
        "order_value": order_value,
        "days_since_last": days_since_last,
        "items": items,
        "discount": discount,
        "mobile": mobile,
    }
)

y = converted

cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

models = {
    "logistic_regression": make_pipeline(
        StandardScaler(),
        LogisticRegression(max_iter=2000),
    ),
    "hist_gradient_boosting": HistGradientBoostingClassifier(
        learning_rate=0.03,
        max_iter=200,
        max_leaf_nodes=15,
        l2_regularization=3.0,
        random_state=42,
    ),
}

for name, model in models.items():
    scores = cross_validate(
        model,
        X,
        y,
        cv=cv,
        scoring=["roc_auc", "neg_log_loss"],
        n_jobs=-1,
    )

    mean_auc = scores["test_roc_auc"].mean()
    mean_log_loss = -scores["test_neg_log_loss"].mean()

    print(
        f"{name:24s} "
        f"AUC={mean_auc:.3f} "
        f"log_loss={mean_log_loss:.3f}"
    )

Run it:

python benchmark.py

A representative run of this fixed-seed experiment produces approximately:

logistic_regression      AUC=0.693 log_loss=0.610
hist_gradient_boosting   AUC=0.742 log_loss=0.573

Do not read this as “boosting wins by 0.049 AUC.”

Read it as an explanation of why model families separate.

The generated target deliberately contains expressions such as:

• order_value > 60 AND discount > 0.22;

• days_since_last > 70 AND items <= 2.

The tree ensemble discovers splits approximating those interactions directly.

The logistic model sees the individual columns but receives no explicit interaction features. If we add good domain-inspired feature crosses, its performance may improve.

That gives us a more transferable lesson:

Model choice and representation choice are coupled.

A sophisticated architecture cannot rescue a representation that discards the information the task depends on.

Three tabular baselines answer three different questions

In 2026, a strong tabular benchmark suite should often be interpreted this way.

Regularized linear model

This asks:

How far can simple additive structure take us?

A strong linear baseline is useful even if you expect it to lose.

If logistic regression reaches 0.91 AUC and an expensive system reaches 0.912, the linear result tells you that most predictive structure is already accessible through a simple decision surface.

Histogram gradient boosting

This asks:

How much do nonlinear thresholds and interactions add?

It is often the highest-value next experiment because the setup cost is low while the representational jump is substantial.

Tabular foundation model

This asks:

Does a pretrained prior over tabular prediction improve on models trained only from this dataset?

That is now a serious benchmark question rather than a speculative one.

The June 2025 TabArena launch paper found boosted trees remained strong contenders and that foundation models were especially competitive on smaller data. TabPFN-2.5, released later in 2025 and updated in early 2026, then reported substantial gains over tuned tree models. TabICLv2’s February 2026 paper described tabular foundation models as having moved to the top of predictive benchmark rankings, and the May 2026 TabPFN-3 report pushed the reported frontier again.

The correct workflow is therefore not to replace the tree baseline.

It is to add the foundation-model candidate when its deployment profile makes sense.

Your validation split can matter more than another model family

Model selection collapses if the validation experiment does not resemble production.

A sophisticated model evaluated with leakage is less informative than logistic regression evaluated honestly.

Time-dependent observations

Suppose every row represents one week of retail demand.

A random fold can train on October and validate on March from the same timeline. That lets the fitting process learn from observations that would have been in the future at the moment of prediction.

Scikit-learn’s current TimeSeriesSplit is designed for time-ordered data and grows training data forward rather than constructing ordinary shuffled folds.

Repeated entities

Suppose you have many rows per:

• store;

• customer;

• machine;

• building;

• seller;

• product family.

A random split may place the same entity into both training and validation sets.

If production requires generalization to unseen entities, that split answers the wrong question.

GroupKFold keeps a group out of both sides of the same fold, while StratifiedGroupKFold also attempts to preserve class proportions when classification balance matters.

Preprocessing leakage

Another common error is:

1. normalize all rows;

2. choose features using all labels;

3. encode categories using the whole dataset;

4. only then start cross-validation.

The validation fold has already influenced preprocessing.

The cure is simple: put learned preprocessing inside the pipeline being cross-validated.

Mixed numeric and categorical data: make the baseline fair

A linear model is not a meaningful baseline if its preprocessing is careless.

Current scikit-learn provides ColumnTransformer specifically for applying different transforms to different column groups, and OneHotEncoder supports controls such as min_frequency for managing infrequent categories.

For example:

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric_columns = [
    "order_value",
    "discount",
    "customer_tenure_days",
]

categorical_columns = [
    "country",
    "traffic_source",
    "device_type",
]

preprocessor = ColumnTransformer(
    transformers=[
        (
            "numeric",
            StandardScaler(),
            numeric_columns,
        ),
        (
            "categorical",
            OneHotEncoder(
                handle_unknown="ignore",
                min_frequency=5,
            ),
            categorical_columns,
        ),
    ]
)

model = Pipeline(
    steps=[
        ("preprocessor", preprocessor),
        (
            "classifier",
            LogisticRegression(
                max_iter=2000,
            ),
        ),
    ]
)

Now cross-validation fits scaling and category handling using only each fold’s training partition.

That is not merely cleaner code. It changes whether your validation result can be trusted.

Tiny data: reduce variance before you increase capacity

“Tiny” is relative.

Five thousand examples may be tiny for training a large image model from random initialization but ample for fitting a ten-feature ridge regression.

What matters is the relationship among:

• independent observations;

• dimensionality;

• label noise;

• class imbalance;

• feature redundancy;

• model capacity.

With scarce observations, a high-capacity learner has more opportunities to fit accidental patterns.

Regularization pushes the solution toward a lower-variance region. Scikit-learn’s current LogisticRegression applies regularization by default; its C parameter is inverse regularization strength, so smaller C means stronger regularization.

For a tiny structured dataset, I would normally begin with:

• domain-aware feature construction;

• a regularized linear model;

• repeated or carefully grouped validation where appropriate;

• a conservative boosted-tree comparison;

• a current tabular foundation model if the task is important enough to justify the additional dependency, hardware, and deployment review.

That last bullet is the major update from older model-selection advice.

In 2024, “try a tabular foundation model” could reasonably sit near the experimental end of the ladder. By 2026, benchmark evidence makes it a legitimate early accuracy candidate, especially in small-data regimes.

Still, do not skip the linear model.

It gives you:

• fast training;

• low inference cost;

• a compact artifact;

• a stable reference point;

• relatively transparent behavior;

• evidence about how much nonlinear capacity is actually necessary.

Text: use embeddings first when the problem is geometric

Many text problems are fundamentally about distance in a semantic representation.

Suppose you have product descriptions and need to:

• retrieve similar products;

• detect near duplicates;

• group related listings;

• build recommendation candidates;

• classify descriptions into categories with a simple downstream boundary.

You may not need an end-to-end fine-tuned language model.

Instead, convert each piece of text into an embedding and operate on those vectors.

For a current example, use IBM’s granite-embedding-97m-multilingual-r2, released in May 2026. Its model card describes it as a Sentence Transformers-compatible multilingual embedding model intended for semantic retrieval and similarity-style applications.

Pin the environment rather than installing an unspecified future version:

python -m pip install "torch==2.13.0" "transformers==5.15.0" "sentence-transformers==5.7.0"

Those package releases are current 2026 versions: PyTorch 2.13.0 and torchvision’s matching release landed in July 2026, Sentence Transformers 5.7.0 in August 2026, and Transformers 5.15.0 in August 2026.

Now embed three descriptions:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "ibm-granite/granite-embedding-97m-multilingual-r2"
)

descriptions = [
    "Compact wireless keyboard with Bluetooth connectivity",
    "Mechanical gaming keyboard with RGB backlighting",
    "Portable Bluetooth keyboard for tablets and phones",
]

embeddings = model.encode(
    descriptions,
    normalize_embeddings=True,
)

similarities = model.similarity(
    embeddings,
    embeddings,
)

print(embeddings.shape)
print(similarities)

The representation step is now reusable.

You can:

• cache the embeddings;

• index them for retrieval;

• compute similarity;

• cluster them;

• train logistic regression over them;

• feed them into a lightweight downstream ranking model.

This can drastically shorten the experimentation cycle because you do not need to rerun a complete transformer training job for every classifier change.

When to move from embeddings to a fine-tuned transformer

A single embedding intentionally compresses a sequence.

That is useful until your label depends on distinctions that the generic embedding space does not preserve well enough.

Fine-tuning becomes more compelling when:

• subtle phrasing changes the class;

• domain terminology is highly specialized;

• labels depend on interactions throughout a long passage;

• you need token-level classification;

• sequence structure is itself part of the task;

• the embedding-plus-simple-model baseline has clearly plateaued.

Current Transformers tooling continues to center task adaptation around pretrained models rather than training general language representations from random initialization.

Before spending that compute, benchmark progressively:

1. TF-IDF plus logistic regression.

2. Current pretrained embeddings plus a linear classifier.

3. Fine-tuned pretrained transformer.

4. Larger models or ensembles only when the measured gain matters.

The transformer should earn its complexity.

Images: transfer learning changes the economics of small datasets

The same principle applies to computer vision.

Suppose you need to classify product images into:

• furniture;

• electronics;

• kitchen;

• clothing.

If you have only a few thousand labeled examples, randomly initializing a large image network throws away an enormous amount of available pretrained structure.

Start with pretrained weights.

For a current reproducible PyTorch environment:

python -m pip install "torch==2.13.0" "torchvision==0.28.0"

PyTorch 2.13.0 and torchvision 0.28.0 were released together in July 2026.

A minimal pretrained ResNet setup is:

from torch import nn
from torchvision.models import ResNet50_Weights, resnet50

weights = ResNet50_Weights.DEFAULT

model = resnet50(
    weights=weights,
)

in_features = model.fc.in_features

model.fc = nn.Linear(
    in_features,
    4,
)

preprocess = weights.transforms()

model.train()

Notice the line:

preprocess = weights.transforms()

Use the preprocessing associated with the pretrained weights rather than casually inventing resize, crop, and normalization values. Torchvision’s current weight API exposes the corresponding transforms alongside pretrained model weights.

Then benchmark against a pretrained Vision Transformer or another modern backbone under the same:

• train/validation split;

• augmentation policy;

• input resolution;

• metric;

• hardware budget.

Torchvision 0.28’s model catalog includes both convolutional and transformer-based vision architectures, so this comparison no longer requires assembling unrelated ecosystems just to test the two families.

CNN versus ViT is not the real decision

“Which architecture is newer?” is rarely the question your production system cares about.

Measure:

• validation quality;

• p50 and p95 inference latency;

• throughput at your expected batch size;

• peak memory;

• accelerator requirement;

• model artifact size;

• sensitivity to image resolution;

• retraining cost.

A slightly more accurate model may be a bad choice if it forces a GPU into a previously CPU-only serving path.

Conversely, if predictions run offline once per day, a larger model may be almost free operationally.

Model selection is architecture plus execution context.

Which boosted-tree implementation should you try in 2026?

If gradient boosting wins a place in the experiment, the next choice is implementation.

Current release levels as of August 18, 2026 include scikit-learn 1.9.0, XGBoost 3.4.1, LightGBM 4.7.0, and CatBoost 1.2.10. XGBoost 3.4.1 was released on August 14, 2026; LightGBM 4.7.0 in July 2026; and CatBoost 1.2.10 in February 2026.

If you want to reproduce a comparison using those versions:

python -m pip install "scikit-learn==1.9.0" "xgboost==3.4.1" "lightgbm==4.7.0" "catboost==1.2.10"

Start with scikit-learn histogram boosting when

• scikit-learn is already your main ML dependency;

• you want a compact stack;

• you want native missing-value support;

• you want current categorical-feature support without immediately adding another boosting library;

• monotonic or interaction constraints are relevant.

Those capabilities are documented in the current histogram gradient-boosting API.

Add XGBoost when

• your organization already operates XGBoost;

• its training or deployment ecosystem fits your platform;

• GPU acceleration is useful;

• you want its objectives and configuration surface;

• native categorical handling fits your feature representation.

Current XGBoost documentation continues to support histogram tree construction and categorical data in supported configurations.

Add LightGBM when

• efficient boosting on large structured datasets matters;

• native categorical splitting fits the feature pipeline;

• its performance characteristics fit your training environment.

Current LightGBM documentation continues to document direct categorical-feature handling rather than requiring universal one-hot expansion.

Add CatBoost when

• categorical variables dominate the problem;

• you want a mature boosted-tree implementation built with categorical processing as a central use case;

• your team already understands its modeling and serving workflow.

Then measure all of them under comparable tuning budgets.

Do not decide because one library dominated somebody’s competition notebook years ago.

Interpretability changes what “best” means

If predictions influence:

• pricing;

• eligibility;

• prioritization;

• resource allocation;

• manual-review queues;

• consequential business rules;

you may have objectives that are not captured by AUC or RMSE.

You may need to know:

• whether a human can reproduce a decision;

• whether the direction of important relationships matches policy;

• whether monotonic behavior is required;

• whether explanations remain stable;

• whether a model can be audited without a specialized GPU stack.

A linear model is relatively transparent because its prediction function is explicit.

That does not mean each coefficient is automatically a causal explanation. Correlated variables, preprocessing, interactions, and feature scaling still affect interpretation.

A shallow decision tree can be inspected path by path.

A 500-tree ensemble is different. Feature attribution can be useful, but a post-hoc explanation of a complicated predictor is not equivalent to an intrinsically simple decision rule.

Current scikit-learn histogram boosting also supports monotonic and interaction constraints, which can be valuable when unconstrained relationships would violate known domain requirements.

If interpretability is mandatory, include it in the optimization objective from the beginning.

Metric choice can reverse the winner

“Best accuracy” is not a complete model-selection criterion.

For binary classification:

• use ROC AUC when ranking across thresholds is relevant;

• inspect precision-recall behavior when positives are rare;

• use log loss when probability quality matters;

• tune an operating threshold when false positives and false negatives have different costs.

For regression:

• MAE rewards absolute error linearly;

• RMSE penalizes large misses more heavily;

• quantile objectives help when you need conditional ranges or asymmetric decisions.

Then connect those technical metrics to the actual downstream action.

A fraud-like ranking system that investigates only the top 0.5% of scores cares about a different region of model behavior than an application that turns every probability into an expected-value calculation.

A model cannot be “best” independently of the decision rule that consumes it.

Give competing models comparable optimization budgets

Another easy way to manufacture a misleading conclusion is:

• spend two days tuning one model;

• compare it against another model’s defaults;

• declare the first family superior.

You have compared engineering effort, not just model families.

The original TabArena work made this particularly visible: model rankings changed with training budget, ensembling, and evaluation design.

Decide what experiment you are running.

You might ask:

Which model performs best after ten minutes on one CPU machine?

Or:

Which system produces the best score after a four-hour GPU budget and automated ensembling?

Both are valid.

They are different questions.

Cherry on the cake: the tabular benchmark crown flipped remarkably fast

The most interesting model-selection story of the last year is not that “trees beat neural networks.”

It is how quickly that storyline stopped being current.

In June 2025, the original TabArena paper presented a nuanced picture: gradient-boosted trees remained strong competitors, deep models became more competitive with greater training budgets and ensembling, and foundation models were particularly strong on smaller datasets.

Then the frontier moved.

In November 2025, TabPFN-2.5 reported substantially outperforming tuned tree-based models in its evaluated TabArena settings and competing with sophisticated AutoML systems.

In February 2026, the TabICLv2 paper reported another step forward and explicitly framed tabular foundation models as having moved ahead of gradient-boosted trees at the top of predictive benchmarks.

In May 2026, the TabPFN-3 technical report reported that a single forward-pass configuration outperformed other models in its standard TabArena evaluation and extended the approach toward substantially larger tabular problem sizes.

That is the surprising fact worth carrying into production work:

A benchmark narrative that was reasonable in June 2025 was materially outdated before mid-2026.

Yet gradient boosting did not become useless during those eleven months.

It remains valuable because benchmark leadership and engineering usefulness are different properties.

A boosted-tree model can still be the rational production winner when it offers:

• sufficient predictive quality;

• CPU-friendly serving;

• fast retraining;

• predictable operational behavior;

• easy integration;

• familiar debugging;

• acceptable interpretability;

• simpler security and dependency review.

Meanwhile, a tabular foundation model may be the rational winner when the additional predictive gain has enough value to justify its serving, hardware, licensing, or integration requirements.

This is the larger lesson:

Build replaceable evaluation pipelines, not identities around model families.

The benchmark crown will move again.

Avoid the six most common model-selection mistakes

Mistake 1: starting with the fanciest architecture

A difficult model is not automatically a strong baseline.

Begin with something that tells you how much signal is available cheaply.

Mistake 2: treating “good first benchmark” as “current state of the art”

This is especially dangerous in tabular ML in 2026.

Boosted trees remain excellent baselines, but current foundation models have taken benchmark leadership in important evaluations.

Use the right wording because it leads to the right experiment.

Mistake 3: leaking information across validation folds

Watch for:

• normalization fitted before splitting;

• feature selection using all targets;

• target encoding performed globally;

• future data entering historical training;

• the same customer, store, or device appearing on both sides when production requires unseen-group generalization.

No architecture compensates for a dishonest evaluation.

Mistake 4: comparing unequal tuning effort

Give candidates comparable budgets or explicitly label the comparison.

“Best after defaults” and “best after an AutoML search” are different results.

Mistake 5: ignoring inference constraints

Track:

• serialized size;

• cold-start time;

• p50 latency;

• p95 or p99 latency;

• batch throughput;

• CPU versus accelerator requirement;

• peak memory;

• retraining duration.

A 0.2-point metric gain can be either priceless or worthless depending on the application.

Mistake 6: skipping the linear baseline

The linear model is a diagnostic instrument.

When it performs nearly as well as a complicated alternative, it tells you the problem may be mostly additive—or that your current feature representation has already done much of the hard work.

That knowledge should affect the production decision.

A practical experiment ladder

The point of an experiment ladder is to spend complexity only after simpler candidates expose what the problem needs.

For tabular classification

Run approximately:

1. Dummy baseline.

2. Regularized logistic regression.

3. Histogram gradient boosting.

4. XGBoost, LightGBM, or CatBoost when another implementation is justified.

5. Current tabular foundation model when accuracy and deployment constraints make it viable.

6. Ensemble only if the incremental gain has real value.

Notice that the foundation model is no longer buried at the end as a curiosity. In an accuracy-critical 2026 project, it may deserve to move earlier.

For text classification

Run:

1. Majority or rule baseline.

2. TF-IDF plus logistic regression.

3. Current pretrained embeddings plus a linear classifier.

4. Fine-tuned pretrained transformer.

5. Larger model or ensemble only if the gain survives cost analysis.

For text retrieval or semantic matching

Run:

1. Lexical retrieval baseline.

2. Current embedding model.

3. Fine-tuned or task-adapted retriever if error analysis shows a need.

4. Reranking only when the latency-quality tradeoff supports it.

For image classification

Run:

1. Operational heuristic if one exists.

2. Frozen pretrained backbone plus a new head.

3. Partial fine-tuning.

4. Full fine-tuning of a pretrained CNN.

5. Full fine-tuning of a pretrained ViT or another current backbone.

6. Larger model only when the serving budget allows it.

For tiny structured datasets

Run:

1. Domain rules.

2. Regularized linear model.

3. Conservative boosted trees.

4. Current tabular foundation model where feasible.

Then stop increasing capacity when validation uncertainty is larger than the improvement you are trying to sell.

The model-selection worksheet

Before approving a model, write down the answers.

Data

• What is the fundamental modality?

• How many independent labeled examples exist?

• Which features are genuinely available at prediction time?

• Are there repeated entities?

• Is there chronological structure?

• Is the label noisy?

• Are important categories likely to appear unseen?

• Is distribution shift expected?

Evaluation

• Which metric maps to the downstream decision?

• What real-world scenario does the validation split simulate?

• What variation exists across folds or repeated runs?

• Are competing models receiving comparable optimization budgets?

• Is there an untouched final test set?

• Has every preprocessing step been fitted inside the training partition?

Operations

• What is the p95 latency budget?

• Is GPU or other accelerator hardware available?

• How often will the model retrain?

• What is the acceptable model size?

• How expensive is a cold start?

• How does the model behave on unseen categories or unusual inputs?

• What happens if the preferred accelerator is unavailable?

Governance

• Is intrinsic interpretability required?

• Are post-hoc explanations sufficient?

• Must some relationships be monotonic?

• Which features are prohibited?

• Must a human reproduce an individual decision?

• What drift and quality monitoring will run after deployment?

A model is not production-ready merely because it won one cross-validation column.

The playbook in one page

When you face a new supervised-learning problem, use these defaults.

Tabular: start with a regularized linear baseline and histogram gradient boosting. Treat boosting as an excellent engineering baseline, not as the universal 2026 benchmark champion. Add a current tabular foundation model early when maximum accuracy makes its additional requirements worthwhile.

Text retrieval, similarity, and clustering: start with a current pretrained embedding model and a simple similarity, retrieval, clustering, or downstream classification layer.

Text understanding and sequence-sensitive classification: establish the embedding or sparse baseline, then fine-tune a pretrained transformer when the task requires richer sequence-level adaptation.

Images: use transfer learning. Benchmark pretrained CNN and ViT-family candidates under the same evaluation and deployment constraints.

Tiny labeled data: begin with regularization and low-variance baselines. For tiny tabular data specifically, also test a current foundation model if feasible because recent benchmark evidence is unusually strong in this regime.

Interpretability-critical systems: prefer transparent linear or tree-based models early, and distinguish intrinsic transparency from post-hoc explanation.

Time-dependent data: validate chronologically.

Repeated entities: validate by group.

Every problem: compare complete, leakage-resistant pipelines—not isolated estimators.

The deeper rule is simple:

Choose model families whose inductive biases match the structure in your data, evaluate them under the split that best imitates production, and make every extra layer of complexity earn its operational cost.

Take the dataset you are working on today and identify its branch in this playbook. Build the simplest credible baseline and the strongest modality-appropriate baseline, run both under a leakage-resistant validation scheme, and only then spend time tuning, scaling, or ensembling. If it is tabular, make that experiment a three-way conversation between linear structure, boosted nonlinearities, and the current foundation-model frontier.