Hyperparameter search done right: GridSearchCV, RandomizedSearchCV, and Optuna with CV built in

Hyperparameter tuning is easy to get started with and surprisingly easy to get wrong.

You define a search space, run a search algorithm, collect the best parameters, and celebrate a higher cross-validation score. But a good-looking score does not necessarily mean you have found a better model.

Common problems include:

• Comparing search algorithms with different computational budgets.

• Using one validation split and accidentally optimizing for its quirks.

• Searching arbitrary parameter ranges that exclude good configurations.

• Reporting the best cross-validation score as though it were an unbiased test result.

• Running dozens of expensive experiments without establishing a random-search baseline.

• Letting preprocessing or feature selection leak information across CV folds.

Three tools cover most practical hyperparameter optimization workflows:

1. GridSearchCV exhaustively evaluates a predefined set of parameter combinations.

2. RandomizedSearchCV samples configurations from specified distributions.

3. Optuna with TPE uses previous trial results to guide its search toward promising configurations.

The right choice depends on your search space, evaluation budget, and need for flexibility—not simply which optimizer sounds more sophisticated.

In this tutorial, we’ll build an executable comparison using one regression model, one dataset, the same cross-validation folds, and exactly the same number of hyperparameter trials for each method.

We’ll also examine why the winning cross-validation score can be misleading, how to use nested CV when evaluation rigor matters, and why random search can outperform a carefully constructed grid.

1. Hyperparameter optimization is an experiment, not a guessing contest

A model has two kinds of parameters.

Learned parameters are estimated during training. For a decision tree, these include split features, thresholds, and leaf predictions.

Hyperparameters control the training process or constrain the model. Examples include tree depth, learning rate, regularization strength, and minimum samples per leaf.

For each proposed hyperparameter configuration, the optimization process needs an estimate of how well the resulting model generalizes.

That estimate should usually come from cross-validation rather than training performance.

What cross-validation actually does

In K-fold cross-validation, the development dataset is divided into K folds. Each configuration is trained K times, with a different fold held out for validation each time.

For four-fold CV:

Four-fold cross-validation

Run 1

Validate

Train

Train

Train

Run 2

Train

Validate

Train

Train

Run 3

Train

Train

Validate

Train

Run 4

Train

Train

Train

Validate

Training folds

Validation fold

The four validation scores are averaged to produce a score for that configuration.

The search algorithm then compares this score with those of other configurations.

This arrangement reduces dependence on a single train-validation split. It does not, however, eliminate all model-selection bias.

There is an important distinction:

• Inner validation: Which hyperparameters should we choose?

• Independent evaluation: How well does the complete tuning procedure generalize to data it has not influenced?

We’ll keep those jobs separate throughout the experiment.

2. Designing a fair comparison

A useful benchmark must control the factors that could otherwise explain differences between search methods.

We’ll use the following setup:

The model is a histogram-based gradient-boosted tree regressor from scikit-learn. It supports several useful hyperparameters and is efficient enough for a practical local experiment.

We will tune three of them:

• learning_rate controls the contribution of each boosting iteration. Smaller values generally require more iterations to achieve a comparable fit.

• max_leaf_nodes limits the complexity of individual trees.

• min_samples_leaf controls the minimum number of training observations in a leaf, influencing how finely the model can partition the data.

Other settings, including the number of boosting iterations, remain fixed.

Defining the budget

Grid search will examine:

That produces:

18 candidates = 3 × 3 × 2

Random search and Optuna will explore the same numerical boundaries, but with finer-grained choices inside them.

Every method receives 18 candidate evaluations and four validation folds.

Therefore:

72 CV fits per method = 18 candidates × 4 folds

Across the three approaches, the tuning stage requires 216 model fits. After comparing CV performance, we’ll train one final model and evaluate it on the test set.

This is an equal-trial comparison, not an identical-search-space comparison. The grid samples a fixed subset, while random search and TPE can explore additional values between the grid points.

That distinction is part of what we’re testing.

3. Environment setup

Use a modern Python installation, preferably Python 3.11 or newer, and create an isolated environment.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade numpy scipy scikit-learn optuna

We’ll use:

• NumPy to generate a reproducible synthetic dataset.

• SciPy to provide search distributions.

• scikit-learn for the estimator, CV, search utilities, and metrics.

• Optuna for adaptive optimization using the Tree-structured Parzen Estimator (TPE).

The example uses current public APIs documented in the scikit-learn model-selection reference and Optuna documentation.

There is no external dataset download, account, or hosted service requirement.

4. Complete runnable benchmark: grid vs. random vs. Optuna

Create a file named hyperparameter_compare.py.

The entire program below can be run as written. It generates the dataset, constructs the searches, performs CV, prints the comparison, and evaluates the selected model.

from time import perf_counter

import numpy as np
import optuna
from scipy.stats import loguniform, randint

from sklearn.base import clone
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import root_mean_squared_error
from sklearn.model_selection import (
    GridSearchCV,
    KFold,
    RandomizedSearchCV,
    cross_val_score,
    train_test_split,
)

SEED = 42
BUDGET = 18
SCORING = "neg_root_mean_squared_error"

# --------------------------------------------------
# 1. Generate synthetic energy-demand observations
# --------------------------------------------------

rng = np.random.default_rng(2026)

X = rng.normal(size=(1200, 8))

y = (
    100
    + 17 * X[:, 0]
    - 9 * X[:, 1]
    + 14 * X[:, 2] * X[:, 3]
    + 6 * np.sin(2 * X[:, 4])
    + 4 * (X[:, 5] > 0)
    + rng.normal(0, 4, size=len(X))
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=SEED,
)

# --------------------------------------------------
# 2. Shared estimator and CV configuration
# --------------------------------------------------

cv = KFold(
    n_splits=4,
    shuffle=True,
    random_state=SEED,
)

base_model = HistGradientBoostingRegressor(
    max_iter=100,
    early_stopping=False,
    random_state=SEED,
)

# Exactly 3 x 3 x 2 = 18 grid configurations.
param_grid = {
    "learning_rate": [0.02, 0.06, 0.20],
    "max_leaf_nodes": [7, 15, 31],
    "min_samples_leaf": [5, 20],
}

# Same bounds, finer-grained search possibilities.
param_distributions = {
    "learning_rate": loguniform(0.02, 0.20),
    "max_leaf_nodes": randint(7, 32),
    "min_samples_leaf": randint(5, 31),
}

results = {}

def record(name, score, params, seconds):
    results[name] = {
        "cv_rmse": float(score),
        "params": params,
        "seconds": seconds,
    }

# --------------------------------------------------
# 3. GridSearchCV
# --------------------------------------------------

start = perf_counter()

grid = GridSearchCV(
    estimator=base_model,
    param_grid=param_grid,
    scoring=SCORING,
    cv=cv,
    n_jobs=1,
    refit=False,
    error_score="raise",
)

grid.fit(X_train, y_train)

record(
    "Grid",
    -grid.best_score_,
    grid.best_params_,
    perf_counter() - start,
)

# --------------------------------------------------
# 4. RandomizedSearchCV
# --------------------------------------------------

start = perf_counter()

random_search = RandomizedSearchCV(
    estimator=base_model,
    param_distributions=param_distributions,
    n_iter=BUDGET,
    scoring=SCORING,
    cv=cv,
    random_state=SEED,
    n_jobs=1,
    refit=False,
    error_score="raise",
)

random_search.fit(X_train, y_train)

record(
    "Random",
    -random_search.best_score_,
    random_search.best_params_,
    perf_counter() - start,
)

# --------------------------------------------------
# 5. Optuna with TPE
# --------------------------------------------------

optuna.logging.set_verbosity(optuna.logging.WARNING)

sampler = optuna.samplers.TPESampler(
    seed=SEED,
    n_startup_trials=5,
)

study = optuna.create_study(
    direction="minimize",
    sampler=sampler,
)

def objective(trial):
    params = {
        "learning_rate": trial.suggest_float(
            "learning_rate",
            0.02,
            0.20,
            log=True,
        ),
        "max_leaf_nodes": trial.suggest_int(
            "max_leaf_nodes",
            7,
            31,
        ),
        "min_samples_leaf": trial.suggest_int(
            "min_samples_leaf",
            5,
            30,
        ),
    }

    candidate = clone(base_model).set_params(**params)

    fold_scores = cross_val_score(
        candidate,
        X_train,
        y_train,
        scoring=SCORING,
        cv=cv,
        n_jobs=1,
    )

    return -float(fold_scores.mean())

start = perf_counter()

study.optimize(
    objective,
    n_trials=BUDGET,
    n_jobs=1,
)

record(
    "Optuna TPE",
    study.best_value,
    study.best_params,
    perf_counter() - start,
)

# --------------------------------------------------
# 6. Compare the tuning results
# --------------------------------------------------

print(
    f"{len(X_train)} training / "
    f"{len(X_test)} held-out examples"
)

print(
    f"{BUDGET} candidates x {cv.get_n_splits()} folds = "
    f"{BUDGET * cv.get_n_splits()} CV fits per method"
)

print(
    f"{'Method':<12} "
    f"{'Best CV RMSE':>14} "
    f"{'Time (s)':>12}"
)

for name, item in sorted(
    results.items(),
    key=lambda pair: pair[1]["cv_rmse"],
):
    print(
        f"{name:<12} "
        f"{item['cv_rmse']:>14.3f} "
        f"{item['seconds']:>12.1f}"
    )
    print(f"  Parameters: {item['params']}")

# --------------------------------------------------
# 7. Fit the CV-selected winner and test once
# --------------------------------------------------

winner = min(
    results,
    key=lambda name: results[name]["cv_rmse"],
)

final_model = clone(base_model).set_params(
    **results[winner]["params"]
)

final_model.fit(X_train, y_train)

predictions = final_model.predict(X_test)

print(f"\nSelected by CV: {winner}")

print(
    "Untouched test RMSE: "
    f"{root_mean_squared_error(y_test, predictions):.3f}"
)

Run it:

python hyperparameter_compare.py

Understanding the code

Several implementation details matter more than they might initially appear.

The test set is created before any tuning. Every configuration is evaluated using only X_train and y_train. The holdout remains untouched until the final evaluation.

Each method receives identical CV partitions. The seeded K-fold splitter makes the data partitions consistent across methods. This removes an unnecessary source of variation.

The estimator is the same throughout. We are comparing optimization methods, not changing models.

All methods optimize RMSE. In scikit-learn, the scorer named neg_root_mean_squared_error returns the negative RMSE because its model-selection interfaces maximize scores. We negate the result before reporting it, so smaller displayed values are better.

We set refit=False for both scikit-learn searches. This prevents an automatic final training run for each search method. Instead, we select one winner and refit it explicitly.

Optuna runs CV inside the objective function. Optuna proposes parameter values, but the objective controls how those values are evaluated. Without the cross_val_score call, Optuna would not automatically perform cross-validation.

We explicitly set n_jobs=1. This makes the benchmark simple and avoids concurrent search-level fitting. Each estimator may still use lower-level numerical parallelism, depending on its implementation and environment.

Finally, error_score="raise" makes failed grid or random-search configurations visible instead of silently replacing their results with a fallback score.

That is usually preferable when developing and debugging a new search workflow.

5. Reading the benchmark results

One execution of the script produced the following results:

Example run · lower RMSE is better

The CV-selected grid-search model achieved a held-out test RMSE of approximately 7.982.

(Measured with scikit-learn 1.9.1, SciPy 1.18.1 and Optuna 5.0.0; the run is seeded and reproduces exactly.)

These numbers come from the seeded synthetic experiment, not from a general benchmark of optimizer quality. Results can vary slightly with software versions and execution environments, and differences of this size should not be overinterpreted.

The particularly interesting outcome is that grid search won this run.

That does not contradict the advantages of random search or Optuna.

Grid search happened to evaluate a strong configuration near the boundary of our search space. Its selected parameters were:

learning_rate:    0.20
max_leaf_nodes:   7
min_samples_leaf: 20

Optuna found a nearby configuration with a slightly lower learning rate and different tree-size settings.

With only 18 evaluations, there was limited opportunity for the adaptive sampler to learn the relationships between parameters and validation performance.

This leads to a fundamental lesson:

A better optimization strategy does not guarantee a better result on every dataset, random seed, or small evaluation budget.

A serious comparison would repeat the experiment across multiple training splits and optimization seeds, summarize the variation, and account for the additional selection decisions.

For day-to-day model development, however, a controlled comparison like this is an excellent starting point.

6. When GridSearchCV is the right tool

Grid search is the simplest of the three approaches conceptually.

You choose a finite set of values for each parameter, and it evaluates their Cartesian product.

Its greatest advantage is completeness within the specified grid.

If your grid contains 18 valid combinations, grid search evaluates all 18.

Where it excels

Grid search is particularly useful when:

• You have only one or two important hyperparameters.

• Each parameter has a small, meaningful set of possible values.

• You want reproducible coverage of every listed combination.

• Each training run is inexpensive.

• You need transparent results that are easy to audit.

For example, imagine a decision tree with three plausible maximum depths and three minimum leaf sizes.

Nine combinations are small enough that exhaustive evaluation may be the simplest sensible choice.

Grid search also makes it easy to inspect a parameter-performance table, because the evaluated configurations follow an intentional structure.

The curse of dimensionality

The problem appears when your grid grows.

Suppose you have six hyperparameters, each with five candidate values.

The number of combinations becomes:

5 × 5 × 5 × 5 × 5 × 5 = 15,625

With five-fold CV, that requires 78,125 model fits, excluding refitting.

If each fit takes two seconds, the serial compute time already exceeds 43 hours.

Adding just one more parameter with five values multiplies that cost by another factor of five.

This is why exhaustive search is generally not a good default for a large, continuous hyperparameter space.

Why coarse grids can miss good configurations

Imagine the best learning rate is approximately 0.13.

A grid containing only 0.02, 0.06, and 0.20 cannot evaluate it directly.

You could construct a finer grid, but that increases the number of combinations.

Grid search offers coverage of its specified points, not guaranteed discovery of the optimum between them.

This limitation becomes especially important when some parameters are highly sensitive to small changes.

7. Why RandomizedSearchCV is a strong baseline

Randomized search takes a different approach.

Instead of evaluating every combination in a predetermined grid, it samples parameter configurations from specified distributions.

The number of evaluations is controlled directly by n_iter.

In our benchmark, that value is 18.

This makes search cost easier to control when the parameter space becomes large.

Distributions matter more than most beginners expect

Consider a learning rate between 0.001 and 0.1.

Sampling uniformly in ordinary numeric space gives much greater emphasis to the high end of the range than to the tiny values near 0.001.

For scale-sensitive parameters, that may not be what you want.

A log-uniform distribution instead allocates comparable sampling probability to multiplicative intervals.

For example:

• 0.001–0.01

• 0.01–0.1

Each interval spans a factor of ten.

In our experiment, SciPy’s loguniform provides log-scale sampling for learning_rate, while randint samples integer tree parameters.

One easy-to-miss implementation detail is that randint(low, high) uses an exclusive upper bound. Our Optuna integer search, in contrast, uses an inclusive upper bound.

That’s why the code uses 32 as the upper argument to SciPy when we want to include 31.

Random search spends less effort on irrelevant dimensions

This is its most important conceptual advantage over grid search.

Suppose you can evaluate only 25 configurations and have two hyperparameters.

With a 5 × 5 grid, you evaluate five distinct values of each parameter.

But imagine that only the first hyperparameter materially influences performance.

The grid spends 25 evaluations covering just five values of the parameter that matters.

A continuous random search can explore approximately 25 distinct values of that important parameter within the same 25 trials.

It does not need to know which parameter matters beforehand.

This intuition is central to one of the most influential papers on hyperparameter optimization.

8. Cherry on the cake: the research result that made random search famous

In 2012, James Bergstra and Yoshua Bengio published Random Search for Hyper-Parameter Optimization in the Journal of Machine Learning Research.

The surprising result was not merely that random search could occasionally find good settings.

Their research showed that randomly sampled configurations could be substantially more efficient than grid search for the problems they studied.

They identified an important phenomenon: typically only a small subset of hyperparameters matters strongly for a given dataset, and that subset can change between datasets.

Consider the following conceptual example.

Grid search

25 trials, only 5 unique x-valuesRandom search

25 trials, broader x coverageIllustrative 2D search spaces with two continuous hyperparameters. The horizontal parameter is assumed to matter most.A grid wastes evaluations varying a parameter that has little influence on performance.

Random search spreads evaluations across more distinct values of both parameters.

The research did not prove that random search wins universally. Its strength is that it is often a remarkably effective baseline when you do not know which dimensions deserve attention.

This finding still surprises practitioners who assume that systematic enumeration must necessarily outperform randomness.

The practical takeaway is simple: before implementing an elaborate optimization algorithm, establish a credible random-search baseline.

If an adaptive optimizer cannot reliably improve upon that baseline within your available compute budget, its added complexity may not be justified.

9. How Optuna’s TPE search works

Optuna adds an adaptive decision-making process.

Instead of drawing every trial independently, it can use completed trials to influence later proposals.

Our benchmark uses Optuna’s TPESampler, based on the Tree-structured Parzen Estimator approach.

The basic intuition

TPE begins by observing parameter configurations and their corresponding objective values.

It builds probabilistic models of parameter values associated with relatively good and less-good outcomes.

It then favors values that appear more promising according to those models.

A simplified description is:

1. Evaluate initial configurations.

2. Separate observations by objective quality.

3. Model the parameter distributions associated with better and worse outcomes.

4. Propose a new configuration with a favorable probability-density relationship.

5. Evaluate it using cross-validation.

6. Update the models and repeat.

This is different from fitting a predictive regression model of RMSE directly over the entire hyperparameter space.

TPE models distributions of promising parameter values and uses them to select candidates.

Optuna’s TPE sampler documentation describes the underlying method and its configurable behavior.

Why startup trials are important

In our implementation, we set n_startup_trials=5.

That means the first five completed trials provide an initial set of observations through random sampling.

With a total budget of 18 trials, the remaining 13 can use the TPE-guided sampling process.

This is a small adaptive-search budget.

For many real workloads, particularly spaces with numerous interacting parameters, substantially more trials may be necessary.

Optuna’s search-space API

The key methods in the objective function are:

• trial.suggest_float(...) for floating-point parameters.

• trial.suggest_float(..., log=True) for logarithmically scaled floating-point parameters.

• trial.suggest_int(...) for integer parameters.

• trial.suggest_categorical(...) for finite categorical choices.

The modern suggest_float interface also avoids relying on older deprecated suggestion methods such as suggest_loguniform.

Another advantage is that search spaces can be defined with ordinary Python control flow.

For example, when optimizing a model that supports several algorithms, you can conditionally search parameters that are relevant only to the algorithm selected in that trial.

This flexibility becomes valuable for conditional search spaces that are awkward to express as one rectangular grid.

Optuna does not automatically guarantee improvement

TPE is an optimization strategy, not an oracle.

It can struggle when:

• The evaluation budget is extremely small.

• CV scores are noisy.

• The search space includes many irrelevant parameters.

• Important configurations fall outside the specified bounds.

• Good performance depends on interactions that are difficult to infer from limited trials.

It also has overhead from fitting and updating its sampling models.

For inexpensive estimators, that overhead may matter. For configurations that take minutes or hours to train, the additional sampling cost is typically much less important than the cost of model fitting.

And remember: Optuna does not automatically add cross-validation. CV is present in our example because we explicitly included it in the objective.

10. The hidden problem: the best CV score is biased

This is one of the most important points in the entire tutorial.

Imagine evaluating 500 configurations.

Each CV score contains some sampling variation.

Even if two configurations have similar true generalization performance, one may appear better because it happens to perform unusually well on those validation folds.

When you select the highest-scoring configuration from hundreds of attempts, you may select a configuration that benefited from favorable validation noise.

The more aggressively you optimize against the same validation folds, the more opportunity there is to overfit the selection procedure to those folds.

The best CV score is therefore useful for choosing hyperparameters, but it is not necessarily an unbiased estimate of future generalization performance.

Why the held-out test set matters

Our benchmark addresses this by isolating the test set before tuning.

All three searches use only the training portion.

We then choose the winning configuration using CV performance, fit it on the full training portion, and evaluate it on the untouched test portion.

That final evaluation estimates the performance of the model-selection procedure used in this run.

There is an important condition: once you inspect the test result, do not use it to revise the search space, select another method, or tune the model further while continuing to claim that the same test set is independent.

Repeated test-set-driven adjustments turn the test set into another validation set.

When nested cross-validation becomes appropriate

If your dataset is small, if you are publishing a performance comparison, or if you need a careful estimate of a tuning procedure’s generalization performance, consider nested CV.

Nested CV uses two levels of cross-validation:

• Inner CV selects hyperparameters.

• Outer CV evaluates the tuning procedure on observations that were excluded from its inner optimization.

Nested cross-validation

Outer fold

Outer training data

Outer test

Inside outer training data

Inner train

Inner validation

Search candidate hyperparameters using inner folds, then refit the chosen configuration on the entire outer-training partition.Evaluate the refitted model on the untouched outer test fold

Repeat across outer folds and average the outer-fold results.This design evaluates the search procedure, not just its best observed inner-CV score.

The scikit-learn nested CV example explains why non-nested model-selection estimates can be optimistic.

A runnable nested-CV example

To evaluate the grid-search procedure with nested CV, append the following code to the benchmark script.

It reuses the estimator, grid, training data, and scoring metric already defined.

# Additional experiment: nested CV for grid search.

outer_cv = KFold(
    n_splits=3,
    shuffle=True,
    random_state=7,
)

inner_cv = KFold(
    n_splits=4,
    shuffle=True,
    random_state=42,
)

nested_search = GridSearchCV(
    estimator=clone(base_model),
    param_grid=param_grid,
    scoring=SCORING,
    cv=inner_cv,
    n_jobs=1,
    refit=True,
    error_score="raise",
)

outer_scores = cross_val_score(
    nested_search,
    X_train,
    y_train,
    scoring=SCORING,
    cv=outer_cv,
    n_jobs=1,
)

outer_rmse = -outer_scores

print("\nNested CV: grid-search procedure")
print(f"Outer-fold RMSE: {outer_rmse.round(3)}")
print(f"Mean outer RMSE: {outer_rmse.mean():.3f}")
print(f"Outer-fold SD: {outer_rmse.std():.3f}")

The crucial detail is refit=True.

Within each outer fold, grid search chooses hyperparameters using inner CV. It then trains a model using the selected parameters on the full outer-training partition.

That fitted model is evaluated on the outer test fold.

With 18 candidates, four inner folds, and three outer folds, this extra example performs 216 inner-CV fits plus three outer-fold refits.

Nested CV costs more because it repeats the search.

Also, this particular extension evaluates only the grid-search procedure. To compare all three search procedures rigorously with nested CV, each procedure should be rerun independently within each outer fold.

If you then select a method using its outer scores, don’t present the winning outer score as completely free of method-selection bias. An independent final test set or a more comprehensive evaluation protocol is still valuable.

11. Cross-validation mistakes that invalidate good-looking search results

Even correctly implemented search APIs cannot protect you against a badly designed experiment.

Several issues deserve attention before increasing the number of trials.

Mistake 1: preprocessing before splitting

Suppose you standardize features, perform feature selection, or impute missing values using the entire dataset before cross-validation.

Information about the validation observations can influence those preprocessing steps.

This introduces leakage.

For operations that learn from data, the usual solution is a scikit-learn Pipeline containing preprocessing and the estimator.

The pipeline is fitted separately within each training fold, ensuring that learned preprocessing statistics are based only on that fold’s training data.

Our histogram-based tree model does not require feature scaling for this example, so the benchmark avoids unnecessary preprocessing.

On real datasets, especially when scaling, encoding, imputation, or feature selection is involved, putting those steps inside CV is essential.

Mistake 2: applying ordinary K-fold CV to time series

Our synthetic observations are independent draws, so shuffled K-fold is a reasonable demonstration.

Real energy-demand data often has timestamps, seasonality, trends, and serial dependence.

If you randomly shuffle historical and future records, you may evaluate an unrealistic forecasting scenario.

Use a time-aware validation procedure, such as scikit-learn’s TimeSeriesSplit, when the real deployment task involves predicting future observations from earlier ones.

You may also need a temporal gap to prevent leakage through lagged or overlapping features.

The splitting strategy should reflect what information would actually be available when predictions are made.

Mistake 3: allowing the same entity into training and validation

Suppose you have multiple observations from the same building, store, device, or customer.

Random CV may put some records from an entity into training and other records from that same entity into validation.

If the deployment requirement is to generalize to entirely new entities, this can produce misleadingly optimistic results.

A group-aware approach, such as GroupKFold, can keep entities separated across training and validation partitions.

The correct splitting method depends on the prediction question—not simply the estimator being used.

Mistake 4: changing the metric mid-experiment

If grid search optimizes RMSE while random search optimizes mean absolute error, the selected configurations are not directly comparable under a common optimization objective.

Choose the primary evaluation metric before running the search.

For regression:

• RMSE penalizes large errors more heavily.

• Mean absolute error is less sensitive to very large residuals.

• R² measures performance relative to predicting the target mean, but is not an error measured in the target’s units.

For classification, accuracy may be appropriate for some tasks, but imbalanced-class problems often require metrics such as balanced accuracy, precision, recall, F1, or average precision.

The choice should follow the real cost of prediction errors.

Mistake 5: giving different methods unequal resources

Suppose grid search evaluates 20 configurations while Optuna evaluates 200.

If Optuna wins, the result does not isolate the value of adaptive search.

The larger evaluation budget may explain much of the difference.

For a fair investigation, track at least:

• Number of candidate evaluations.

• Number of CV folds.

• Total model fits.

• Wall-clock time.

• Search bounds and parameter types.

• Random seeds and software versions.

Equal trial counts do not guarantee equal wall-clock costs. Some hyperparameter settings, especially larger trees or more iterations, can be more expensive to fit.

A strong benchmark therefore reports both trial-budget and time-budget perspectives when practical.

12. How to choose search ranges intelligently

An optimizer cannot find a configuration outside the search space you define.

Search-space design is often more important than minor differences between optimization algorithms.

Start with meaningful bounds

Use model knowledge, data size, and reasonable training constraints to construct an initial space.

Avoid searching enormous ranges without justification.

For example, our max_leaf_nodes search runs from 7 to 31. This is intentionally manageable given the dataset and the 100-iteration boosting limit.

It is not a universal range for gradient-boosted trees.

Larger or more complicated datasets may justify more complex trees.

Use log scales for multiplicative parameters

Parameters such as learning rates and regularization strengths often deserve logarithmic sampling.

A tenfold difference between 0.001 and 0.01 can be just as meaningful as a tenfold difference between 0.01 and 0.1.

Uniform numerical sampling does not reflect that structure.

Treat boundary winners as signals

Our grid-search winner used a learning rate of 0.20, the highest value permitted.

That does not prove a larger learning rate would improve performance.

It does, however, suggest a useful follow-up experiment: investigate whether the optimum lies near or beyond that boundary.

There is a complication: in boosting, learning rate interacts with the number of iterations.

Because our iteration count is fixed, increasing the learning rate changes more than just a generic regularization setting. A follow-up search might include the boosting iteration count as another dimension.

Importantly, such revisions should be based on development data and CV evidence, not on repeatedly inspecting the held-out test set.

Avoid too many parameters too early

It is tempting to tune everything the estimator exposes.

That can make the search less efficient.

Start with a small set of parameters that have a plausible effect on model quality.

Then expand the space when the first experiment provides evidence that additional flexibility is needed.

A search over three well-chosen parameters is often more informative than an unfocused search over fifteen.

13. Reproducibility, parallelism, and compute budgets

Reproducibility is about more than setting one random seed.

In the benchmark, we control:

• Synthetic data generation.

• Train-test splitting.

• CV fold assignment.

• Random-search sampling.

• Optuna’s sampler.

• The estimator’s random state.

This makes repeated executions more comparable.

It does not guarantee bit-for-bit identical results across all software versions, CPUs, execution environments, or parallel numerical implementations.

For production experiments, record dependency versions and the full experiment configuration.

Be careful with parallel optimization

scikit-learn can execute CV fits in parallel using its n_jobs settings.

That can accelerate searches, especially when each model fit is substantial.

However, running many model fits simultaneously while each estimator also uses multithreaded numerical routines can oversubscribe CPU resources.

More concurrent work does not automatically mean faster execution.

Optuna can also run multiple trials concurrently, but parallel adaptive optimization introduces another consideration: several trials may be proposed before the optimizer has received results from other running trials.

Consequently, asynchronous or parallel evaluation can change the sequence of suggestions and affect reproducibility.

Our benchmark uses serial search-level execution to keep the comparison straightforward.

For a real workload, benchmark different levels of concurrency rather than assuming that the maximum possible parallelism is best.

Measure the appropriate budget

For expensive training workloads, total model-fit time may dominate everything else.

For cheap estimators, search bookkeeping and Python overhead can become noticeable.

A useful optimization report includes both the best quality found and the resources required to find it.

Consider two search procedures:

• Method A achieves an RMSE of 10.0 in ten minutes.

• Method B achieves an RMSE of 9.98 in four hours.

Method B has a slightly better validation score, but the improvement may not justify its cost.

Your practical objective is rarely to minimize validation error regardless of compute.

It is usually to find a sufficiently good model within constraints on time, memory, energy, and operational complexity.

14. Which method should you use in practice?

All three methods deserve a place in a modern machine-learning workflow.

A workflow I recommend for medium-sized tabular ML projects is:

1. Establish a baseline model using a valid evaluation split.

2. Identify a small number of high-impact hyperparameters.

3. Run random search with a modest, fixed budget.

4. Examine the score distribution and promising parameter regions.

5. Use Optuna for a more targeted adaptive search when there is a reason to expect additional gains.

6. Consider a small local grid for a carefully defined region if exhaustive comparisons there are valuable.

7. Evaluate the selected procedure using untouched data or a rigorous outer-CV protocol.

The order is not mandatory. A tiny, well-defined discrete space may justify starting and ending with grid search.

The essential principle is to let experimental evidence—not optimizer branding—guide the choice.

15. A practical checklist before trusting your search results

Before claiming that one tuning approach is better than another, confirm the following:

• Every search uses the same base estimator and relevant fixed model settings.
• Candidate budgets and CV strategies are documented.
• Search-space bounds and sampling distributions are explicit.
• The same primary scoring metric is used throughout.
• Data-dependent preprocessing occurs inside the training folds.
• Group or temporal splitting is used when the prediction task requires it.
• Random seeds are recorded.
• Failed trials are identified rather than silently overlooked.
• Computational cost is measured, not just model quality.
• The test set is not used to revise the search.
• The winning CV score is not misrepresented as an unbiased final performance estimate.
• Important model-selection claims are supported by repeated experiments or independent evaluation.

This checklist matters more than whether the search algorithm is exhaustive, random, or adaptive.

Conclusion: better hyperparameter tuning starts with better experimental design

Grid search, random search, and Optuna solve the same fundamental problem: choosing hyperparameter configurations using a limited number of model evaluations.

They differ in how those evaluations are allocated.

Grid search prioritizes exhaustive coverage of a predefined set of choices.

Random search spreads evaluations across a broader set of possibilities, often providing stronger coverage of important continuous dimensions.

Optuna’s TPE takes adaptation a step further, using observed results to guide subsequent evaluations.

Our executable comparison demonstrated that an adaptive method does not necessarily win a small benchmark. With the same model, four CV folds, and 18 candidates per method, grid search found the best configuration in the example run.

The more important achievement was that the comparison was controlled, reproducible, and evaluated against an independent test set.

A credible hyperparameter-search experiment has three foundations: a meaningful search space, a valid evaluation procedure, and an explicit computational budget.

Without those foundations, running more trials can simply produce a more convincing-looking overfit.

Your next experiment

Run the benchmark on your machine, inspect the results, and then make three changes:

• Increase the budget from 18 to 50 or 100 trials, using a correspondingly larger grid if you want to retain equal candidate counts.

• Repeat the search using several optimization seeds to investigate how stable each result is.

• Add max_iter to the search space and observe how its interaction with learning rate changes the preferred configurations.

Keep the held-out test set untouched while exploring those changes. Compare the methods using CV evidence, select a final approach, and evaluate it independently.

The goal is not to prove that one optimizer always wins. It is to build a repeatable process that finds good models—and produces performance estimates you can trust.