Random forests: why averaging many bad trees makes one great model

The puzzle: why should averaging unstable trees help?

A fully grown decision tree is one of machine learning’s most intuitive models.

It repeatedly asks questions such as:

• Is feature 7 less than 1.43?

• Is feature 2 greater than -0.18?

• Did feature 11 cross some learned threshold?

Eventually, each observation reaches a leaf, and that leaf produces a prediction.

The problem is that a tree can be too responsive to its training data.

Change a handful of rows, and the best split near the top of the tree may change. Once an early split changes, many later splits change too. You can end up with a visibly different tree even though the underlying dataset barely changed.

That makes deep decision trees high-variance estimators.

A random forest attacks that weakness with an idea that initially sounds almost suspiciously simple:

1. Train many different trees.

2. Deliberately make those trees somewhat different from one another.

3. Average their predictions.

For classification, that averaging happens through class probabilities and voting behavior. For regression, it is literally an average of tree predictions.

The important part is not merely “use lots of trees.” If every tree made exactly the same mistakes, averaging them would accomplish almost nothing.

The real trick is:

Build trees that are individually useful but imperfectly correlated.

That single idea explains bootstrap sampling, random feature selection, out-of-bag evaluation, and much of the practical behavior of random forests.

As of scikit-learn 1.9.0, released in June 2026, RandomForestClassifier still defaults to 100 trees, bootstrap sampling enabled, and max_features="sqrt" for classification.

Set up a current environment

The examples below target scikit-learn 1.9.0.

Create an isolated environment:

python -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

On Windows PowerShell:

.venv\Scripts\Activate.ps1

Install the version used here:

python -m pip install --upgrade pip
python -m pip install "scikit-learn==1.9.0"

Check the installation:

python -c "import sklearn; print(sklearn.__version__)"

You should see:

1.9.0

Scikit-learn recommends using an isolated environment and installing the latest official release for normal use.

Start with one deliberately unstable tree

We will use synthetic classification data rather than a domain-specific dataset. That keeps the experiment reproducible and lets us control how much signal, redundancy, and noise exist.

The dataset will contain:

• 6,000 observations

• 20 numerical features

• 8 genuinely informative features

• 6 redundant features derived from useful signal

• some deliberately flipped labels

• two classes

Here is a complete comparison between one decision tree and a random forest:

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier


X, y = make_classification(
    n_samples=6000,
    n_features=20,
    n_informative=8,
    n_redundant=6,
    n_repeated=0,
    n_classes=2,
    class_sep=1.0,
    flip_y=0.08,
    random_state=42,
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y,
    random_state=42,
)

tree = DecisionTreeClassifier(
    random_state=42,
)

forest = RandomForestClassifier(
    n_estimators=500,
    oob_score=True,
    n_jobs=-1,
    random_state=42,
)

tree.fit(X_train, y_train)
forest.fit(X_train, y_train)

tree_pred = tree.predict(X_test)
forest_pred = forest.predict(X_test)
forest_prob = forest.predict_proba(X_test)[:, 1]

print(f"Single-tree accuracy: {accuracy_score(y_test, tree_pred):.3f}")
print(f"Forest accuracy:      {accuracy_score(y_test, forest_pred):.3f}")
print(f"Forest ROC AUC:       {roc_auc_score(y_test, forest_prob):.3f}")
print(f"Forest OOB score:     {forest.oob_score_:.3f}")

With this fixed random seed, results should be around:

Single-tree accuracy: 0.753
Forest accuracy:      0.875
Forest ROC AUC:       0.927
Forest OOB score:     0.885

Do not become attached to those exact values as universal properties of the algorithms. They are properties of this generated dataset and this random seed.

What matters is the pattern.

The unrestricted tree fits very flexible boundaries. That flexibility is useful when the data contain complicated interactions, but it also allows the tree to react strongly to noise.

The forest still contains deep, flexible trees. It does not solve overfitting simply by making each individual model conservative.

Instead, it solves much of the variance problem by averaging many different versions of a high-variance model.

Why the single tree is so unstable

Imagine two candidate splits at a node.

Suppose split A reduces impurity by 0.1842 and split B reduces it by 0.1838.

On today’s training sample, A wins.

Now replace ten training observations.

Perhaps B reduces impurity by 0.1844 while A reaches only 0.1840.

The tree chooses B instead.

That sounds like a tiny difference, but a split near the root changes which rows reach every descendant node. The entire lower portion of the tree can be reorganized.

This is why trees have an unusual combination of properties:

• they can model nonlinear relationships;

• they automatically capture feature interactions;

• they need little numerical preprocessing;

• but deep trees can have high variance.

High variance does not mean the model’s predictions are random.

It means the learned model is sensitive to which particular sample happened to be used for training.

Random forests exploit exactly that weakness.

Averaging cancels unstable mistakes

Suppose you have 500 trees.

For one customer, server event, transaction, sensor reading, or other observation, the first few trees might estimate positive-class probabilities like:

Tree 1: 0.91
Tree 2: 0.44
Tree 3: 0.73
Tree 4: 0.32
Tree 5: 0.81
...

Any one tree may be overly confident because of the particular sample and splits it saw.

The forest averages across the trees.

If one tree makes an unusually high prediction while another makes an unusually low prediction, their errors can partly cancel.

There is a statistical intuition behind this.

If B estimators each have error variance σ² and their errors have average pairwise correlation ρ, the variance of their average behaves roughly like:

Variance of average = σ² × [ρ + (1 − ρ) / B]

Two things jump out.

First, increasing B helps.

Second, increasing B cannot completely rescue you if ρ is close to 1.

If every model makes almost identical errors, the correlation term dominates.

That is why a useful forest must create diversity among trees, not merely duplicate one tree hundreds of times.

Source of diversity number one: bootstrap samples

By default, each tree in scikit-learn’s random forest is trained on a bootstrap sample when bootstrap=True.

Suppose the training set contains 4,500 rows.

To create the training data for one tree, the algorithm draws 4,500 row positions with replacement.

Because sampling uses replacement, the same original observation may appear several times.

Another observation may not appear at all.

Conceptually, a tiny bootstrap sample might look like:

Original row IDs:
0 1 2 3 4 5 6 7 8 9

Bootstrap sample:
7 2 2 9 1 7 4 4 4 0

Rows 3, 5, 6, and 8 never appeared in that particular sample.

The next tree receives a different bootstrap sample.

That means every tree sees a slightly different version of reality.

You can inspect which training rows scikit-learn selected for individual trees through estimators_samples_:

sample_indices = forest.estimators_samples_[0]

print(sample_indices[:20])
print(f"Draws for first tree: {len(sample_indices)}")
print(f"Unique rows used:      {len(set(sample_indices))}")

The number of draws equals the training-set size when max_samples=None, but the number of unique observations is substantially smaller because some rows are selected repeatedly.

That repetition is not an accident. It is the foundation of bootstrap resampling.

The famous 63.2% result

Here is one of the most useful facts to understand about bootstrap sampling.

Take a dataset with n observations.

For any single draw, the probability that one particular observation is not selected is:

1 - 1/n

A bootstrap sample performs n draws with replacement, so the probability that the observation survives all n draws without being selected is:

(1 - 1/n)^n

As n becomes large, that approaches:

1/e ≈ 0.368

So approximately 36.8% of the original training observations are absent from a particular tree’s bootstrap sample.

Equivalently, only about 63.2% of the distinct training rows appear at least once.

That does not mean each tree trains on only 63.2% as many row positions. The bootstrap sample still contains n draws. Some observations simply appear multiple times.

This gives us a useful side effect: the observations that were not drawn can act as miniature validation data for that tree.

Those are the out-of-bag observations.

Out-of-bag scoring: validation hiding inside the forest

When you set:

forest = RandomForestClassifier(
    n_estimators=500,
    oob_score=True,
    random_state=42,
    n_jobs=-1,
)

scikit-learn tracks predictions for training observations using trees for which those observations were out of bag.

For observation 143, perhaps trees 1, 4, 8, 10, 12, and many others did not train on it.

Those trees can predict observation 143 without having fitted on that observation.

Aggregate those predictions across the forest, repeat the procedure for other observations, and you get an out-of-bag estimate of model performance.

You read it from:

print(forest.oob_score_)

For RandomForestClassifier, oob_score=True uses accuracy by default. Current scikit-learn also accepts a callable metric through oob_score, provided bootstrap sampling is enabled.

OOB is useful, but it is not magical

Out-of-bag scoring is convenient because you can get a generalization estimate without carving another explicit validation split out of the training data.

That is particularly helpful when:

• data are moderately scarce;

• you want quick comparisons;

• you are studying the effect of forest size;

• you want a sanity check alongside held-out evaluation.

But OOB performance should not automatically become your final reported benchmark.

A clean test set still has a major conceptual advantage: it is entirely separate from model development.

If you repeatedly change hyperparameters because they improve the OOB score, then the OOB estimate becomes part of your model-selection process.

For serious evaluation, keep an untouched test set or use an appropriate cross-validation strategy.

And if your observations have temporal, grouped, geographic, or user-level dependencies, random bootstrap behavior may not represent the real deployment boundary. Your validation design must match how future data arrive.

Source of diversity number two: random subsets of features

Bootstrap sampling changes the rows.

Random forests also change which features compete at individual splits.

For classification, current scikit-learn uses:

max_features="sqrt"

by default.

Suppose your dataset contains 100 features.

At one split, a tree may consider roughly 10 randomly chosen features.

At another node, it receives another random subset.

Why deliberately prevent the tree from always seeing every feature?

Because one exceptionally strong predictor can otherwise dominate nearly every tree.

Imagine a dataset where feature A is much more predictive than everything else.

If every tree sees every feature at every node, many bootstrap samples may still choose feature A near the root. The resulting trees become similar.

Their errors become more correlated.

The forest becomes less effective at averaging those errors away.

Random feature selection sometimes forces a tree to discover another route:

• feature B combined with feature C;

• feature D with a different threshold;

• a weaker feature interaction;

• a secondary predictor that would otherwise never beat feature A.

Each individual tree may become slightly worse.

The collection can become better.

That is the core paradox.

“Bad trees” needs one important qualification

It is tempting to describe a random forest as hundreds of bad trees somehow becoming one good model.

That is memorable, but technically incomplete.

The trees should not be useless.

If every tree performs at chance level and contains no meaningful relationship with the target, averaging will not manufacture signal from nothing.

A more precise description is:

Random forests combine many high-variance, reasonably predictive, deliberately decorrelated trees.

The individual trees are often weaker than the final forest.

They are not supposed to be incompetent.

A forest works when its trees have both:

• strength: each tree captures meaningful predictive structure;

• diversity: the trees do not all make the same mistakes.

Hyperparameters such as max_features, max_samples, tree depth, and leaf-size constraints influence that trade-off.

What does n_estimators really do?

n_estimators controls how many trees are built.

Current scikit-learn defaults to:

n_estimators=100

Increasing the number of trees typically reduces Monte Carlo noise in the ensemble.

For example:

small_forest = RandomForestClassifier(
    n_estimators=20,
    random_state=42,
    n_jobs=-1,
)

large_forest = RandomForestClassifier(
    n_estimators=1000,
    random_state=42,
    n_jobs=-1,
)

More trees generally make predictions more stable.

But returns diminish.

Going from 10 trees to 100 can matter substantially.

Going from 1,000 to 10,000 often changes far less while multiplying:

• training time;

• prediction time;

• memory use;

• serialized model size.

Unlike gradient boosting, adding more trees to a standard random forest does not normally produce the same kind of overfitting curve where performance rises and then sharply collapses merely because the ensemble has “too many rounds.”

Once enough trees exist to stabilize the average, additional trees tend mostly to increase computation.

Inspecting convergence with OOB score

You can empirically examine how the out-of-bag score stabilizes as the forest grows:

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier


X, y = make_classification(
    n_samples=5000,
    n_features=20,
    n_informative=8,
    n_redundant=6,
    flip_y=0.08,
    random_state=42,
)

for n_trees in [10, 25, 50, 100, 250, 500, 1000]:
    model = RandomForestClassifier(
        n_estimators=n_trees,
        oob_score=True,
        n_jobs=-1,
        random_state=42,
    )
    model.fit(X, y)

    print(
        f"{n_trees:4d} trees | "
        f"OOB accuracy = {model.oob_score_:.4f}"
    )

You will usually see larger changes early and increasingly small improvements later.

That is a better way to think about tree count than memorizing “500 trees is always enough.”

The right number depends on:

• dataset size;

• complexity;

• feature count;

• class structure;

• latency requirements;

• acceptable model size;

• randomness in your particular problem.

Tree depth is different from forest size

One source of confusion is treating max_depth and n_estimators as interchangeable forms of complexity control.

They are not.

max_depth changes the complexity of each individual tree.

n_estimators changes how many trees get averaged.

Scikit-learn’s forest trees are unrestricted by depth by default:

max_depth=None

The documentation explicitly notes that default tree-size parameters can produce fully grown, unpruned trees that may become very large.

On a large dataset, you may want to constrain tree size for both regularization and memory reasons:

forest = RandomForestClassifier(
    n_estimators=500,
    max_depth=18,
    min_samples_leaf=4,
    n_jobs=-1,
    random_state=42,
)

Useful controls include:

• max_depth

• min_samples_split

• min_samples_leaf

• max_leaf_nodes

• min_impurity_decrease

• ccp_alpha

Of these, min_samples_leaf is often especially interpretable.

Increasing it means predictions cannot be based on extremely tiny leaves.

max_samples: control row randomness directly

By default, bootstrap sampling draws as many row positions as exist in the training set.

You can change that with max_samples.

For example:

forest = RandomForestClassifier(
    n_estimators=500,
    bootstrap=True,
    max_samples=0.7,
    n_jobs=-1,
    random_state=42,
)

Now each tree draws a bootstrap sample containing 70% as many draws as there are training observations.

Smaller samples can:

• increase tree diversity;

• reduce the work per tree;

• sometimes increase bias;

• sometimes improve generalization when individual trees are too similar.

It is another bias-variance-correlation knob rather than a parameter with one universally correct setting.

Forest predictions are averages, not mysterious consensus

For binary classification, each tree exposes class probabilities through predict_proba.

The forest prediction can be understood as averaging those per-tree probability estimates.

You can verify the relationship directly:

import numpy as np


row = X_test[[0]]

tree_probabilities = np.array(
    [
        estimator.predict_proba(row)[0, 1]
        for estimator in forest.estimators_
    ]
)

manual_average = tree_probabilities.mean()
forest_probability = forest.predict_proba(row)[0, 1]

print(f"Manual average: {manual_average:.6f}")
print(f"Forest output:  {forest_probability:.6f}")

You should see matching values apart from negligible floating-point formatting.

This is worth internalizing.

The forest is not learning a second neural network or fitting another meta-model on top of the trees.

Its power comes from constructing the collection wisely and aggregating it.

Feature importance: useful, convenient, and easy to misuse

After fitting a forest, scikit-learn gives you:

forest.feature_importances_

These are impurity-based feature importances, often called mean decrease in impurity.

Let’s give our synthetic columns readable names and rank them:

feature_names = [
    f"feature_{i}"
    for i in range(X.shape[1])
]

importance_pairs = sorted(
    zip(feature_names, forest.feature_importances_),
    key=lambda pair: pair[1],
    reverse=True,
)

for name, importance in importance_pairs:
    print(f"{name:>12}: {importance:.4f}")

A feature receives importance when splits using that feature reduce the tree’s impurity.

Across the forest, those decreases are accumulated and normalized.

This is fast because the required information already exists inside the fitted trees.

But convenience is not the same thing as truth.

Pitfall 1: impurity importance can favor high-cardinality features

Scikit-learn’s current documentation explicitly warns that impurity-based importances can be misleading for features with many unique values.

A feature with many candidate split points gets many opportunities to produce an apparently useful split.

That can make it look more important than it deserves.

So this:

print(forest.feature_importances_)

should not automatically become a business explanation like:

Feature 12 causes 31% of the model’s decisions.

That interpretation is wrong.

The values are not causal contributions.

They describe how the fitted trees used variables to reduce their internal splitting criterion.

Permutation importance asks a better predictive question

Permutation importance asks:

How much worse does the fitted model become when I destroy the information in this feature?

The procedure is straightforward.

For one feature:

1. Measure baseline model performance.

2. Shuffle that feature across observations.

3. Predict again.

4. Measure the performance drop.

5. Repeat the shuffle several times.

If destroying the feature badly damages performance, the feature was useful to the fitted model.

Use a held-out set:

from sklearn.inspection import permutation_importance


result = permutation_importance(
    forest,
    X_test,
    y_test,
    scoring="accuracy",
    n_repeats=20,
    random_state=42,
    n_jobs=-1,
)

permutation_pairs = sorted(
    zip(
        feature_names,
        result.importances_mean,
        result.importances_std,
    ),
    key=lambda row: row[1],
    reverse=True,
)

for name, mean_drop, std_drop in permutation_pairs:
    print(
        f"{name:>12}: "
        f"mean drop={mean_drop:.4f}, "
        f"std={std_drop:.4f}"
    )

Scikit-learn recommends permutation importance as an alternative when the high-cardinality bias of impurity importance matters, and it can be calculated on left-out test data.

That last point is crucial.

Impurity importance answers a question about the fitted tree structures.

Permutation importance can answer a question about predictive performance on unseen data.

Pitfall 2: correlated features can fool your interpretation

Permutation importance has its own trap.

Suppose two features contain nearly the same information:

temperature_sensor_A
temperature_sensor_B

If they are strongly correlated, the forest may use either one.

Now shuffle sensor A.

The model can still obtain similar information from sensor B.

Performance barely changes.

You might incorrectly conclude:

Sensor A is unimportant.

Then you shuffle sensor B independently.

The model falls back to sensor A.

Again the performance drop is small.

Now both features look unimportant even though the underlying signal is extremely important.

This is not a bug in permutation importance.

It reflects the question being asked:

Given that all other columns remain available, how much unique predictive information does this column contribute?

With correlated predictors, importance can be shared, substituted, or masked.

Practical responses include:

• inspect correlations or feature clusters;

• evaluate groups of related variables together;

• use domain knowledge;

• compare multiple interpretation methods;

• avoid turning importance rankings into causal claims.

Pitfall 3: importance does not explain individual predictions

Global feature importance tells you something about a model across a dataset.

It does not tell you why one particular observation received probability 0.87.

Those are different questions.

Global importance asks:

Which features matter to predictive behavior overall?

Local explanation asks:

What drove this particular prediction?

Do not substitute one for the other.

And neither automatically answers:

What would happen in the real world if I intervened and changed this feature?

That is a causal question, which ordinary predictive feature importance does not solve.

OOB score versus test score: what should you expect?

In our first experiment, the values were roughly:

Test accuracy: 0.875
OOB accuracy:  0.885

They are close, but they are not supposed to be identical.

The estimates are created differently.

Test accuracy uses one explicitly held-out partition.

OOB accuracy aggregates predictions from different subsets of trees for different training observations.

A modest difference is normal.

A large difference deserves investigation.

Possible causes include:

• a small or unrepresentative test set;

• distribution shift;

• accidental leakage;

• grouped observations split incorrectly;

• temporal structure ignored by random splitting;

• repeated entities appearing on both sides;

• extensive tuning against one evaluation mechanism.

Treat disagreement as diagnostic information rather than deciding one score must always be correct.

When random forests shine

Random forests remain an excellent baseline for structured, tabular problems because they combine strong nonlinear modeling with relatively little preprocessing.

They are especially attractive when:

• relationships are nonlinear;

• interactions matter;

• numerical scaling would otherwise be annoying;

• you want a strong baseline quickly;

• the dataset is not enormous;

• low-latency inference is still practical;

• interpretability through tree-level inspection and importance tools is useful.

They can capture patterns such as:

high traffic
AND
low cache hit rate
AND
weekend deployment

without you manually constructing that interaction.

A linear model would need suitable transformations or interaction features to represent the same rule naturally.

When they are less attractive

Random forests are not automatically the best choice for every tabular task.

They can become cumbersome when:

• the dataset contains millions of rows and hundreds of features;

• memory usage from deep trees becomes large;

• extremely low inference latency is required;

• model files must be tiny;

• extrapolation beyond observed numerical ranges matters;

• sparse, very high-dimensional inputs dominate;

• a carefully tuned boosting approach achieves substantially better accuracy.

Random forests also produce piecewise-constant predictions.

For regression, that means they do not naturally extrapolate a smooth trend beyond the target patterns represented in training leaves.

If your training data show energy demand only between temperatures of -5°C and 35°C, a forest does not learn a formula that confidently extends some continuous trend to 70°C.

That conservatism can be either helpful or limiting depending on the application.

Random forests versus boosting

Both methods use many trees, but the training philosophy is very different.

A random forest trains trees largely independently.

Conceptually:

Tree 1 ─┐
Tree 2 ─┤
Tree 3 ─┼─> average
Tree 4 ─┤
Tree 5 ─┘

A boosting system is sequential.

Conceptually:

Model 1
   ↓
Model 2 learns from remaining error
   ↓
Model 3 learns from remaining error
   ↓
Model 4
   ↓
combined prediction

That distinction matters operationally.

Random forest training parallelizes naturally across trees, which is why:

n_jobs=-1

can use available processors for tree-level work.

Boosting often extracts more predictive performance from structured datasets, but typically requires more careful tuning and follows a different bias-reduction strategy.

A random forest remains one of the best “strong baseline before getting fancy” models you can train.

A practical tuning order

You do not need to grid-search every parameter simultaneously.

Start with a sensible baseline:

model = RandomForestClassifier(
    n_estimators=500,
    oob_score=True,
    n_jobs=-1,
    random_state=42,
)

Then investigate the parameters that address an actual problem.

If the model uses too much memory

Try:

model = RandomForestClassifier(
    n_estimators=500,
    max_depth=20,
    min_samples_leaf=2,
    n_jobs=-1,
    random_state=42,
)

If individual trees appear too correlated

Experiment with stronger feature randomness:

model = RandomForestClassifier(
    n_estimators=500,
    max_features=0.5,
    n_jobs=-1,
    random_state=42,
)

Or stronger row randomness:

model = RandomForestClassifier(
    n_estimators=500,
    bootstrap=True,
    max_samples=0.7,
    n_jobs=-1,
    random_state=42,
)

If leaves seem too specific

Increase min_samples_leaf:

model = RandomForestClassifier(
    n_estimators=500,
    min_samples_leaf=5,
    n_jobs=-1,
    random_state=42,
)

If results vary too much between repeated fits

First make sure you have fixed:

random_state=42

Then consider whether your evaluation sample itself is too small or unstable.

Reproducibility and statistical stability are not the same thing.

A fixed seed gives you the same random experiment again.

It does not guarantee that the experiment is representative.

The cherry on the cake: the forest’s bootstrap trick came from statistical uncertainty

Bootstrap sampling was not invented for random forests.

Its origin is much broader and, in a way, more surprising.

Statistician Bradley Efron introduced the bootstrap in 1979 as a general method for estimating uncertainty by repeatedly resampling the data you already have. Stanford’s statistics history describes the central idea: draw many bootstrap samples from the observed dataset, recompute a statistic for each one, and use the variation among those results to estimate uncertainty.

The name is a joke with serious mathematics behind it.

The method tries to learn about sampling uncertainty using only the sample already available — metaphorically pulling yourself up by your own bootstraps.

Decades later, the same resampling mechanism became one of the two central randomization engines inside random forests.

That is a beautiful conceptual migration.

In classical statistics, bootstrap resampling asks:

If the data had come out a little differently, how much might my statistic change?

Inside a forest, resampling effectively says:

Let’s intentionally show every tree a slightly different version of the dataset and then average away some of their instability.

The weakness of the tree — its sensitivity to the sample — becomes something the algorithm deliberately generates and exploits.

A compact end-to-end experiment

Here is one runnable script containing the main ideas from this tutorial:

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier


X, y = make_classification(
    n_samples=6000,
    n_features=20,
    n_informative=8,
    n_redundant=6,
    flip_y=0.08,
    random_state=42,
)

feature_names = [
    f"feature_{i}"
    for i in range(X.shape[1])
]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y,
    random_state=42,
)

tree = DecisionTreeClassifier(
    random_state=42,
)

forest = RandomForestClassifier(
    n_estimators=500,
    bootstrap=True,
    oob_score=True,
    n_jobs=-1,
    random_state=42,
)

tree.fit(X_train, y_train)
forest.fit(X_train, y_train)

tree_accuracy = accuracy_score(
    y_test,
    tree.predict(X_test),
)

forest_accuracy = accuracy_score(
    y_test,
    forest.predict(X_test),
)

print(f"Tree accuracy:   {tree_accuracy:.4f}")
print(f"Forest accuracy: {forest_accuracy:.4f}")
print(f"OOB accuracy:    {forest.oob_score_:.4f}")

print("\nImpurity-based feature importance")

mdi_ranking = sorted(
    zip(feature_names, forest.feature_importances_),
    key=lambda pair: pair[1],
    reverse=True,
)

for name, value in mdi_ranking:
    print(f"{name:>12}: {value:.4f}")

permutation = permutation_importance(
    forest,
    X_test,
    y_test,
    scoring="accuracy",
    n_repeats=10,
    random_state=42,
    n_jobs=-1,
)

print("\nPermutation feature importance")

permutation_ranking = sorted(
    zip(
        feature_names,
        permutation.importances_mean,
        permutation.importances_std,
    ),
    key=lambda row: row[1],
    reverse=True,
)

for name, mean_drop, std_drop in permutation_ranking:
    print(
        f"{name:>12}: "
        f"{mean_drop:.4f} +/- {std_drop:.4f}"
    )

Run it:

python forest_demo.py

Then start experimenting.

Change one parameter at a time:

max_features=0.3

Try a smaller bootstrap sample:

max_samples=0.6

Force less-specific leaves:

min_samples_leaf=10

Increase the forest size:

n_estimators=1000

Watch what happens to:

• held-out accuracy;

• OOB accuracy;

• training time;

• prediction time;

• feature rankings;

• stability across random seeds.

That experimentation makes the theory concrete.

The mental model to keep

You do not need to remember every parameter to understand random forests.

Remember four ideas.

One: deep decision trees are powerful but unstable.

Two: averaging reduces variance when the models do not make perfectly correlated errors.

Three: forests manufacture diversity through bootstrap row sampling and random feature selection.

Four: the final prediction is powerful not because every tree is brilliant, but because many imperfect views of the data are combined.

So the apparent contradiction disappears.

A forest does not somehow convert useless models into intelligence.

It takes a model whose greatest weakness is instability, creates many deliberately different versions of it, and turns that instability into something averaging can suppress.

That is why one noisy tree can disappoint while hundreds of related-but-different trees can become an exceptionally strong baseline.

Your next step

Copy the end-to-end script, run it, and then replace the synthetic data with one of your own neutral tabular problems: retail conversion, housing characteristics, energy consumption, server telemetry, delivery timing, sensor events, or another structured dataset.

Train one unrestricted tree first.

Then train the forest.

Compare held-out performance, inspect the OOB score, calculate both impurity and permutation feature importances, and deliberately change max_features, max_samples, and min_samples_leaf.

Do not just use a random forest.

Make the trees disagree, measure what averaging buys you, and learn exactly why the forest works.