LEARN · CLASSICAL MACHINE LEARNING
A decision tree turns prediction into a sequence of readable questions:
Is worst radius <= 16.795?
A “yes” sends the sample left; a “no” sends it right. After several questions, the sample reaches a leaf containing a class prediction and probabilities.
That makes DecisionTrees one of the most approachable tools in ClassicalML. The same simplicity also hides their main failure mode: an unconstrained tree can memorize noise almost perfectly.
Use a pinned, current environment
These examples target scikit-learn 1.9.0, released in June 2026, with NumPy 2.5.1 and Matplotlib 3.11.1. Use Python 3.12–3.14 for this pinned environment.
python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install \ "scikit-learn==1.9.0" \ "numpy==2.5.1" \ "matplotlib==3.11.1" python -c "import sklearn, numpy, matplotlib; print(sklearn.__version__, numpy.__version__, matplotlib.__version__)"
Version pins matter. A fixed random_state controls algorithmic randomness, but exact thresholds can still shift across library releases or numerical environments.
The mental model: recursive partitioning
A classification tree repeatedly divides training rows into smaller groups. At each node, it searches for a rule of the form:
feature_j <= threshold
Rows satisfying the rule go left; the rest go right. The algorithm greedily chooses the split with the largest immediate impurity reduction, then repeats inside each child.
“Greedy” means the tree optimizes the next split, not the complete future tree. Globally optimal tree learning is computationally hard, so practical implementations use local heuristics.
The final tree partitions feature space into axis-aligned regions. Every sample landing in the same leaf receives the same learned class distribution.
Gini impurity and entropy
A node is pure when all its samples belong to one class. Suppose a binary node contains 80 benign and 20 malignant samples:
-
p(benign) = 0.8
-
p(malignant) = 0.2
Gini impurity
Gini measures how mixed the classes are:
Gini = 1 − Σ pᵢ²
For the 80/20 node:
Gini = 1 − (0.8² + 0.2²) = 0.32
For binary classification:
-
Gini 0 means pure.
-
Gini 0.5 means a 50/50 mixture.
-
Lower is better.
Entropy
Entropy measures uncertainty:
Entropy = −Σ pᵢ log₂(pᵢ)
For the same node:
Entropy ≈ 0.722
Binary entropy ranges from 0 for a pure node to 1 for a 50/50 node.
In scikit-learn 1.9, DecisionTreeClassifier supports criterion="gini", "entropy", and "log_loss"; entropy and log loss both use Shannon information gain. Gini remains the default.
How a split is scored
The tree evaluates both children, weighted by their sizes:
Weighted child impurity = (n_left / n_parent) × impurity_left + (n_right / n_parent) × impurity_right
Impurity decrease = impurity_parent − weighted child impurity
Imagine a parent with Gini 0.48. A candidate split creates:
-
Left: 60 samples, Gini 0.10
-
Right: 40 samples, Gini 0.30
Then:
Weighted child Gini = 0.6 × 0.10 + 0.4 × 0.30 = 0.18
Gini decrease = 0.48 − 0.18 = 0.30
A large decrease means the children are collectively purer. Scikit-learn’s min_impurity_decrease applies an additional weight for the fraction of the full training set reaching that node and incorporates sample weights when supplied.
Train and plot a real tree
This complete block loads the breast-cancer dataset, creates a stratified split, trains a deliberately small tree, and plots it.
import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier, plot_tree data = load_breast_cancer(as_frame=True) X = data.data y = data.target X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, stratify=y, random_state=42, ) model = DecisionTreeClassifier( criterion="gini", max_depth=3, min_samples_leaf=10, random_state=42, ) model.fit(X_train, y_train) train_predictions = model.predict(X_train) test_predictions = model.predict(X_test) print(f"Training accuracy: {accuracy_score(y_train, train_predictions):.3f}") print(f"Test accuracy: {accuracy_score(y_test, test_predictions):.3f}") print(f"Tree depth: {model.get_depth()}") print(f"Number of leaves: {model.get_n_leaves()}") plt.figure(figsize=(18, 9)) plot_tree( model, feature_names=X.columns, class_names=data.target_names.tolist(), filled=True, rounded=True, node_ids=True, precision=3, ) plt.tight_layout() plt.show()
plot_tree is scikit-learn’s supported Matplotlib tree visualizer. It can show split rules, impurity, sample counts, class distributions, node IDs, and predicted classes.
How to read a plotted node
A node such as worst radius <= 16.795 means:
-
Go left when the condition is true.
-
Go right when it is false.
The remaining fields are:
-
gini: class impurity among training samples reaching the node.
-
samples: the number of samples reaching it; displayed counts reflect sample weights when used.
-
value: weighted class counts with the default
proportion=False. -
class: the class with the highest learned probability.
At prediction time, a sample follows rules from root to leaf. predict_proba returns the class fractions learned in that leaf, while predict chooses the class with the largest probability.
Verify the actual root split
This block depends on model, X, and data created in the previous training block.
import numpy as np tree = model.tree_ root = 0 left = tree.children_left[root] right = tree.children_right[root] feature_index = tree.feature[root] feature_name = X.columns[feature_index] threshold = tree.threshold[root] n_parent = tree.weighted_n_node_samples[root] n_left = tree.weighted_n_node_samples[left] n_right = tree.weighted_n_node_samples[right] weighted_child_gini = ( (n_left / n_parent) * tree.impurity[left] + (n_right / n_parent) * tree.impurity[right] ) gini_decrease = tree.impurity[root] - weighted_child_gini parent_counts = tree.value[root][0] * n_parent left_counts = tree.value[left][0] * n_left right_counts = tree.value[right][0] * n_right print(f"Root rule: {feature_name} <= {threshold:.3f}") print(f"Class order: {data.target_names.tolist()}") print(f"Parent counts: {np.round(parent_counts, 1)}") print(f"Left counts: {np.round(left_counts, 1)}") print(f"Right counts: {np.round(right_counts, 1)}") print(f"Parent Gini: {tree.impurity[root]:.4f}") print(f"Weighted child Gini: {weighted_child_gini:.4f}") print(f"Gini decrease: {gini_decrease:.4f}")
With the pinned setup, the root should split on worst radius near 16.795. The left child is mostly benign; the right child is mostly malignant. The important point is that the model selected this rule for weighted impurity reduction—not because it directly maximized node-level accuracy.
Current scikit-learn stores class proportions in tree_.value; multiplying by weighted_n_node_samples recovers weighted counts, as the code does.
Why trees overfit
Default size controls can produce a fully grown, unpruned tree whose leaves are pure or too small to split further. Such a model can isolate outliers, measurement errors, rare combinations, and individual samples.
This creates:
-
Low bias: the tree can fit complicated training patterns.
-
High variance: small data changes can restructure early splits and entire subtrees.
-
False confidence: tiny leaves can output probabilities of 0 or 1 from only a few observations.
A tree is not automatically interpretable. It is interpretable only while it remains small enough to inspect.
Control complexity with cross-validation
The most useful controls are:
-
max_depth: caps the number of decisions from root to leaf. -
min_samples_leaf: prevents predictions based on tiny groups. -
max_leaf_nodes: directly limits terminal regions. -
ccp_alpha: prunes branches through minimal cost-complexity pruning.
Scikit-learn explicitly recommends controlling tree size because defaults can create very large trees.
This tuning block depends on X_train, X_test, y_train, and y_test from the training example.
from sklearn.model_selection import GridSearchCV, StratifiedKFold from sklearn.tree import DecisionTreeClassifier parameter_grid = { "max_depth": [2, 3, 4, None], "min_samples_leaf": [1, 5, 10, 20], "ccp_alpha": [0.0, 0.002, 0.01], } cross_validation = StratifiedKFold( n_splits=5, shuffle=True, random_state=42, ) search = GridSearchCV( estimator=DecisionTreeClassifier(random_state=42), param_grid=parameter_grid, scoring="balanced_accuracy", cv=cross_validation, n_jobs=-1, ) search.fit(X_train, y_train) print("Best parameters:", search.best_params_) print(f"Cross-validation score: {search.best_score_:.3f}") print(f"Test score: {search.score(X_test, y_test):.3f}")
Do not choose complexity from training accuracy. Select it with cross-validation, then evaluate the untouched test set once.
Cherry on the cake: a tree can “solve” random noise
The following block is independent of the earlier examples. It creates 2,000 random rows and random binary labels—there is no real signal.
import numpy as np from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier rng = np.random.default_rng(7) X_noise = rng.normal(size=(2_000, 20)) y_noise = rng.integers(0, 2, size=2_000) X_train_noise, X_test_noise, y_train_noise, y_test_noise = train_test_split( X_noise, y_noise, test_size=0.30, stratify=y_noise, random_state=7, ) noise_tree = DecisionTreeClassifier(random_state=7) noise_tree.fit(X_train_noise, y_train_noise) print(f"Training accuracy: {noise_tree.score(X_train_noise, y_train_noise):.3f}") print(f"Test accuracy: {noise_tree.score(X_test_noise, y_test_noise):.3f}") print(f"Tree depth: {noise_tree.get_depth()}") print(f"Number of leaves: {noise_tree.get_n_leaves()}")
Expect training accuracy of 1.000 while test accuracy stays near 0.50. The tree grows roughly two dozen levels deep and creates hundreds of leaves to carve random points into pure regions.
That is overfitting in its clearest form: perfect memory, no useful knowledge.
A reliable workflow
-
Split off the test set before model selection.
-
Fit a shallow baseline.
-
Plot the tree and trace several predictions manually.
-
Compare training and cross-validation scores.
-
Tune depth, leaf size, and pruning strength.
-
Evaluate the test set only after selecting the model.
-
Reject rules that are unstable or nonsensical in the application domain.
Run the plotted-tree example, verify its root Gini calculation, then repeat the noise experiment with max_depth values of 2, 4, 8, and None. Watch training accuracy rise—and decide where validation performance, stability, and readability matter more than memorization.