PYTHON / MACHINE LEARNING WITH PYTHON
Decision trees and random forests
Train and tune decision trees and random forests in scikit-learn, compare train and test accuracy, and read feature importances.
What you will learn
- Fit a DecisionTreeClassifier and read the thresholds it learns
- Control overfitting with max_depth and min_samples_leaf
- Fit a RandomForestClassifier and explain why averaging helps
- Compare train vs test accuracy to diagnose overfitting
Understanding Decision trees and random forests
A decision tree classifies a point by asking a chain of yes/no questions: at each node it picks one feature and one threshold, and sends the sample left or right. The split it chooses is the one that best reduces impurity, usually Gini impurity, which is roughly the chance that a random sample from the node would be mislabeled if you guessed the node's majority class. Every split is chosen greedily, one at a time, with no lookahead, so the tree never revisits or undoes an earlier choice.
That greedy habit is why a single tree overfits: with no depth limit it keeps splitting until every leaf is pure, so it also splits on noise, and its training score hits 1.0 by construction rather than by merit. max_depth and min_samples_leaf are the brake; they force leaves to cover several samples, trading a perfect training fit for a test score that holds up on new data. You tune them by comparing scores on a held-out set, the same way you would tune k in k-nearest neighbours.
A random forest trains many deep trees and averages their votes, but that only helps if the trees disagree. Bootstrap sampling, giving each tree a random sample of the rows with replacement, plus looking at a random subset of features at every split, which defaults to the square root of the feature count, deliberately makes trees differ so that errors one tree makes are usually balanced by others. The cost is interpretability: you no longer have one readable tree, only feature_importances_, a rough ranking of which features the forest leaned on.
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
tree = DecisionTreeClassifier(random_state=0)
tree.fit(X_train, y_train)
forest = RandomForestClassifier(n_estimators=100, random_state=0)
forest.fit(X_train, y_train)
print(f"tree train: {tree.score(X_train, y_train):.3f} | test: {tree.score(X_test, y_test):.3f}")
print(f"forest train: {forest.score(X_train, y_train):.3f} | test: {forest.score(X_test, y_test):.3f}")
print(f"tree test errors: {(tree.predict(X_test) != y_test).sum()}")
print(f"forest trees: {len(forest.estimators_)}")A decision tree is a greedy sequence of single-feature threshold questions; a random forest is a majority vote of many trees trained on bootstrap samples and random feature subsets.
Worked examples
Reading a tree as rules
Fitting a tiny three-class dataset and printing the exact questions the tree learned.
from sklearn.tree import DecisionTreeClassifier, export_text
X = [[0], [1], [2], [3], [4], [5]]
y = [0, 0, 1, 1, 1, 2]
tree = DecisionTreeClassifier(max_depth=2, random_state=0)
tree.fit(X, y)
print(export_text(tree))
print(tree.predict([[1], [4.9]]))Example explained
Line 1export_text renders the fitted tree as readable rules, so you can see exactly which thresholds the search produced.
Line 2At each node the algorithm picks the threshold that most reduces Gini impurity; 1.50 and 4.50 are midpoints between neighbouring training values.
Line 3Predicting is just walking the rules: 1 goes left to class 0, while 4.9 goes right twice and lands in class 2.
Depth as a bias-variance knob
Fitting the same data with max_depth=1 and max_depth=2 to see what a single extra question buys.
from sklearn.tree import DecisionTreeClassifier
X = [[0], [1], [2], [3], [4], [5]]
y = [0, 0, 1, 1, 1, 2]
for depth in (1, 2):
tree = DecisionTreeClassifier(max_depth=depth, random_state=0)
tree.fit(X, y)
print(f"depth {depth}: {tree.score(X, y)}")Example explained
Line 1With max_depth=1 the tree may ask a single question, so it separates class 0 from the rest and the right leaf votes for its majority class, 1.
Line 2The sample at 5 has true class 2 but lands in that leaf, so only five of six predictions are right, giving 5/6 = 0.8333.
Line 3Depth 2 lets the tree ask one more question, isolating class 2, and the score reaches 1.0.
Important notes
Forests reduce variance, not bias: if the features carry little signal, a random forest will do poorly too, just like a single tree.
Beyond a few hundred trees, n_estimators adds runtime without adding accuracy; lowering it is the right way to speed up training.
Common mistakes
Assuming a training score of 1.0 means the model is good: an unpruned tree reaches 1.0 by memorizing the training set, so the test score is the only number that matters.
Tuning max_depth by watching the test score until it peaks: the test set becomes a second training set, and the reported accuracy turns optimistic. Use cross-validation instead.
Reading feature_importances_ as causal importance: correlated features split the credit arbitrarily, so a low score does not prove a feature is useless.
Try it yourself
Change, predict, then run
In an online editor, load the iris data, split with random_state=0, and fit a DecisionTreeClassifier(max_depth=2) and a RandomForestClassifier(n_estimators=100). Print both test accuracies, then change max_depth to 5 and explain which score moves and why.
Open the Python workspaceCheck your understanding
A single decision tree scores 1.000 on training and 0.82 on test; a random forest trained on the same data scores 0.97 on test. Which explanation is correct?
- The tree overfit by splitting until leaves were pure, and the forest averages many trees built on bootstrap samples and random feature subsets so individual errors cancel.
- The forest finds better split thresholds than the tree because each threshold is chosen by majority vote across the trees.
- The tree underfit, and the forest fixed it by manufacturing new features out of combinations of the originals.
- The forest reduces bias by training every tree on the full training set, so each tree sees more samples than the single tree did.
Show answer
The symptom is a perfect training score with a much weaker test score, which is the signature of high variance: the tree memorized the training set. Bagging, sampling rows with replacement, and random feature subsets exist to make the trees differ from each other, and averaging cancels errors that are not shared, a variance reduction rather than a bias fix. The tempting wrong option about full training sets gets the mechanism backwards: forests resample and subsample precisely so their trees are not identical.