Decision trees and random forests: learn rules and combine predictions

A library records an incoming visitor count and a staff label describing a period as “quiet” or “busy”. Can a model use the count to predict that category for another period? A decision tree learns questions about input features and follows their answers along branches until it reaches a prediction.

A question might be “Is the count at most five?” The starting question is the root. Each answer sends the observation along a branch; a final prediction location is a leaf. Training chooses the questions from labelled examples. We do not write the threshold into the training program ourselves.

A random forest combines many trees built with randomized training choices. It can reduce dependence on one tree’s quirks, but is not guaranteed to beat a single tree. We will first inspect six invented library records, then compare both models on the classic Iris dataset using separate test flowers. Review classification inputs and targets and held-out evaluation if needed.

Our six records pair one count with one known category. The count is the feature; the staff label is the target. They are invented so that the two groups are easy to inspect. A real “busy” label needs a consistent definition and may depend on staffing or other context, not count alone.

The count must be available when the category prediction is requested. If the application instead predicts tomorrow’s workload, a future observed count cannot be an input available today. Define the question before fitting; a good score for the wrong information timing is not useful evidence.

At prediction time we supply the count, not the unknown category. The examples are sparse: there are no observations at counts four, five or six. A tree still assigns them to a branch, but these training records alone do not verify those assignments.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.tree import DecisionTreeClassifier

# Invented library observations: count is known before classification.
X = [[1], [2], [3], [7], [8], [9]]
y = ["quiet", "quiet", "quiet", "busy", "busy", "busy"]
tree = DecisionTreeClassifier(max_depth=1, random_state=3).fit(X, y)
print("Threshold:", round(tree.tree_.threshold[0], 1))
print("Depth and leaves:", tree.get_depth(), tree.get_n_leaves())
print("Predictions:", tree.predict([[2], [8]]).tolist())

Expected output

Threshold: 5.0
Depth and leaves: 1 2
Predictions: ['quiet', 'busy']

X holds one count per row; the nested brackets give the rows-and-features shape. y keeps the label for each matching row. max_depth=1 allows one level of splitting, while random_state makes randomized choices repeatable for these inputs.

The fitted root threshold is five. A count of two satisfies “at most five” and enters the quiet leaf. Eight does not and enters the busy leaf. Depth one means one split along either path; two leaves means two final prediction locations, not two questions.

tree.tree_.threshold[0] inspects the root of this fitted tree. A larger tree has several split nodes, so reading only this value would not describe every decision. get_depth() and get_n_leaves() report the fitted structure. The DecisionTreeClassifier API documents these settings and methods.

A classifier tree compares candidate splits by how well they separate the training labels. Impurity measures how mixed the labels are within a group. All-quiet or all-busy groups are pure; mixed groups have positive impurity.

The default Gini impurity is one minus the sum of squared class fractions. Our root has three quiet and three busy records: 1 − (3/6)² − (3/6)² = 0.5. A split at five creates two pure groups, each with impurity zero, so their weighted impurity is zero.

A split at 2.5 would put two quiet records on one side and one quiet plus three busy records on the other. The mixed side has impurity 1 − (1/4)² − (3/4)² = 0.375. Weighting by group size gives (2/6 × 0) + (4/6 × 0.375) = 0.25. The split at five separates these particular labels better.

The threshold five is the midpoint between adjacent observed counts three and seven. It is not a real library policy discovered with certainty. Training typically makes locally useful split choices in sequence, rather than proving that the whole resulting tree is the best possible tree. The tree formulation explains the split objective.

At a leaf, classification predicts the class with the largest represented share. For unweighted training records, its probability estimate is the fraction of that leaf’s records in each class. A leaf with three quiet and one busy record would estimate quiet at 0.75 and busy at 0.25, then predict quiet.

Our first tree’s leaves are pure, so their winning class has estimated probability one. That means all training records reaching that leaf share the label. It does not guarantee that a new period reaching it will have the same outcome or that the estimates are calibrated.

The leaf prediction is the same for inputs following the same path. Counts two and four both reach the quiet leaf in this tree, even though four was not observed. Class probability columns follow classes_; do not assume the category you care about always appears first. We use unweighted examples here; sample or class weights can alter the represented shares.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.tree import DecisionTreeClassifier

# Invented observations with one quiet record among higher counts.
X = [[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]]
y = ["quiet", "quiet", "quiet", "quiet", "busy", "quiet",
     "busy", "busy", "busy", "busy"]
for limit in [1, None]:
    tree = DecisionTreeClassifier(max_depth=limit, random_state=3).fit(X, y)
    correct = sum(tree.predict(X) == y)
    print("Depth limit:", limit)
    print("Learned depth and leaves:", tree.get_depth(), tree.get_n_leaves())
    print("Correct training labels:", int(correct), "of", len(y))
    print("At 5.2 and 5.8:", tree.predict([[5.2], [5.8]]).tolist())

Expected output

Depth limit: 1
Learned depth and leaves: 1 2
Correct training labels: 9 of 10
At 5.2 and 5.8: ['busy', 'busy']
Depth limit: None
Learned depth and leaves: 3 4
Correct training labels: 10 of 10
At 5.2 and 5.8: ['busy', 'quiet']

This separate invented dataset repeats all ten records. One period with count six is labelled quiet among nearby busy periods. We have not established whether it is a recording error, a special staffing situation or a genuine pattern; calling it unusual does not justify deleting it.

The depth-one tree gets nine training labels correct and predicts busy at both 5.2 and 5.8. The unrestricted tree grows to depth three with four leaves, matches all ten training labels and changes the 5.8 prediction to quiet. It can isolate the count-six observation using extra questions.

The deeper tree’s better training result shows greater flexibility, not proven better generalization. There are no held-out targets for these two queries. To decide whether the extra splits help new observations, compare on separate data. This is the issue developed in overfitting and underfitting.

max_depth limits how many successive questions a path can contain. min_samples_leaf requires enough training observations in each resulting leaf. A leaf-size limit can stop a tree from isolating a single observation even when another depth level is available.

Pruning removes parts of an overly detailed tree according to a complexity trade-off. In scikit-learn, ccp_alpha controls cost-complexity pruning. Larger restrictions can also remove useful structure, so “simpler” is not automatically the right answer.

Use validation or cross-validation to compare restrictions against the intended prediction task. Keep final test data outside selection. A small tree is easier to inspect, but an understandable path is not proof that its labels are accurate, fair or causally justified.

A forest is an ensemble, a model that combines several models. In a typical random forest, each tree gets a bootstrap sample: a sample of training rows drawn with replacement, so a row can appear more than once and other rows can be absent.

For six source row positions, an illustrative bootstrap draw might be [0, 0, 2, 4, 5, 5]. It has six draws but only four distinct rows. The repeated positions retain their original features and labels. This is an example of sampling, not a printed sample from our later fitted forest and not a held-out evaluation.

The forest also considers a random subset of feature candidates at each split. With several features, one tree may consider petal width while another considers length. In our one-feature library example there is no alternative feature to sample, so row sampling supplies the main randomized difference. The full Iris comparison has four feature columns.

Combining less-identical trees can reduce dependence on one unstable fit. Trees trained on the same faulty labels or missing information still share those problems. A forest is not a sequence in which the second tree repairs the first; that is a different ensemble idea.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.ensemble import RandomForestClassifier

# Same six invented library observations, repeated for this program.
X = [[1], [2], [3], [7], [8], [9]]
y = ["quiet", "quiet", "quiet", "busy", "busy", "busy"]
forest = RandomForestClassifier(
    n_estimators=30, max_depth=2, random_state=3
).fit(X, y)
print("Number of trees:", len(forest.estimators_))
print("Class order:", forest.classes_.tolist())
print("Predictions:", forest.predict([[2], [8]]).tolist())
print("Probabilities:", forest.predict_proba([[2], [8]]).round(3).tolist())

Expected output

Number of trees: 30
Class order: ['busy', 'quiet']
Predictions: ['quiet', 'busy']
Probabilities: [[0.0, 1.0], [1.0, 0.0]]

The six original observations are repeated, so this program runs independently. n_estimators=30 requests thirty trees; max_depth=2 limits each tree. The class order is busy, then quiet, so the first probability belongs to busy.

With this specified seed, the two query rows get the same categories as the small tree. The probability rows say quiet for count two and busy for count eight. These simple queries and pure training groups make the example easy; agreement is not evidence that the tree and forest are equally useful for more varied inputs.

More trees generally make the ensemble’s aggregation more stable, but add training, prediction and storage work. They do not add new independent observations or automatically fix overfitting. Their depth and leaf restrictions still matter.

In scikit-learn’s classification forest, each tree gives class probabilities for a query. The forest averages the probabilities for each class, then selects the class with the largest average. This can differ from a simple majority of each tree’s winning label.

Two trees individually favor busy. But the average busy probability is (0.51 + 0.51 + 0.01) / 3, about 0.343. Average quiet probability is about 0.657, so this probability-averaging classifier predicts quiet. The third tree’s stronger estimate matters in the aggregate.

Regression forests instead average numerical tree predictions, such as price estimates. Averaging either kind of output does not automatically establish calibration or usefulness. The forest guide describes the ensemble and its aggregation.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=7, stratify=iris.target
)
# Fixed teaching settings, not selected by searching the test score.
models = {
    "Tree": DecisionTreeClassifier(max_depth=3, random_state=3),
    "Forest": RandomForestClassifier(n_estimators=100, max_depth=3, random_state=3)
}
print("Training and test rows:", len(y_train), len(y_test))
for name, model in models.items():
    model.fit(X_train, y_train)
    train_correct = int(np.sum(model.predict(X_train) == y_train))
    test_correct = int(np.sum(model.predict(X_test) == y_test))
    print(name, "training correct:", train_correct, "of", len(y_train))
    print(name, "test correct:", test_correct, "of", len(y_test))

Expected output

Training and test rows: 120 30
Tree training correct: 116 of 120
Tree test correct: 29 of 30
Forest training correct: 116 of 120
Forest test correct: 30 of 30

This separate experiment uses the classic packaged Iris dataset, all four flower measurements and three species. The stratified split reserves 30 of 150 records. Both classifiers receive exactly the same training and test rows, so the comparison is not confounded by changing the partition.

The stated tree correctly predicts 29 test species; the forest predicts 30. Both match 116 training labels. Equal training counts can therefore hide different predictions. One extra correct test flower is a result for this particular small split, not proof that the forest always wins.

No scaler is included: ordinary tree threshold comparisons do not depend on feature distances like KNN does. A consistent positive change of units preserves feature ordering and can usually be represented by an equivalent threshold, though floating-point or implementation details can affect exact fits.

The chosen depths and tree count are fixed teaching settings. To compare candidate settings in real development, use validation within training data, then evaluate the selected procedure. See model-setting selection and metrics suited to the error costs.

Your task: Replace the prediction query in this complete first-tree program with counts four, five and six. Predict all three labels, including the exact-threshold case, before running it.

Python · complete program to modify

from sklearn.tree import DecisionTreeClassifier

# Invented library observations: count is known before classification.
X = [[1], [2], [3], [7], [8], [9]]
y = ["quiet", "quiet", "quiet", "busy", "busy", "busy"]
tree = DecisionTreeClassifier(max_depth=1, random_state=3).fit(X, y)
print("Threshold:", round(tree.tree_.threshold[0], 1))
print("Depth and leaves:", tree.get_depth(), tree.get_n_leaves())
print("Predictions:", tree.predict([[2], [8]]).tolist())

Original output

Threshold: 5.0
Depth and leaves: 1 2
Predictions: ['quiet', 'busy']

Worked answer

Change the last statement to:

print("Predictions:", tree.predict([[4], [5], [6]]).tolist())

Predictions: ['quiet', 'quiet', 'busy']

Four and five satisfy count ≤ 5 and go to the quiet leaf. Six follows count > 5 and goes to busy. None of these values was an observed training count. The learned question still provides an answer, but only new labelled records could check whether that answer is useful.

These scikit-learn examples use numeric features and complete data. String categories generally need encoding for these estimators. Supported missing-value behavior depends on the estimator, configuration and library version; do not assume every tree library accepts every blank value. Our mixed-column Titanic project prepares its inputs explicitly.

A tree path describes what this model did, not why the real outcome occurred. A feature-importance ranking is also not a causal explanation. Built-in impurity-based importance can favor features with many possible split values; related features can distribute apparent importance in misleading ways.

Held-out permutation importance measures how a chosen evaluation score changes when one feature’s values are shuffled. It answers a predictive question about that fitted model and evaluation data, not a causal question, and correlated features can still complicate the interpretation. See the interpretation guide for the limits.

A tiny tree can be inspected directly; a large forest is harder to summarize and takes more resources. Both need reliable labels, appropriate inputs and evaluation matching their use. Regression trees also make constant predictions within leaves, so they do not extend a rising price relationship beyond training values like a linear formula would.

The library tree learned a count threshold from labelled examples. A deeper tree could match an unusual training record, while a forest combined randomized trees instead of relying on one fit.

• Trace the path: a root question sends an input along branches to a leaf prediction.
• Understand fitting: useful splits reduce label mixing in the training groups.
• Control detail: depth, leaf size and pruning affect how closely a tree follows the examples.
• Read the forest: bootstrap rows and feature choices create diverse trees; scikit-learn averages class probabilities before selecting a category.
• Check new data: compare procedures on shared held-out observations and keep selection outside the final test.
• Respect interpretation: a path or importance ranking explains the model’s behavior, not a proven real-world cause.

Before moving on: trace the threshold case, explain why a better training fit may not help new periods and calculate the three-tree probability average. Next, we will choose evaluation measures that match the kinds of prediction errors that matter.