Bias and variance

In the previous school-lab example, a model estimated an accelerating robot's distance from time. Now imagine recording its motion again: the sensor readings change slightly, and we fit the same procedure afresh. We still ask the same question—where is it at 2.5 seconds—but the fitted answers may move.

Bias describes the difference between a procedure's average prediction across possible training datasets and the underlying average outcome at a given input. Variance describes how much its predictions change across those training datasets. A procedure can be consistently off-target, variable around the right answer, or both.

These ideas concern a repeated fitting procedure, not just one trained model. They help explain the underfitting and overfitting patterns we already compared. They do not create a universal rule that larger models must always perform worse or that one setting perfectly balances every task.

We will keep the original one-record sensitivity example, then repeat a controlled simulation with a known time² relationship. Knowing the reference lets us inspect mean offsets honestly; real data rarely provides that exact reference. Review squared error and error units before the worked calculation.

Our input times are always 0, 1, 2, 3 and 4 seconds. The ideal simulated distance is time² metres. In each repetition, add a fresh independent noise value to each of the five training targets, then fit each model again. Query all models at 2.5 seconds, whose known reference is 6.25 metres.

This is a fixed-design experiment: the training times do not change, but their measured targets do. Other experiments might also resample times, students or entire datasets. Bias and variance depend on that sampling process as well as the estimator, features and sample size.

The generated sensor noise has mean zero and standard deviation 0.5 metres. Some individual deviations can be larger or negative; this is a mathematical noise model, not a calibrated account of a real sensor. We repeat it 30 times with a fixed seed to make the example inspectable.

Every model sees the same five noisy targets within a repetition. We hold the query fixed so that spread reflects changes in fitted predictions, rather than mixing prediction changes caused by different input times. The plotted vertical staggering is only layout, not another measured quantity.

Suppose a procedure tends to predict about 8 metres at 2.5 seconds across fresh measurements, when the underlying average is 6.25. Its average prediction is about 1.75 metres too high at that query. Even if its individual predictions are tightly grouped, that persistent offset matters.

A straight line cannot exactly represent the time² curve. With our five fixed training times and ideal targets 0, 1, 4, 9 and 16, its fitted prediction at 2.5 is 8 metres. Zero-mean target noise does not remove that representation mismatch in the procedure's expected prediction.

Formal bias uses an expectation over possible training datasets. A mean from 30 simulated fits is a finite estimate, not the exact expectation. We will call its difference from the known reference a mean offset when reporting the numbers, rather than pretending we observed every possible dataset.

This statistical use of bias differs from unfair treatment of people or an unrepresentative data collection process. Those concerns can also matter, but a bias-variance calculation does not measure social fairness or establish a cause of discrimination.

At the same 2.5-second input, fresh fits can disagree because different measurements influence the fitted coefficients. Prediction variance measures the mean squared distance of those predictions from their own average. It is measured here in metres squared, while their standard deviation would be in metres.

This is not the variance of the robot's input times, the spread of all its travelled distances, or the variance explained by PCA. It specifically concerns repeated fitted predictions at a fixed query. A single prediction does not have the across-training-datasets variance by itself.

A flexible curve can react strongly to small target changes, while a restrictive procedure may move less. Low sensitivity is useful only alongside an appropriate average prediction. A stable but systematically wrong estimate still has substantial error.

Changing a random seed on an unchanged dataset probes randomness in a learner or split. Collecting a fresh dataset probes sampling sensitivity. They are different sources of variation; deterministic LinearRegression can still have prediction variance across fresh training targets.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. All simulated measurements are included or generated here; no data file is needed.

Python · complete runnable example

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

X = np.array([[0.], [1.], [2.], [3.], [4.]])
base_y = X.ravel() ** 2
for degree in [1, 4]:
    predictions = []
    for offset in [-0.5, 0.0, 0.5]:
        y = base_y.copy()
        y[2] += offset
        model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X, y)
        predictions.append(float(model.predict([[2.5]])[0]))
    print("Degree", degree, "prediction spread:", round(max(predictions) - min(predictions), 3))

Expected output

Degree 1 prediction spread: 0.2
Degree 4 prediction spread: 0.703

The program repeats the complete five-time table. Its ideal targets are 0, 1, 4, 9 and 16. It changes only the target at row position 2, corresponding to two seconds: 3.5, 4.0 or 4.5 metres. Each changed table receives a fresh fit, and every fit predicts at 2.5 seconds.

The degree-one prediction range is 0.200 metres; the degree-four range is about 0.703 metres. max(predictions) - min(predictions) calculates a range, not variance. It reveals greater reaction to this specific perturbation for the flexible model.

This deliberately chosen change is not a sample of complete independent datasets. It does not measure the full theoretical bias, variance or irreducible noise. Nor does its larger range prove that degree four is always worse; the next program changes all five training targets under a stated noise model.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. All simulated measurements are included or generated here; no data file is needed.

Python · complete runnable example

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Repeat the same school-lab times with fresh simulated sensor noise.
X = np.array([[0.], [1.], [2.], [3.], [4.]])
query = np.array([[2.5]])
reference = 2.5 ** 2
rng = np.random.default_rng(19)
predictions = {degree: [] for degree in [1, 2, 4]}
for repeat in range(30):
    y = X.ravel() ** 2 + rng.normal(0.0, 0.5, size=len(X))
    for degree in predictions:
        model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X, y)
        predictions[degree].append(float(model.predict(query)[0]))
print("Reference at 2.5 seconds:", reference)
for degree, values in predictions.items():
    p = np.array(values)
    print("Degree:", degree, "fresh fits:", len(p))
    print("Mean prediction:", round(float(p.mean()), 3))
    print("Mean minus reference:", round(float(p.mean() - reference), 3))
    print("Prediction variance:", round(float(p.var()), 3))

Expected output

Reference at 2.5 seconds: 6.25
Degree: 1 fresh fits: 30
Mean prediction: 7.996
Mean minus reference: 1.746
Prediction variance: 0.084
Degree: 2 fresh fits: 30
Mean prediction: 6.209
Mean minus reference: -0.041
Prediction variance: 0.12
Degree: 4 fresh fits: 30
Mean prediction: 6.284
Mean minus reference: 0.034
Prediction variance: 0.177

rng.normal(0.0, 0.5, size=len(X)) supplies five new noise values on each repetition. We add them to the five ideal distances, then construct a fresh pipeline for each degree. predictions collects 30 answers per procedure; each answer comes from its own fit.

p.mean() averages those answers. Subtracting 6.25 gives the signed mean offset. p.var() uses the population denominator 30 to describe this finite collection's squared spread. It is not an unbiased estimate obtained by dividing by 29.

The line averages about 7.996 metres, well above the reference, with variance about 0.084 m². Degree two averages about 6.209 with variance about 0.120 m²; degree four averages about 6.284 with variance about 0.177 m². Those are the exact collections shown in the introduction, rounded for display.

The average offset of degree four happens to be smaller than degree two's here, but its larger spread raises its mean squared reference error. Thirty repeats at one query do not establish rankings across every input, noise level or training population. The plotted mean is a summary for analysis, not a deployed ensemble trained from data we normally have available.

Use the same 30 predictions at 2.5 seconds and reference 6.25 metres. For each prediction, subtract the reference and square the difference, then average. This gives the collection's mean squared reference error, in m².

The exact finite-collection identity is: mean squared reference error = squared mean offset + population prediction variance. A prediction's error can be written as its deviation from the prediction mean plus that mean's offset from the reference. Squaring and averaging makes the cross term vanish because deviations around their mean sum to zero.

For degree one, the terms are approximately 3.047 + 0.084 = 3.131 m². For degree two, they are about 0.002 + 0.120 = 0.122 m². For degree four, about 0.001 + 0.177 = 0.178 m². The code uses full precision; rounded printed terms can differ slightly from the rounded sum.

Under repeated sampling, the analogous squared-loss decomposition uses theoretical squared bias and variance. Our finite arithmetic shows the relationship clearly, while its estimated terms still reflect only these generated fits. This additive identity is for squared error; it is not the same decomposition for MAE, classification accuracy or F1.

Suppose a new recorded target at 2.5 seconds equals the ideal 6.25 plus independent, mean-zero sensor noise with standard deviation 0.5 metres. A procedure that predicts exactly 6.25 still cannot anticipate that particular random deviation. Its expected squared error against the noisy recorded target is the noise variance, 0.5² = 0.25 m².

Under these assumptions, expected squared error adds that noise variance to the squared bias and prediction variance. Independence from the training process and the zero conditional mean of target noise are important: they make the noise cross terms vanish. We specify them for this simulation rather than applying the formula to any arbitrary error source.

Irreducible noise is uncertainty that remains for the current inputs, target and observation process. Better measurement equipment, relevant extra inputs or a different target definition can change what is predictable. “Irreducible” does not mean every source of error is permanently beyond improvement.

In real data, the underlying conditional mean and noise level are usually unknown. One training/validation split cannot separately reveal exact theoretical bias, variance and noise. It can supply performance evidence and diagnostic clues, which still need careful interpretation.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. All simulated measurements are included or generated here; no data file is needed.

Python · complete runnable example

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

# Repeat the same school-lab times with fresh simulated sensor noise.
X = np.array([[0.], [1.], [2.], [3.], [4.]])
query = np.array([[2.5]])
reference = 2.5 ** 2
rng = np.random.default_rng(19)
predictions = {degree: [] for degree in [1, 2, 4]}
for repeat in range(30):
    y = X.ravel() ** 2 + rng.normal(0.0, 0.5, size=len(X))
    for degree in predictions:
        model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X, y)
        predictions[degree].append(float(model.predict(query)[0]))
# Future target noise is independent, mean zero, with standard deviation 0.5 m.
noise_variance = 0.5 ** 2
for degree, values in predictions.items():
    p = np.array(values)
    squared_offset = (p.mean() - reference) ** 2
    spread = p.var()
    reference_mse = np.mean((p - reference) ** 2)
    print("Degree:", degree)
    print("Squared mean offset:", round(float(squared_offset), 3))
    print("Prediction variance:", round(float(spread), 3))
    print("Reference MSE:", round(float(reference_mse), 3))
    print("Identity holds:", bool(np.isclose(reference_mse, squared_offset + spread)))
    print("Expected noisy-target MSE for these fits:",
          round(float(reference_mse + noise_variance), 3))

Expected output

Degree: 1
Squared mean offset: 3.047
Prediction variance: 0.084
Reference MSE: 3.131
Identity holds: True
Expected noisy-target MSE for these fits: 3.381
Degree: 2
Squared mean offset: 0.002
Prediction variance: 0.12
Reference MSE: 0.122
Identity holds: True
Expected noisy-target MSE for these fits: 0.372
Degree: 4
Squared mean offset: 0.001
Prediction variance: 0.177
Reference MSE: 0.178
Identity holds: True
Expected noisy-target MSE for these fits: 0.428

This complete program reproduces all 30 fits; it does not depend on variables from an earlier section. np.isclose checks the squared-error identity within floating-point tolerance. It compares mean squared error against the known noiseless reference, not against the training targets.

The last line per degree adds the specified 0.25 m² future-noise variance. Its meaning is precise: choose equally among these 30 fitted predictors, then average over an independent new target-noise draw. That conditional expected error is calculated analytically; we did not collect real test targets or measure a deployed error rate.

For the entire population of possible training datasets, the corresponding expectation needs theoretical quantities or enough appropriate repeated-sampling evidence. The finite simulation's results are an illustration and estimate, not a claim that the library has directly measured formal bias from an ordinary dataset.

The official bias-variance example also uses a controlled data-generating process so it can examine the separate squared-error terms.

A restrictive family can have a systematic mismatch; increasing useful flexibility can reduce that mismatch while increasing sensitivity to limited noisy observations. Regularization can make predictions less sensitive but introduce more average offset. The helpful balance depends on the relationship, sample size, noise, features and fitting procedure.

In our simulation, degree two has the smallest mean squared reference error of the three. That is evidence for this setting at this query, not a command to use quadratic models everywhere. There is no universal curve that forces bias to decrease and variance to increase monotonically with every model-size change.

More representative data can reduce sampling sensitivity for some procedures. Better features or a better model family may address systematic misses. The regularization and learning-curve examples show how changes can help or hurt, so validate the total performance for the actual task.

Bagging averages models fitted to resampled training records, often stabilizing unstable estimators such as trees. Its members share data and can be strongly correlated, limiting variance reduction; it does not automatically remove systematic bias. A random forest uses additional tree diversity. Our independent simulated measurement repetitions are not a bagging implementation.

Your task: Three separate fitted predictors give 5.5, 6.25 and 7.0 metres at the same query. The known average outcome is 6.25. Does the correct average eliminate squared prediction error? Calculate the mean offset, population variance and expected noisy-target MSE with independent noise variance 0.25 m².

Python · complete arithmetic practice

import numpy as np

# Three illustrative predictions from separate fits at one fixed input.
p = np.array([5.5, 6.25, 7.0])
reference = 6.25
noise_variance = 0.5 ** 2
mean_offset = p.mean() - reference
print("Mean prediction:", float(p.mean()))
print("Squared mean offset:", float(mean_offset ** 2))
print("Prediction variance:", float(p.var()))
print("Reference MSE:", float(np.mean((p - reference) ** 2)))
print("Expected noisy-target MSE:", float(mean_offset ** 2 + p.var() + noise_variance))

Worked answer

Mean prediction: 6.25
Squared mean offset: 0.0
Prediction variance: 0.375
Reference MSE: 0.375
Expected noisy-target MSE: 0.625

The mean offset is zero, but the individual fits differ from the reference by −0.75, 0 and +0.75 metres. Their squared deviations average to (0.5625 + 0 + 0.5625) ÷ 3 = 0.375 m². Adding 0.25 m² independent target-noise variance gives 0.625 m² expected error for an equally chosen predictor.

This finite example uses the population denominator three so its identity holds exactly. A different variance estimator or an accuracy metric is a different calculation. Predicting the average 6.25 as a separate ensemble would remove this collection's spread for that query, but that is a different prediction procedure and does not remove future target noise.

High training and validation errors can suggest a restrictive representation, but also poor features, incorrect targets or unsuccessful fitting. Low training error with poorer validation can suggest sensitivity to training details, but also a changed population or a flawed split. These patterns are clues, not exact measurements of bias and variance.

A cross-validation score's spread is not this theoretical prediction variance. Fold scores summarize different held-out outcomes and are dependent because training portions overlap. Use appropriate group, time or ordinary folds and report what was actually calculated.

Compare candidate procedures on development validation and keep a protected final evaluation. Investigate data quality and leakage before attributing every performance difference to model flexibility.

Bias and variance can differ from one query or subgroup to another. Our single-time diagram does not summarize all robot motion, future conditions or classification tasks. The next lessons apply the evaluation habits to projects where the underlying relationship is not supplied.

• Bias concerns an average: compare the repeated procedure's mean prediction with the underlying conditional mean.
• Variance concerns fresh fits: keep the query fixed and inspect how training-sample changes move predictions.
• Squared loss separates terms: squared bias, prediction variance and independent target-noise variance have distinct roles.
• Know what you measured: a perturbation range, finite simulation variance and fold-score spread are different quantities.
• Improve with evidence: compare total task performance, choose an appropriate representation and protect final evaluation.

Before moving on: explain why the stable degree-one predictions still miss the robot's distance, calculate the three-prediction practice variance, and state the assumptions behind adding 0.25 m² of noise. Next, we will build an explained Titanic survival project with real data and an unknown underlying relationship.