Overfitting and underfitting

In a school science lab, students record the distance travelled by an accelerating toy robot at different times. A straight line may miss its curved movement. A very flexible curve may pass through every recorded dot, including small measurement deviations, yet make worse estimates between those times.

Underfitting means the fitted procedure fails to capture important structure relevant to the task. Overfitting means it fits training-specific details that do not transfer well to separate records. The goal is generalization: useful predictions beyond the examples used for fitting, under the conditions we intend to evaluate.

We will preserve the existing lesson's numerical example and give it a school-lab setting. It is a controlled simulation: ideal distance is time squared, and five training measurements include artificial deviations. We know the reference relationship only because we constructed it; real datasets do not reveal a perfect underlying answer.

Review linear regression, mean absolute error and training and validation boundaries. The examples use those ideas to compare model flexibility fairly.

The five training pairs are (0 seconds, 0 metres), (1, 1.2), (2, 3.7), (3, 9.4) and (4, 15.5). The simulator's ideal relationship is distance = time², giving ideal distances 0, 1, 4, 9 and 16. The training deviations are 0, +0.2, −0.3, +0.4 and −0.5 metres.

These deviations stand in for recording noise: differences that a prediction procedure should not necessarily reproduce. We know which differences are artificial in this lesson. In real work, unusual observations can instead reveal a real subgroup or a faulty assumption, so investigate before dismissing them as noise.

The separate comparison times are 0.5, 1.5, 2.5 and 3.5 seconds. Their simulator reference distances are 0.25, 2.25, 6.25 and 12.25 metres. None of these four records is fitted by our models. They demonstrate predictions between training times, called interpolation.

The comparison references are noiseless while the training values contain deviations. That deliberate difference makes the mechanism clear, but is not a standard train/test experiment sampled from the same noisy population. We use them for visible teaching comparisons, not as proof of deployment accuracy.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The constructed measurements are included; no data file is needed.

Python · complete runnable example

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Constructed school-lab robot data: seconds and metres.
X = np.array([[0.], [1.], [2.], [3.], [4.]])
y = np.array([0., 1.2, 3.7, 9.4, 15.5])
new_X = np.array([[0.5], [1.5], [2.5], [3.5]])
new_y = new_X.ravel() ** 2
for degree in [1, 2, 4]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X, y)
    train_error = mean_absolute_error(y, model.predict(X))
    new_error = mean_absolute_error(new_y, model.predict(new_X))
    print("Degree", degree, "train MAE", round(train_error, 3), "new MAE", round(new_error, 3))

Expected output

Degree 1 train MAE 1.432 new MAE 1.0
Degree 2 train MAE 0.224 new MAE 0.103
Degree 4 train MAE 0.0 new MAE 0.306

PolynomialFeatures builds powers of the time input. Degree one supplies a straight-line representation. Degree two adds time², allowing a quadratic curve. Degree four also allows time³ and time⁴, making the fitted shape more flexible.

make_pipeline applies those features before LinearRegression. Each loop creates a fresh pipeline, fits the same five training pairs and predicts both the training times and the separate comparison times. The fitted coefficients change between degrees; the target measurements remain fixed.

mean_absolute_error averages the absolute differences between recorded or reference values and predictions. Both reported errors are in metres. Degree four reaches zero after rounding on training data, yet has larger new-point error than degree two. Training fit alone would select the wrong candidate for these references.

The new MAE label means error against our four constructed reference points; it does not claim an independent real-world test. Degree is a hyperparameter: comparisons on these points make them selection evidence, so they must not also be called a protected final test.

For the same training times 0, 1, 2, 3 and 4, a degree-two model predicts a + b × time + c × time². It is curved as a function of time, but linear in the fitted coefficients a, b and c. That is why LinearRegression can fit it after the powers are provided as features.

At two seconds, those features include 2 and 4; at three seconds, they include 3 and 9. The degree-four representation also includes 8 and 16 at two seconds, and 27 and 81 at three seconds. These are derived values from the same time measurement, not additional independently observed features.

The default PolynomialFeatures includes a constant column while LinearRegression also fits an intercept. That redundancy is harmless for this small numerical demonstration; later regularized examples use include_bias=False and let the estimator handle the intercept.

Five distinct times allow a degree-four polynomial to interpolate five supplied target values in this setting, within numerical precision. More freedom helps fit those dots. It does not ensure the intervening curve follows the process that generated them.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The constructed measurements are included; no data file is needed.

Python · complete runnable example

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Constructed school-lab robot measurements: seconds and metres.
X = np.array([[0.], [1.], [2.], [3.], [4.]])
y = np.array([0., 1.2, 3.7, 9.4, 15.5])
# Known simulator reference at different times; not real test measurements.
new_X = np.array([[0.5], [1.5], [2.5], [3.5]])
new_y = new_X.ravel() ** 2
for degree in [1, 2, 4]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X, y)
    predictions = model.predict(new_X)
    print("Degree:", degree)
    print("Reference:", new_y.tolist())
    print("Predictions:", np.round(predictions, 3).tolist())
    print("Absolute errors:", np.round(np.abs(new_y - predictions), 3).tolist())

Expected output

Degree: 1
Reference: [0.25, 2.25, 6.25, 12.25]
Predictions: [0.08, 4.0, 7.92, 11.84]
Absolute errors: [0.17, 1.75, 1.67, 0.41]
Degree: 2
Reference: [0.25, 2.25, 6.25, 12.25]
Predictions: [0.312, 2.375, 6.295, 12.072]
Absolute errors: [0.062, 0.125, 0.045, 0.178]
Degree: 4
Reference: [0.25, 2.25, 6.25, 12.25]
Predictions: [0.74, 2.059, 6.215, 12.759]
Absolute errors: [0.49, 0.191, 0.035, 0.509]

This complete program repeats the training measurements and four comparison references. At 0.5 seconds, the ideal simulator distance is 0.25 metres. Degree one predicts 0.08, degree two about 0.312, and degree four about 0.740. Their absolute errors are about 0.170, 0.062 and 0.490 metres.

The flexible polynomial passes through the recorded 0 and 1-second targets, but bends differently between them. That is a concrete way fitting deviations can hurt new estimates. The quadratic is not perfect either: it fitted noisy measurements and does not know the simulator's coefficients.

A smaller overall MAE does not require being best at every reference point. Compare the full error list, not one convenient row. np.abs takes absolute differences, and np.round rounds only for display; the earlier MAE calculation uses unrounded predictions.

Degree one has training MAE 1.432 metres and reference MAE 1.000. Its straight-line family misses the curved construction even on the data it fits. Here that supports an underfitting diagnosis. A restrictive assumption can cause this, but missing relevant features or an unsuccessful optimization process can also leave training error high.

Degree two lowers both errors, to 0.224 and 0.103. Degree four lowers training error to displayed zero while reference error rises to 0.306. Relative to degree two on this controlled comparison, extra flexibility has fitted training deviations at the expense of the new-point predictions.

A large training/validation gap is useful evidence to investigate, not a definition that proves overfitting by itself. Distribution changes, leakage, different measurement noise, small validation samples or mismatched metric calculations can produce confusing results. Check the evaluation setup before changing model size.

There is no required rule that training error must always be smaller than validation error. Our degree-one reference error is smaller than its training error because the particular records and targets differ. Error magnitudes also need context: compare a baseline and the task's acceptable error, not an arbitrary universal threshold.

When training error stays high, check that the features and targets are correct and the fit completed. Consider a model family capable of the needed relationship, more relevant features, or relaxing excessive constraints. For our robot, adding the squared time feature addresses the specific straight-line limitation.

When training is much better than representative validation, possible remedies include reducing flexibility, adding an appropriate penalty, reducing training-sensitive choices or obtaining more representative observations. A tree may need depth or leaf constraints; an iterative learner may support early stopping. These are different tools, not settings that every estimator implements.

More data can help a procedure separate repeatable structure from accidental details, but duplicated records or more data from the wrong population may not help. More features can improve representation or add noise. More training iterations can fix incomplete optimization or worsen training-specific fitting, depending on the method.

Use development validation or appropriate cross-validation to assess each change, and keep a final test separate. Do not simplify until a reported test score looks appealing.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The constructed measurements are included; no data file is needed.

Python · complete runnable example

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Constructed school-lab robot measurements: seconds and metres.
X = np.array([[0.], [1.], [2.], [3.], [4.]])
y = np.array([0., 1.2, 3.7, 9.4, 15.5])
# Known simulator reference at different times; not real test measurements.
new_X = np.array([[0.5], [1.5], [2.5], [3.5]])
new_y = new_X.ravel() ** 2
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge

for alpha in [0.001, 0.1, 10.0]:
    model = make_pipeline(PolynomialFeatures(4, include_bias=False),
                          StandardScaler(), Ridge(alpha=alpha)).fit(X, y)
    train_error = mean_absolute_error(y, model.predict(X))
    new_error = mean_absolute_error(new_y, model.predict(new_X))
    print("alpha:", alpha, "train MAE:", round(float(train_error), 3),
          "reference MAE:", round(float(new_error), 3))

Expected output

alpha: 0.001 train MAE: 0.167 reference MAE: 0.073
alpha: 0.1 train MAE: 0.339 reference MAE: 0.267
alpha: 10.0 train MAE: 1.933 reference MAE: 1.651

Regularization adds a preference or constraint that discourages some fitted solutions. Ridge balances squared training error with a penalty on squared coefficient magnitudes. Its alpha sets the penalty strength: increasing it places more emphasis on smaller coefficients.

This complete program retains degree-four powers but standardizes each derived feature before fitting Ridge. Scaling matters because the penalty compares coefficient magnitudes, which depend on feature units. The scaler learns only from the five training measurements. Neither the original polynomial nor this pipeline is selected by a protected test.

A small penalty produces nonzero training error and improves the four-reference comparison here. A much stronger penalty makes both errors worse. That shows why regularization strength needs evidence: stronger does not automatically mean better. These three alpha values are teaching choices, not recommendations for other datasets.

This penalty changes the fitting objective; it does not delete suspicious rows or give the estimator knowledge of the ideal relationship. See the Ridge reference for its objective and parameters.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The constructed measurements are included; no data file is needed.

Python · complete runnable example

import numpy as np
from sklearn.model_selection import KFold, learning_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import Ridge

# Separate simulation: 60 records with repeatable added measurement noise.
rng = np.random.default_rng(12)
X = rng.uniform(0, 4, size=(60, 1))
y = X.ravel() ** 2 + rng.normal(0, 0.4, size=60)
cv = KFold(n_splits=3, shuffle=True, random_state=12)
for degree in [1, 2]:
    model = make_pipeline(PolynomialFeatures(degree, include_bias=False),
                          StandardScaler(), Ridge(alpha=0.01))
    sizes, train_scores, valid_scores = learning_curve(
        model, X, y, cv=cv, train_sizes=[0.25, 0.5, 1.0],
        scoring="neg_mean_absolute_error", shuffle=True, random_state=12)
    print("Degree:", degree, "training sizes per fold:", sizes.tolist())
    print("Mean train MAE:", np.round(-train_scores.mean(axis=1), 3).tolist())
    print("Mean validation MAE:", np.round(-valid_scores.mean(axis=1), 3).tolist())

Expected output

Degree: 1 training sizes per fold: [10, 20, 40]
Mean train MAE: [0.79, 0.968, 0.976]
Mean validation MAE: [1.107, 1.054, 1.045]
Degree: 2 training sizes per fold: [10, 20, 40]
Mean train MAE: [0.27, 0.277, 0.295]
Mean validation MAE: [0.33, 0.307, 0.312]

A learning curve compares scores as the number of training records changes. This separate simulation generates 60 time values within 0–4 seconds and adds repeatable normally distributed noise with standard deviation 0.4 metres. Both training and validation targets now contain noise, unlike the earlier noiseless references.

Three cross-validation splits have 40 available training records each. The requested fractions use 10, 20 and 40 training records per fold, while its held-out validation portion stays separate. Each degree and size is freshly fitted. The fixed seeds reproduce this example; they do not create independent datasets from repeated evaluations.

The output reports mean MAE across folds. Negative scorer values are negated back to nonnegative metre errors. Degree one remains limited by its straight-line representation; degree two has lower validation errors in this simulation. Validation error need not fall monotonically at every increase in data size.

In practice, examine score variation too. Persistently poor training and validation performance can suggest a representation problem; a gap that narrows as training grows can suggest more representative data may help. Neither pattern is a guaranteed remedy. The official learning-curve guide distinguishes training-size curves from validation curves that vary a setting.

All four original reference times lie between recorded training times. Predicting beyond 0–4 seconds would be extrapolation. Different fitted shapes can diverge substantially outside that range; the displayed curve and small interpolation MAE do not establish reliable movement estimates there.

Our ideal robot rule is chosen for teaching. A real robot may change speed, meet friction, stop or have sensor problems. Student quiz scores likewise do not obey a known time-squared law. Do not transfer the simulation's formula into those applications just because the school setting is familiar.

A highly flexible model is not automatically overfitted, and a straight line is not automatically underfitted. With different data or a different task, either could be appropriate. Overfitting concerns transfer of fitted details; underfitting concerns missed useful structure under the chosen procedure.

When validation seems unexpectedly excellent, revisit the split and preparation for leakage. When it seems unexpectedly poor, examine target quality, new populations and missing features. A prettier curve cannot repair a flawed evaluation.

Your task: Keep the five times and four comparison times, but replace training targets by the exact simulator rule time². Predict which degrees can now reconstruct that rule. Must the most flexible candidate still have worse new-point error?

Python · complete practice program

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Constructed school-lab robot measurements: seconds and metres.
X = np.array([[0.], [1.], [2.], [3.], [4.]])
y = np.array([0., 1.2, 3.7, 9.4, 15.5])
# Known simulator reference at different times; not real test measurements.
new_X = np.array([[0.5], [1.5], [2.5], [3.5]])
new_y = new_X.ravel() ** 2
# Remove the artificial training deviations for this separate experiment.
y = X.ravel() ** 2
for degree in [1, 2, 4]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X, y)
    print("Degree", degree, "train MAE", round(float(mean_absolute_error(y, model.predict(X))), 3),
          "new MAE", round(float(mean_absolute_error(new_y, model.predict(new_X))), 3))

Worked answer

Degree 1 train MAE 1.6 new MAE 1.0
Degree 2 train MAE 0.0 new MAE 0.0
Degree 4 train MAE 0.0 new MAE 0.0

Degree one still cannot represent the quadratic relationship. Degrees two and four both have zero error after rounding on these training and comparison points. A degree-four model can represent a quadratic with its unnecessary higher-order coefficients effectively zero here.

This does not prove degree four will always generalize, or that removing all real-world variation is possible. It shows why degree alone is not a diagnosis: the relationship, observations, fitting method and separate evaluation all matter. For the noisy original data, degree four remains worse than degree two on the given references.

• Underfitting: important task structure remains uncaptured; inspect data, representation, constraints and fit quality.
• Overfitting: training-specific details hurt transfer; compare representative held-out predictions.
• Training perfection is insufficient: our noisy degree-four fit reaches zero training error but loses to degree two on separate reference points.
• Remedies need validation: complexity, regularization and more data can help or hurt depending on the cause.
• Keep the experiment honest: simulation references, development validation and protected tests have different roles.

Before moving on: explain the 0.5-second prediction differences, why excessive regularization can underfit, and why the noise-free degree-four result changes the diagnosis. Next, bias and variance will help describe how fitted predictions behave across different training samples.