Cross-validation

A school wants to estimate quiz points from study hours. A model might look good on one held-out group simply because that group was easy to predict. Before trusting the procedure, we want to check it on several different held-out groups.

Cross-validation repeats training and evaluation using different training/validation partitions. In k-fold cross-validation, divide the development records into k portions. Train a fresh model on all but one portion, score it on the remaining portion, then repeat with a different portion held out. A fold is one such training-and-validation round.

We will use nine invented independent students, A through I, and three folds. Each round trains on six students and validates on three. A separate final-test group, J through L, stays outside all these comparisons. The small invented scores explain the mechanics; they do not establish how real students learn.

Start with training, validation and test roles, linear regression and mean absolute error. We will explain the splitting tools as we use them.

Development data is the data available for fitting and comparing candidate procedures. Cross-validation rotates training and validation roles within that development set. It does not rotate the protected final test into training.

Validation scores help choose models, settings and representations. Once those decisions are made, a final test estimates performance on a separate held-out set. Using the test repeatedly to revise the procedure turns it into another selection set and weakens the meaning of the reported result.

The rehearsal analogy is useful: each fold checks a newly prepared version using records it did not fit on. But this is not one model gradually improving after each fold. Each round starts a fresh fit, and the folds' fitted parameters can differ.

In our illustration, validation students are marked with both a label and a different color. Students J, K and L remain in their separate test box throughout. Cross-validation needs validation targets to calculate supervised scores, but those targets must not enter that round's fitting.

Our shuffled partitions are fixed for the examples: fold 1 validates B, E and G; fold 2 validates A, C and D; fold 3 validates F, H and I. Each round trains on the other six students. Within a round, training and validation records do not overlap.

A student can validate in one round and train in another. That is the intended reuse of k-fold cross-validation: each development record is held out once, so every validation prediction comes from a model that did not fit that record. It is not permission to train on a validation row before predicting it in the same round.

KFold(n_splits=3, shuffle=True, random_state=2) produces these partitions. Shuffling assigns rows to portions without preserving their original order. The fixed seed makes this teaching example repeatable; it does not make the split inherently fair or representative.

Ordinary shuffled folds are suitable only when mixing records matches the application. If rows repeat the same student or represent successive weeks, the split needs extra structure. We will examine both cases after following an ordinary fold.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import LinearRegression

X = np.arange(1, 10).reshape(-1, 1)
y = 10 * X.ravel() + 40
cv = KFold(n_splits=3, shuffle=True, random_state=2)
scores = cross_val_score(LinearRegression(), X, y, cv=cv, scoring="neg_mean_absolute_error")
print("Fold MAE:", [round(float(-v), 6) for v in scores])
print("Mean MAE:", round(float(-scores.mean()), 6))

Expected output

Fold MAE: [0.0, 0.0, 0.0]
Mean MAE: 0.0

The feature contains study hours 1 through 9. Here the target is constructed as exactly 10 × hours + 40, rather than observed from a real quiz. Each fold's linear regression can recover that relationship from its six training rows, so all displayed validation errors are zero.

reshape(-1, 1) creates nine rows with one feature; ravel() makes a one-dimensional target array. cross_val_score performs the separate fits and returns one score per fold. It does not leave the supplied estimator as a final model fitted on all nine rows.

The scorer name neg_mean_absolute_error follows scikit-learn's larger-is-better score convention: it returns negative MAE. Negating each returned value reports the familiar nonnegative error magnitude. Perfect toy data explains the mechanics, but it is a poor demonstration of variation between folds; the next example supplies that variation.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.model_selection import KFold
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Nine independent invented students: hours and quiz points out of 150.
X = np.arange(1, 10, dtype=float).reshape(-1, 1)
y = np.array([52., 57., 73., 79., 93., 97., 112., 118., 134.])
ids = np.array(list("ABCDEFGHI"))
cv = KFold(n_splits=3, shuffle=True, random_state=2)
for fold, (train, valid) in enumerate(cv.split(X), start=1):
    model = LinearRegression().fit(X[train], y[train])
    predictions = model.predict(X[valid])
    print("Fold", fold, "train:", " ".join(ids[train]))
    print("Validate:", " ".join(ids[valid]))
    print("Predictions:", np.round(predictions, 3).tolist())
    print("Actual:", y[valid].tolist())
    print("MAE:", round(float(mean_absolute_error(y[valid], predictions)), 3))

Expected output

Fold 1 train: A C D F H I
Validate: B E G
Predictions: [60.94, 90.523, 110.246]
Actual: [57.0, 93.0, 112.0]
MAE: 2.724
Fold 2 train: B E F G H I
Validate: A C D
Predictions: [47.011, 68.232, 78.843]
Actual: [52.0, 73.0, 79.0]
MAE: 3.305
Fold 3 train: A B C D E G
Validate: F H I
Predictions: [101.8, 122.486, 132.829]
Actual: [97.0, 118.0, 134.0]
MAE: 3.486

This complete program changes the invented targets to 52, 57, 73, 79, 93, 97, 112, 118 and 134 points, in student order A through I. The quiz has a maximum of 150 points. Hours still run from 1 through 9; the points are no longer an exact straight line.

cv.split(X) yields two arrays of zero-based row positions for each fold. Indexing X[train] and y[train] supplies only training records. A new LinearRegression() is constructed inside the loop, and predict(X[valid]) evaluates the held-out rows.

The validation IDs, predicted points and actual points appear in matching order. A different model is fitted each time; we do not copy the previous round's fitted line. The final-test students are absent from this program.

Repeat the records needed for this calculation: training students A, C, D, F, H and I have (hours, points) pairs (1, 52), (3, 73), (4, 79), (6, 97), (8, 118) and (9, 134). Validation students B, E and G have (2, 57), (5, 93) and (7, 112).

Fitting the six training pairs gives approximately predicted points = 41.2171 + 9.86121 × hours. For B's two hours, that line predicts 60.9395 points. Its absolute error is |57 − 60.9395| ≈ 3.9395 points. E and G have absolute errors of approximately 2.4769 and 1.7544 points.

The first-fold MAE is their average: (3.9395 + 2.4769 + 1.7544) ÷ 3 ≈ 2.724 points. The library calculates using full precision before we round for display. The three errors are not training errors: B, E and G were excluded from this fit.

In the next round, B, E and G become eligible for training and A, C and D supply the validation score. That new fit can have a different slope and intercept. Cross-validation evaluates the repeated fitting procedure rather than one fixed set of parameters.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.model_selection import KFold, cross_validate, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.dummy import DummyRegressor

# Nine independent invented students: hours and quiz points out of 150.
X = np.arange(1, 10, dtype=float).reshape(-1, 1)
y = np.array([52., 57., 73., 79., 93., 97., 112., 118., 134.])
cv = KFold(n_splits=3, shuffle=True, random_state=2)
procedure = make_pipeline(StandardScaler(), LinearRegression())
result = cross_validate(procedure, X, y, cv=cv,
                        scoring="neg_mean_absolute_error", return_estimator=True)
errors = -result["test_score"]
means = [float(model.named_steps["standardscaler"].mean_[0])
         for model in result["estimator"]]
baseline_errors = -cross_val_score(DummyRegressor(strategy="mean"), X, y,
                                   cv=cv, scoring="neg_mean_absolute_error")
print("Fold training-hour means:", np.round(means, 3).tolist())
print("Fold MAE:", np.round(errors, 3).tolist())
print("Mean MAE:", round(float(errors.mean()), 3))
print("Fold spread (population SD):", round(float(errors.std()), 3))
print("Baseline mean MAE:", round(float(baseline_errors.mean()), 3))

Expected output

Fold training-hour means: [5.167, 6.167, 3.667]
Fold MAE: [2.724, 3.305, 3.486]
Mean MAE: 3.171
Fold spread (population SD): 0.325
Baseline mean MAE: 30.37

This program repeats the same nine hours and noisy targets. make_pipeline connects StandardScaler to LinearRegression. Cross-validation fits a fresh copy of the complete procedure in each fold: first learn that fold's scaling statistics, then fit the regression using its transformed training rows.

cross_validate can return the fitted fold models when return_estimator=True. Their scaler means differ because the training students differ. result["test_score"] is the library's name for these held-out fold validation scores; it does not mean our protected final-test students were evaluated.

Scaling does not improve this unregularized one-feature linear regression's predictions merely by changing its units. It is included to make the learned preprocessing boundary visible. For scaling, imputation, PCA or feature selection, fitting once on all development rows before cross-validation would let validation rows influence the representation.

DummyRegressor(strategy="mean") predicts the training-fold mean target regardless of hours. It is fitted separately on the same partitions, so its larger MAE provides a meaningful simple comparison here. A lower error on this invented dataset does not establish real-world reliability.

The noisy example's fold MAEs round to 2.724, 3.305 and 3.486 points. Their mean is about 3.171 points. Report the metric and units, the splitter, the number of folds and the individual values so another reader knows what that summary describes.

Our folds each contain three validation records, so averaging their MAEs is also the pooled average of the nine held-out absolute errors. With unequal fold sizes, an unweighted fold mean and a record-weighted pooled error can differ. Nonlinear metrics such as F1 need additional care: averaging fold scores need not equal calculating one metric from pooled predictions.

errors.std() reports the population standard deviation of the three displayed fold errors. It describes variation across these rounds, not a guaranteed interval for future performance. Fold scores are dependent because their training sets overlap; do not turn their spread into a claimed confidence interval without an appropriate statistical method.

One unusually poor fold can reveal a difficult subgroup, a data problem or a mismatch between train and validation distributions. Inspect the records and deployment question instead of choosing a random seed merely because it gives a nicer average.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.model_selection import KFold, StratifiedKFold

# Separate classification example: 0 = no extra practice, 1 = extra practice.
# These are invented recorded targets, not categories derived from the regression scores.
X = np.arange(12).reshape(-1, 1)
y = np.array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1])
for name, splitter in [("Ordinary", KFold(n_splits=3)),
                       ("Stratified", StratifiedKFold(n_splits=3))]:
    counts = [np.bincount(y[valid], minlength=2).tolist()
              for _, valid in splitter.split(X, y)]
    print(name, "validation [class 0, class 1]:", counts)

Expected output

Ordinary validation [class 0, class 1]: [[4, 0], [4, 0], [1, 3]]
Stratified validation [class 0, class 1]: [[3, 1], [3, 1], [3, 1]]

This is a separate classification task with 12 invented recorded labels. Nine records have class 0 and three have class 1. The labels are deliberately ordered to show a failure of ordinary unshuffled folds: one validation portion contains all three class-1 records, while another fold's training data has no class-1 example at all.

StratifiedKFold distributes the classes so each validation portion here has three class-0 records and one class-1 record. np.bincount(..., minlength=2) counts both labels, including a zero when a class is absent. Stratification uses class membership to construct the split, not to fit validation records into that fold's model.

Preserving proportions makes class-based evaluation more workable; it does not repair biased data, guarantee model accuracy or prevent repeated-person leakage. A very rare class may not support the requested number of folds. Choose the metric and fold count with the available examples in mind; a zero positive count can make some metrics undefined.

See the StratifiedKFold reference for its class-balance behavior. When classes and related-person groups both matter, consider a splitter designed to respect both, rather than treating stratification alone as sufficient.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.model_selection import GroupKFold

# Two records for each student; the deployment question concerns new students.
X = np.arange(8).reshape(-1, 1)
student = np.array(["A", "A", "B", "B", "C", "C", "D", "D"])
cv = GroupKFold(n_splits=2)
for fold, (train, valid) in enumerate(cv.split(X, groups=student), start=1):
    overlap = set(student[train]) & set(student[valid])
    print("Fold", fold, "training students:", sorted(set(student[train].tolist())))
    print("Validation students:", sorted(set(student[valid].tolist())))
    print("Student overlap:", bool(overlap))

Expected output

Fold 1 training students: ['A', 'C']
Validation students: ['B', 'D']
Student overlap: False
Fold 2 training students: ['B', 'D']
Validation students: ['A', 'C']
Student overlap: False

Now each of four students has two observations, for eight rows in total. If the goal is to predict for a student never seen before, splitting rows independently could let a student's first observation train the model while their second observation validates it. Familiar-person patterns would make that an easier and different question.

GroupKFold receives student IDs through groups and keeps each student entirely on one side of a fold. The intersection check confirms no student ID appears in both training and validation. There are only four distinct groups, so requesting more group folds than that is not possible.

The printed row values are placeholders for inspecting the split; no predictor or score is fitted in this program. If deployment instead concerns a later observation of an existing student, preserving time within students may be more appropriate. Choose the split for the future situation being evaluated.

Final-test separation must respect the same grouping rule. See the GroupKFold reference; its group boundary does not automatically impose time ordering.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

# Eight equally spaced weeks, sorted from oldest to newest.
weeks = np.arange(1, 9)
cv = TimeSeriesSplit(n_splits=3, test_size=2)
for fold, (train, valid) in enumerate(cv.split(weeks), start=1):
    print("Fold", fold, "train weeks:", weeks[train].tolist())
    print("Validate weeks:", weeks[valid].tolist())
    print("Training ends before validation:", bool(weeks[train].max() < weeks[valid].min()))

Expected output

Fold 1 train weeks: [1, 2]
Validate weeks: [3, 4]
Training ends before validation: True
Fold 2 train weeks: [1, 2, 3, 4]
Validate weeks: [5, 6]
Training ends before validation: True
Fold 3 train weeks: [1, 2, 3, 4, 5, 6]
Validate weeks: [7, 8]
Training ends before validation: True

Imagine forecasting future weekly school-library visits. The eight row positions represent equally spaced weeks, ordered from oldest to newest. Randomly mixing future weeks into training would give the fit information that would not exist at the time of a real forecast.

TimeSeriesSplit creates expanding training windows: weeks 1–2 validate on 3–4; weeks 1–4 validate on 5–6; weeks 1–6 validate on 7–8. Early weeks are never validation records here, so this is not the ordinary k-fold rule that validates every row exactly once. No visit model is fitted in this split-only program.

test_size=2 means two validation rows per split. Equal spacing helps those windows represent comparable durations. If a feature or target needs time to become available, a suitable gap or a more specialized split may be necessary. Chronological splitting alone cannot fix a feature that already contains future information.

Keep the final future test period separate too. The TimeSeriesSplit reference explains expanding windows and its gap parameter.

Compare candidates on the same partitions and with the same metric. Then a score difference is less entangled with which students happened to validate each candidate. Keep required preparation inside each candidate's pipeline, and examine both scores and practical cost.

The folds produce several fitted models, but ordinary cross-validation does not automatically combine them into an ensemble or provide a final fit on every development row. After selecting a procedure, fit a new instance on the full development set. Its parameters can differ from each smaller fold fit.

The selected procedure's final test result need not equal its cross-validation mean: it is a different fit evaluated on different records. Report both with their roles clearly named. If that test result reveals a problem and you revise the method, be clear that the test has influenced selection; a fresh independent evaluation may be needed.

In hyperparameter tuning, we will compare settings systematically. The same protected-test boundary applies whether you try two choices manually or hundreds automatically.

Your task: Assume the scaling-plus-linear-regression procedure has been selected using development folds only. In the complete program below, how many students should fit its final scaler and regression? Should you average the three earlier fitted slopes instead?

Python · complete practice program

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Nine independent invented students: hours and quiz points out of 150.
X = np.arange(1, 10, dtype=float).reshape(-1, 1)
y = np.array([52., 57., 73., 79., 93., 97., 112., 118., 134.])
# Three different students reserved before comparing procedures.
X_test = np.array([[2.5], [5.5], [8.5]])
y_test = np.array([68., 95., 124.])
# Assume the procedure has been selected using development folds only.
final_model = make_pipeline(StandardScaler(), LinearRegression()).fit(X, y)
predictions = final_model.predict(X_test)
print("Fitted training rows:", len(X))
print("Test predictions:", np.round(predictions, 3).tolist())
print("Final test MAE:", round(float(mean_absolute_error(y_test, predictions)), 3))

Worked answer

Fit both stages on all nine development students. Do not average the earlier slopes: that would create a different procedure. The three reserved students J, K and L supply only this final evaluation. Their hours are 2.5, 5.5 and 8.5, and their invented recorded points are 68, 95 and 124.

Fitted training rows: 9
Test predictions: [65.264, 95.614, 125.964]
Final test MAE: 1.771

The program computes the mean absolute difference between these three targets and their predictions. Its small invented test is enough to verify the mechanics, not to support a claim about school-wide accuracy. For additional practice, change the earlier perfect-line program's fold count from three to nine: it fits eight records per round and validates one, still giving displayed zero error for that exactly constructed relationship.

More folds mean more fits, with additional cost when comparing many candidates. Five or ten folds are common starting choices for suitable ordinary datasets, not mandatory rules. With few examples, rare classes or few groups, a sensible count depends on what each training and validation portion can support.

Leave-one-out uses one validation record per round and fits once for every record. It maximizes the training portion in each round, but is not universally more reliable and can be expensive. Repeated k-fold methods rerun partitions to inspect sensitivity; repeated use still does not create independent copies of the dataset.

Trying many choices against the same validation folds can overfit the selection process. Nested cross-validation uses inner folds to select a procedure and separate outer folds to evaluate that entire selection process. Outer validation must remain outside the inner search. We introduce its purpose here; the next lesson develops the actual search workflow.

A pipeline prevents specified learned preparation from being fitted on a validation fold, but cannot fix duplicated people, future-derived features, unsuitable targets or an unrepresentative population. See the official data-leakage guidance and cross-validation guide for these boundaries. Revisit overfitting and underfitting when interpreting gaps between training and validation.

• Each fold is fresh: train on its training records and predict only its held-out validation records.
• Read scores in context: compare the same partitions, inspect individual errors and name the units.
• Refit preparation too: a pipeline keeps learned means, scales and representations inside training.
• Match the real question: class balance, related people and future periods need appropriate splitting rules.
• Protect final evaluation: select using development folds, refit the chosen procedure and then check the final test.

Before moving on: identify B's roles across the three rounds, explain why fold spread is not a guaranteed confidence interval, and decide how you would split repeated student records. Next, we will use these boundaries when choosing model settings.