Training and test splits: evaluate on unseen data

A model can fit the examples it has already seen and still struggle with unfamiliar ones. If we evaluate only on its training records, a good score may partly reflect remembering those records rather than a useful relationship for new inputs.

A training/test split partitions the available records into separate roles. Training records supply the evidence used to fit the model and its learned preparation. Test records are reserved for evaluating the selected procedure on held-out examples. Separation makes the check more informative, but does not by itself guarantee that future data will behave the same way.

We will use the classic Iris teaching dataset: 150 flower samples, four measurements per sample and three species labels, with 50 samples per species. The measurements describe lengths and widths of petals and sepals, parts of a flower. Our question is how to partition these paired measurements and labels; we are not training a species classifier in this lesson.

The roles depend on what you do with the records, not what you name their files. A training set is used for fitting. A validation set helps choose a method or settings. A test set is kept for a final check after those decisions.

A student practicing the exact exam questions is a useful analogy for evaluating on familiar records. Its limit is that a model’s generalization depends on statistical relationships and the evaluation design, not human understanding. Simply hiding a few rows does not turn a result into evidence for every future population.

If you repeatedly inspect a test score and change the model to improve it, that set is helping select the model. It has effectively become validation data. Keep its role clear and report the evaluation procedure honestly.

Prepare and run

Use Python in the course environment with NumPy and scikit-learn. All data is included or comes from a packaged dataset; no download is required.

Python · complete runnable example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X, y = iris.data, iris.target
row_ids = np.arange(len(y))
X_train, X_test, y_train, y_test, id_train, id_test = train_test_split(
    X, y, row_ids, test_size=0.2, random_state=7, stratify=y
)
print("Training shape:", X_train.shape)
print("Test shape:", X_test.shape)
print("Training class counts:", np.bincount(y_train).tolist())
print("Test class counts:", np.bincount(y_test).tolist())
print("Shared row IDs:", np.intersect1d(id_train, id_test).tolist())
print("Training rows still paired:", np.array_equal(y_train, y[id_train])
      and np.array_equal(X_train, X[id_train]))

Expected output

Training shape: (120, 4)
Test shape: (30, 4)
Training class counts: [40, 40, 40]
Test class counts: [10, 10, 10]
Shared row IDs: []
Training rows still paired: True

Follow the row selection

X contains the four measurements for each sample; y contains the matching species code, 0, 1 or 2. These are class identities, not numerical species distances. Row_ids labels the original positions so we can inspect the partition; it is not added as a model feature.

Passing all three arrays to one train_test_split call applies the same selection to each. The returned arrays come in train/test pairs for each input: measurements first, then targets, then row IDs. A test_size of 0.2 reserves 30 of the 150 samples, leaving 120 for training.

Stratify=y requests a split that preserves the label proportions. Each species has 50 samples, so this example allocates 40 to training and 10 to test. Bincount counts the class codes; intersect1d finds any original row position shared between sets.

The last check uses the retained IDs to confirm that each training measurement row still has its original target. This is stronger than checking only that X_train and y_train have equal lengths. No model score is produced: we have verified the partition, not demonstrated prediction quality.

Test_size can be a fraction or a number of records. An 80/20 split is a common starting choice, not a universal optimum. You need enough training evidence to fit and enough independent evaluation evidence to measure the behavior that matters. A tiny test set gives a noisy estimate even when the percentage looks reasonable.

Random_state=7 fixes this shuffle for the given inputs and procedure. The number seven has no special statistical meaning. A fixed seed helps reproduce the split; it does not make that split representative, prove the model is reliable or justify trying seeds until a favorable score appears. Keep data ordering, versions and split settings in the experiment record.

Stratification helps maintain classification label proportions when a random row split is appropriate. It does not create examples of a rare class, balance the original dataset or prevent related records from appearing on both sides. Each class and the selected partition sizes need enough records for the operation to succeed.

The train_test_split reference explains these settings. If your goal is future prediction or prediction for new people, choose the relevant structure before deciding whether random stratification fits it.

Prepare and run

Use Python in the course environment with NumPy and scikit-learn. All data is included or comes from a packaged dataset; no download is required.

Python · complete runnable example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=7, stratify=iris.target
)
scaler = StandardScaler().fit(X_train)
prepared_train = scaler.transform(X_train)
prepared_test = scaler.transform(X_test)
print("Rows used to fit scaler:", int(scaler.n_samples_seen_))
print("Prepared test rows:", len(prepared_test))
print("Training means close to zero:", np.allclose(prepared_train.mean(axis=0), 0))
print("Test means close to zero:", np.allclose(prepared_test.mean(axis=0), 0))

Expected output

Rows used to fit scaler: 120
Prepared test rows: 30
Training means close to zero: True
Test means close to zero: False

This complete program reloads Iris and repeats the split, so it does not rely on a variable from an earlier section. StandardScaler fits only X_train, learning four training-column means and scales. Its sample count confirms that the 30 reserved test records were not used in fitting.

Transform applies the same learned settings to both arrays. The training-column means are approximately zero after standardization. The transformed test columns are not forced to have mean zero because they retain their position relative to the training reference. Allclose checks numerical proximity rather than exact floating-point equality.

Fitting a second scaler on X_test would adapt preparation to the reserved inputs. Fitting once on all 150 records before splitting would also let test data influence the learned scale. The same rule applies to imputation and learned category vocabularies.

A pipeline keeps preparation with the prediction model, including within each cross-validation training fold. Review feature scaling for the transformation itself and the scikit-learn leakage guide for the evaluation issue. This example checks preparation; it does not train a classifier.

Suppose you are comparing two scaling methods or several model settings. Use a validation portion from the training/development data, or cross-validation within that portion, to choose among the candidates. Keep the final test records outside that selection.

For the 150-row Iris dataset, an illustrative three-way allocation could be 90 training, 30 validation and 30 test records. One way is to first reserve 20% as test, then reserve 25% of the remaining 120 as validation. That second percentage is relative to the remaining data, not the original 150. Each stage must preserve the intended alignment and split structure.

After choosing the procedure, you can refit it on the combined development records—training plus validation—before applying it to the final test. This refit learns preparation statistics again from the permitted development portion. The test must remain outside fitting and selection.

Cross-validation reuses different development partitions for selection, helping when data is limited. We cover it in cross-validation without leakage and apply it in model-setting selection. These roles are a workflow choice, not a requirement to always create exactly three files.

A row ID overlap check only shows that the same numbered row was not selected twice. If a student has multiple assessment records, distinct rows may still belong to the same person. Near-duplicate records or repeated measurements can share information across a random row split.

If the application predicts results for new students, keep all observations from each student on one side. If it predicts for new schools, the school may be the group to hold out. The group must match the independent entity in the deployment question.

A group-aware split uses a group identifier for separation, without making that identifier a prediction feature. It does not automatically preserve class proportions, and uneven group sizes can produce a row fraction different from the selected group fraction. A random split for known people can answer a different question from a split for new people; define which one you actually need.

Prepare and run

Use Python in the course environment with NumPy and scikit-learn. All data is included or comes from a packaged dataset; no download is required.

Python · complete runnable example

import numpy as np
from sklearn.model_selection import GroupShuffleSplit

# Invented repeated observations: two rows per student.
students = np.array(["A", "A", "B", "B", "C", "C", "D", "D"])
X = np.arange(8).reshape(-1, 1)
splitter = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=7)
train_rows, test_rows = next(splitter.split(X, groups=students))
train_students = set(students[train_rows].tolist())
test_students = set(students[test_rows].tolist())
print("Training students:", sorted(train_students))
print("Test students:", sorted(test_students))
print("Training and test rows:", len(train_rows), len(test_rows))
print("Shared students:", sorted(train_students & test_students))

Expected output

Training students: ['A', 'B', 'D']
Test students: ['C']
Training and test rows: 6 2
Shared students: []

The data is invented: each of four students has two rows. X contains simple row values solely to give the splitter eight observations; there is no fitted student-performance model. Students supplies the group labels used to keep related observations together.

GroupShuffleSplit selects groups rather than independently selecting rows. Test_size=0.25 reserves one of the four student groups. That student’s two rows both go to test; the other three students supply six training rows. Next retrieves the single requested split, and the row-position arrays index the original observations.

The printed student sets contain no shared identity. This matters for evaluating new students. With unequal numbers of observations per person, a quarter of groups would not necessarily be a quarter of rows. The GroupShuffleSplit API specifies that its proportions refer to groups.

Do not assume this small example proves that all other relationships were separated. Shared schools, duplicated records or chronological information can still require a more appropriate design.

For forecasting, a random shuffle can put later observations into training while earlier observations are evaluated. That can give the learning procedure information it would not have when making a real forecast. Train on earlier data and evaluate on a later period when that matches the use case.

The timestamp of a row is not the only availability check. Its target may become known several days later, and a feature may be computed from future records. Define a cutoff at which all training inputs and labels would have been available. Depending on the prediction horizon and overlapping windows, leave a gap around the boundary.

Do not stratify or shuffle just to make a future period resemble the past. A change in class proportions or demand can be exactly what the application must handle. TimeSeriesSplit provides expanding-window evaluation, with configuration for a gap; its assumptions and data spacing need attention. See the TimeSeriesSplit reference.

Prepare and run

Use Python in the course environment with NumPy and scikit-learn. All data is included or comes from a packaged dataset; no download is required.

Python · complete runnable example

# Invented daily observations, already sorted by date.
records = [
    ("2026-01-01", 10), ("2026-01-02", 12),
    ("2026-01-03", 11), ("2026-01-04", 15),
    ("2026-01-05", 14), ("2026-01-06", 16),
]
train = records[:4]
test = records[4:]
print("Training dates:", [date for date, value in train])
print("Test dates:", [date for date, value in test])
print("Training ends before test starts:", train[-1][0] < test[0][0])

Expected output

Training dates: ['2026-01-01', '2026-01-02', '2026-01-03', '2026-01-04']
Test dates: ['2026-01-05', '2026-01-06']
Training ends before test starts: True

Each invented record contains a date and an observed daily amount. The records are already sorted. The first slice takes four earlier records; the second takes two later records. Because the dates use a consistent year-month-day format, the final string comparison reflects their chronological order.

This code inspects a partition; it does not construct forecasting features, fit a model or prove that future information is absent. In a real task, sort and validate timestamps, choose a meaningful horizon and inspect how every feature and target was produced. A correctly ordered row list can still contain a future-derived feature.

Scenario A: your table has three records per student, and the application will predict for students who have never appeared in training. Would an empty overlap of row numbers be enough?

Scenario B: the application predicts next month’s demand. A colleague shuffles all months and stratifies them by whether demand is high or low. Does preserving those proportions make the evaluation appropriate?

Worked answer

For A, check student identity overlap and keep each student’s records together. Different row numbers can still describe the same person. For B, preserve the past-to-future boundary and check when features and labels become available. Stratification does not remove information arriving from future periods.

Small coding variation: a 30% stratified Iris test split should reserve 45 records, including 15 per species, leaving 105 training records. The complete program below lets you check that prediction.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=7, stratify=iris.target
)
print("Training and test rows:", len(X_train), len(X_test))
print("Test class counts:", np.bincount(y_test).tolist())

Expected output

Training and test rows: 105 45
Test class counts: [15, 15, 15]

The changed fraction changes the allocation, not the meaning of the roles. It is still necessary to check whether random stratification answers the intended application question.

Disjoint rows are necessary for this simple holdout, but not sufficient evidence of independence. Check related people, duplicated source records, shared sites and features recorded after the outcome. A post-outcome field can leak the answer even if the row IDs never overlap.

A single test score is an estimate for a particular dataset and procedure. Small evaluation samples can produce substantial uncertainty; a class with very few examples can be especially poorly assessed. Compare with an appropriate baseline and inspect relevant errors, not only the overall score. We cover those choices in model evaluation metrics.

Do not cycle through random seeds to report the most flattering partition. Save the split definition, source version and relevant group/time assumptions so the experiment can be reproduced. If the final test reveals a problem and you use it to redesign the model, acknowledge its role in that development and obtain fresh evaluation evidence where appropriate.

Even a carefully held-out Iris result does not guarantee performance for every future flower sample or collection process. The evaluation needs to resemble the intended use case, and changes in the data may require further checks.

The Iris split kept each flower’s measurements and species target together, reserved 30 of 150 rows and verified the selected class counts and original row pairing. We then applied training-fitted preparation to the held-out portion.

• Define roles: training fits, validation selects and test evaluates the selected procedure.
• Keep alignment: split related arrays together and inspect counts, identity overlap and target pairing.
• Match the use case: random stratification, group separation and time separation answer different questions.
• Protect preparation: learn imputers, encoders and scales from the allowed training portion, including each cross-validation fold.
• Interpret evidence: a fixed seed and a clean split help reproducibility; neither guarantees independent, representative future performance.

Before the next lesson: explain which records may teach a model its parameters and which records should remain reserved. Next, we will build a linear regression model and inspect how it learns a numerical relationship.