Hyperparameter tuning

A K-nearest neighbors (KNN) classifier predicts a flower's species by consulting nearby labelled flowers. Should it use three neighbors, five or seven? Choosing whichever setting looks nicest on the final test would use the answers we wanted to keep separate.

A hyperparameter is a setting supplied to a learning procedure, such as KNN's neighbor count. Hyperparameter tuning compares candidate settings using validation evidence. It is a controlled comparison, not a promise to find the best possible model for every future flower.

We will use Iris, the small bundled flower dataset already used in the KNN lesson. Reserve 30 flowers for final testing, and compare three neighbor counts using three validation folds within the other 120. The numerical illustration below comes from the runnable search, not invented benchmark claims.

Be comfortable with cross-validation and fresh fold fits, feature scaling and classification metrics. We will explain the search tools step by step.

In linear regression, fitting learns the slope and intercept from training examples. Those fitted quantities are model parameters. A decision tree learns feature thresholds and branches; its maximum permitted depth is a hyperparameter controlling that learning process.

KNN has a different mechanism: ordinary fitting retains the reference observations rather than learning a regression slope. The neighbor count n_neighbors, distance rule and voting weights configure its predictions. Calling fit does not automatically compare neighbor counts unless a separate search procedure is wrapped around it.

A search can select a hyperparameter from data, but it does so outside the basic estimator's fit. In that search, each candidate gets its own fits, and each fit learns only from its training portion. Dataset targets and held-out outcomes are observations, not hyperparameters.

You can also compare preparation choices or model families, but a wider search is a wider selection process with more computational and statistical cost. Start with meaningful settings for the chosen method rather than changing every available constructor option.

Iris contains 150 records, with four measurements per flower and one species label from three classes. Its features measure sepal and petal length and width in centimetres: sepals are the outer flower parts and petals are the colored inner parts. This teaching dataset does not represent every flower population or application.

The example uses a stratified 80/20 development/test split, preserving class counts: 120 development flowers and 30 test flowers. Three stratified development folds each fit on 80 flowers and validate on 40. All candidates use the same folds.

We preselect neighbor counts 3, 5 and 7, uniform voting and an accuracy scorer. Each of the three candidates requires three validation fits, for nine fits, then the selected setting is refitted once on all 120 development rows. That is ten pipeline fits in total, not three fits and not a new fit for every test prediction.

Accuracy is suitable for explaining this balanced teaching example. For a rare event or unequal consequences of mistakes, choose a metric that matches the task before searching. Fixing seeds makes the demonstration reproducible; it does not prove the chosen partition is representative.

Prepare and run

Use Python with scikit-learn in the course environment. Iris is bundled with the library; no separate file is needed.

Python · complete runnable example

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
train_X, test_X, train_y, test_y = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=5)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=5)
model = make_pipeline(StandardScaler(), KNeighborsClassifier())
search = GridSearchCV(model, {"kneighborsclassifier__n_neighbors": [3, 5, 7]},
                      cv=cv, scoring="accuracy", refit=True, n_jobs=1,
                      error_score="raise")
search.fit(train_X, train_y)
print("Development rows:", len(train_X), "protected test rows:", len(test_X))
for row, params in enumerate(search.cv_results_["params"]):
    scores = [float(search.cv_results_["split" + str(fold) + "_test_score"][row])
              for fold in range(3)]
    print("k:", params["kneighborsclassifier__n_neighbors"],
          "fold accuracy:", [round(score, 3) for score in scores],
          "mean:", round(float(search.cv_results_["mean_test_score"][row]), 3))
print("Chosen k:", search.best_params_["kneighborsclassifier__n_neighbors"])
print("Final refit rows:", search.best_estimator_.named_steps["kneighborsclassifier"].n_samples_fit_)

Expected output

Development rows: 120 protected test rows: 30
k: 3 fold accuracy: [0.875, 0.925, 0.925] mean: 0.908
k: 5 fold accuracy: [0.925, 0.925, 0.9] mean: 0.917
k: 7 fold accuracy: [0.95, 0.925, 0.975] mean: 0.95
Chosen k: 7
Final refit rows: 120

train_test_split creates the protected test portion before searching. stratify=y preserves the species proportions. make_pipeline puts scaling and KNN in a single procedure so every fold learns its scaler from its own training rows.

kneighborsclassifier__n_neighbors means the n_neighbors setting inside the pipeline step named kneighborsclassifier. The two underscores separate the step name and its setting. scoring="accuracy" states the selection rule explicitly; larger validation accuracy is better.

GridSearchCV.fit(train_X, train_y) evaluates all supplied candidates. cv_results_ stores their fold scores and means; its test_score keys name fold validation scores, not our 30 protected test records. best_params_ reports the selected setting.

With refit=True, the search then fits the selected complete pipeline on all 120 development records. best_estimator_ is that final fitted pipeline. Its stored reference-row count confirms the refit; it is not the average of three fold models. This program does not evaluate the protected test.

For the seven-neighbor candidate, round 1 fits a new scaler and KNN on its 80 training flowers and correctly classifies 38 of 40 validation flowers: accuracy 0.95. Round 2 starts again with a different training portion and correctly classifies 37 of 40: 0.925. Round 3 correctly classifies 39 of 40: 0.975.

The mean is (0.95 + 0.925 + 0.975) ÷ 3 = 0.95. The three-neighbor and five-neighbor candidates repeat those same partition roles with their own fits. Their means are approximately 0.9083 and 0.9167, so seven has the largest mean in the supplied grid.

The selected pipeline is then refitted on all development flowers. Validation flowers can now participate in this final fit because selection has finished and the protected test is still outside it. No fold's fitted scaler is simply reused as the final scaler.

KNN tie behavior and small score differences depend on data and implementation details. These checked results describe the specified dataset, versions, split, scaling and candidate set; they are not universal recommendations to use seven neighbors.

Repeat the candidate means for this comparison: k=3 gives 0.9083, k=5 gives 0.9167 and k=7 gives 0.95. best_score_ is the selected candidate's mean cross-validation score under the stated scorer. It is not final-test accuracy and not a guarantee for future data.

Inspect individual folds, score spread, training scores where useful and timing information in cv_results_. A small average difference may not justify extra prediction cost or reduced interpretability. Fold standard deviation is a descriptive spread; overlapping training portions make it unsuitable as an automatic confidence interval.

Trying many settings can favor one that was lucky on these validation records. The winning validation score can be optimistic as an estimate of the whole selection process. An independent final test or properly nested evaluation helps assess the selected procedure.

Exact ties can make the selected candidate depend on enumeration and refit rules. Do not treat a tie as evidence that one choice is scientifically superior. The GridSearchCV reference documents result attributes and refitting behavior.

Prepare and run

Use Python with scikit-learn in the course environment. Iris is bundled with the library; no separate file is needed.

Python · complete runnable example

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
train_X, test_X, train_y, test_y = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=5)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=5)
model = make_pipeline(StandardScaler(), KNeighborsClassifier())
grid = {"kneighborsclassifier__n_neighbors": [3, 5, 7],
        "kneighborsclassifier__weights": ["uniform", "distance"]}
search = GridSearchCV(model, grid, cv=cv, scoring="accuracy", n_jobs=1,
                      error_score="raise")
search.fit(train_X, train_y)
print("Candidates:", len(search.cv_results_["params"]))
print("CV fits:", len(search.cv_results_["params"]) * cv.get_n_splits())
print("Chosen k:", search.best_params_["kneighborsclassifier__n_neighbors"])
print("Chosen weights:", search.best_params_["kneighborsclassifier__weights"])
print("Selected CV accuracy:", round(float(search.best_score_), 3))

Expected output

Candidates: 6
CV fits: 18
Chosen k: 7
Chosen weights: uniform
Selected CV accuracy: 0.95

This is a separately preplanned demonstration, not a reaction to a final-test score. uniform gives all chosen neighbors equal votes. distance gives closer neighbors more weight. Both are settings of the same classifier; the learned scaling remains inside each fit.

There are three neighbor counts and two voting rules, so this grid contains 3 × 2 = 6 combinations. Three folds require 18 validation fits, plus one final refit. The printed count excludes that additional refit.

A grid is exhaustive only within the values you supplied. It does not try every integer neighbor count, every distance rule or every classifier. Conditional settings can be represented with separate grid dictionaries when some combinations do not make sense; blindly multiplying every choice can waste fits.

The selected pair can differ from the first search because the candidate set changed. Compare searches according to their planned purpose and evaluation boundary, not by choosing whichever protected-test score looks higher.

Prepare and run

Use Python with scikit-learn in the course environment. Iris is bundled with the library; no separate file is needed.

Python · complete runnable example

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
train_X, test_X, train_y, test_y = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=5)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=5)
model = make_pipeline(StandardScaler(), KNeighborsClassifier())
from sklearn.model_selection import RandomizedSearchCV

space = {"kneighborsclassifier__n_neighbors": [1, 3, 5, 7, 9],
         "kneighborsclassifier__weights": ["uniform", "distance"]}
search = RandomizedSearchCV(model, space, n_iter=4, random_state=5, cv=cv,
                            scoring="accuracy", n_jobs=1, error_score="raise")
search.fit(train_X, train_y)
print("Sampled candidates:", len(search.cv_results_["params"]))
for params in search.cv_results_["params"]:
    print("k:", params["kneighborsclassifier__n_neighbors"],
          "weights:", params["kneighborsclassifier__weights"])
print("CV fits:", len(search.cv_results_["params"]) * cv.get_n_splits())
print("Chosen k:", search.best_params_["kneighborsclassifier__n_neighbors"])
print("Chosen weights:", search.best_params_["kneighborsclassifier__weights"])
print("Selected CV accuracy:", round(float(search.best_score_), 3))

Expected output

Sampled candidates: 4
k: 9 weights: distance
k: 5 weights: distance
k: 3 weights: uniform
k: 5 weights: uniform
CV fits: 12
Chosen k: 9
Chosen weights: distance
Selected CV accuracy: 0.95

This independent example plans five neighbor counts and two voting rules: ten possible combinations. n_iter=4 asks RandomizedSearchCV to evaluate four sampled candidates, giving 4 × 3 = 12 validation fits plus one refit. The printed settings make the sampled scope visible.

When every supplied choice is a finite list, candidates are sampled without replacement. If at least one supplied value is a probability distribution, sampling can repeat combinations and continuous values need an appropriate range and distribution. A fixed random seed reproduces this sample; it cannot ensure that the strongest untried choice was sampled.

Random search makes a candidate budget explicit. For settings spanning orders of magnitude, thoughtfully chosen logarithmic ranges can be useful; an arbitrary broad range can waste the budget. More elaborate search methods may allocate resources adaptively, but they still need fair validation and a protected evaluation boundary.

See the RandomizedSearchCV reference for finite-list and distribution sampling behavior. None of this program's search decisions uses the protected test.

Accuracy counts correct predictions. A medical screening or rare-alert task may instead prioritize recall, precision, a cost rule or a justified combination. Two settings can rank differently under different scorers, so “best” must always name the rule being optimized.

For regression, scikit-learn uses scorer names such as neg_mean_absolute_error so larger is always preferred. A score of −2 beats −5 because its underlying MAE is 2 rather than 5. Convert negative error scores back to ordinary positive units for reporting; do not sort them in the wrong direction.

Multi-metric searches can record several measures while naming one for refitting. A callable refit rule can incorporate other criteria, but the selection policy still needs a clear explanation. Do not quietly switch metrics after seeing which one makes a preferred model win.

Keep a sensible untuned or simple baseline for comparison. A search can consume many fits and still fail to improve the result that matters. The metrics lesson explains what the common scores count.

The scaler, imputer, PCA or feature selector must be fitted within each candidate's training fold. Fitting preparation on all 120 development rows before the search exposes each validation fold to learned statistics, even though the final classifier is fitted separately.

A pipeline automates that preparation boundary, but cannot repair repeated people split across sides, features calculated from future outcomes or unsuitable target definitions. Use group-aware or time-aware splits when the deployment question requires them. The cross-validation lesson shows those different partition rules.

Keep the final test outside both candidate fitting and selection. If seeing its score causes you to add k=9, change the scaler or try another classifier, the test has influenced development. Treat it accordingly; a fresh independent evaluation may be needed for a new final claim.

Checking a protected test once does not make the result automatically representative. Its population, timing, data quality and task still determine what it can support.

Prepare and run

Use Python with scikit-learn in the course environment. Iris is bundled with the library; no separate file is needed.

Python · complete runnable example

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
train_X, test_X, train_y, test_y = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=5)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=5)
model = make_pipeline(StandardScaler(), KNeighborsClassifier())
from sklearn.model_selection import cross_validate

inner_search = GridSearchCV(model, {"kneighborsclassifier__n_neighbors": [3, 5, 7]},
                            cv=cv, scoring="accuracy", n_jobs=1, error_score="raise")
outer = StratifiedKFold(n_splits=3, shuffle=True, random_state=11)
result = cross_validate(inner_search, train_X, train_y, cv=outer,
                        scoring="accuracy", return_estimator=True, error_score="raise")
print("Outer accuracy:", [round(float(v), 3) for v in result["test_score"]])
print("Mean outer accuracy:", round(float(result["test_score"].mean()), 3))
print("Inner chosen k:", [s.best_params_["kneighborsclassifier__n_neighbors"]
                          for s in result["estimator"]])
print("Each outer-refit training size:",
      [s.best_estimator_.named_steps["kneighborsclassifier"].n_samples_fit_
       for s in result["estimator"]])

Expected output

Outer accuracy: [0.95, 0.9, 0.975]
Mean outer accuracy: 0.942
Inner chosen k: [7, 3, 5]
Each outer-refit training size: [80, 80, 80]

This program repeats the complete input setup and the original three-count search. An inner three-fold search chooses k using only an outer round's training records. The separate outer validation portion scores the selected, refitted pipeline. It never supplies an inner selection score.

Each outer round receives 80 of the 120 development flowers for its search and holds out 40. The inner search fits three candidates across three inner folds, then refits its selected pipeline on those 80 outer-training flowers. The stored final sizes confirm that no outer-validation flowers enter those refits.

The chosen k can differ across outer rounds because the selection data differs. The mean outer accuracy estimates this defined selection procedure under these splits; it is not the mean of the inner winning scores. It is still limited by the dataset and split assumptions, not a guarantee of future performance.

Three outer rounds each perform nine candidate fits plus one refit: 30 pipeline fits. The 30 protected final-test flowers remain outside this nested demonstration. To obtain a deployable fit, run the chosen search procedure again on the full development set; the nested fold models are not automatically combined. The official nested-validation example explains the separation of selection and evaluation.

Prepare and run

Use Python with scikit-learn in the course environment. Iris is bundled with the library; no separate file is needed.

Python · complete runnable example

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
train_X, test_X, train_y, test_y = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=5)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=5)
model = make_pipeline(StandardScaler(), KNeighborsClassifier())
search = GridSearchCV(model, {"kneighborsclassifier__n_neighbors": [3, 5, 7]},
                      cv=cv, scoring="accuracy", refit=True, n_jobs=1,
                      error_score="raise")
search.fit(train_X, train_y)
print("Candidates:", len(search.cv_results_["params"]))
print("Chosen k:", search.best_params_["kneighborsclassifier__n_neighbors"])
print("Held-out accuracy:", round(float(search.score(test_X, test_y)), 3))
print("Correct test flowers:", int((search.predict(test_X) == test_y).sum()), "of", len(test_y))

Expected output

Candidates: 3
Chosen k: 7
Held-out accuracy: 0.967
Correct test flowers: 29 of 30

This complete program reproduces the original three-candidate experiment and adds its protected test evaluation. Conceptually, you planned the candidates and scoring rule first, compared them using development folds, and only then requested this final result. The other search examples above are independent demonstrations, not successive revisions based on this test.

With the default refit enabled, search.score(test_X, test_y) uses the selected pipeline fitted on all 120 development flowers. It applies that pipeline's learned scaler and KNN to the 30 held-out flowers; it does not run another search on them.

Twenty-nine of thirty classifications are correct, so accuracy is 29 ÷ 30 ≈ 0.967. That differs from the selected mean validation accuracy of 0.95 because the fit and evaluated records differ. One test error is a large proportion of such a small test, so keep the count visible alongside the percentage.

The result retains the original lesson's checked demonstration. It is not a production performance claim or an instruction to modify the search after reading the test score.

Your task: Before opening the final test result, you plan k values 3, 5, 7 and 9 with the same three development folds and uniform voting. How many validation fits are needed? How many rows enter the selected pipeline's refit? Does adding a candidate necessarily improve independent performance?

Python · complete practice program

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
train_X, test_X, train_y, test_y = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=5)
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=5)
model = make_pipeline(StandardScaler(), KNeighborsClassifier())
# Plan this expanded candidate set before inspecting the protected test.
search = GridSearchCV(model, {"kneighborsclassifier__n_neighbors": [3, 5, 7, 9]},
                      cv=cv, scoring="accuracy", n_jobs=1, error_score="raise")
search.fit(train_X, train_y)
print("Candidates:", len(search.cv_results_["params"]))
print("CV fits:", len(search.cv_results_["params"]) * cv.get_n_splits())
print("Refit training rows:", search.best_estimator_.named_steps["kneighborsclassifier"].n_samples_fit_)
print("Chosen k:", search.best_params_["kneighborsclassifier__n_neighbors"])

Worked answer

Four candidates across three folds require 12 validation fits. Refitting adds one more fit on all 120 development flowers. Adding k=9 expands the searched set; it does not guarantee better final-test or future performance.

Candidates: 4
CV fits: 12
Refit training rows: 120
Chosen k: 7

This program deliberately prints no final-test score. If you change the candidate list only after inspecting a protected test, it is no longer the originally planned final evaluation. Even when selection legitimately improves its development score, that improvement may not transfer to new records.

A neighbor count must fit within each fold's available reference rows. Invalid settings or incompatible combinations can cause fitting or scoring failures. The examples use error_score="raise" so a failed candidate stops visibly rather than quietly receiving a missing score. In ordinary searches, inspect warnings and failed-score entries before interpreting rankings.

Choose reasonable ranges from the estimator's behavior, keep required class/group/time boundaries and budget for both candidate fits and final refitting. n_jobs=1 makes these small programs run sequentially; parallel workers can reduce elapsed time but increase memory pressure. Timing results depend on hardware and workload.

Neighbor count affects local averaging, but larger is not always better. Voting weights and feature scales interact with it. Search scope, score uncertainty, prediction latency and resource requirements all matter when choosing a practical procedure.

Track the dataset, split seeds, preparation, candidate space, metric, library versions and selection rule. A reproducible experiment is easier to inspect; it still needs representative evaluation before deployment. The official tuning guide covers supported search strategies and their assumptions.

• Settings differ from learned parameters: the search compares configurations, and each candidate fit learns from its own training rows.
• Keep comparisons fair: choose a metric, use the same appropriate folds and fit preprocessing inside each pipeline.
• Count the work: grid combinations or sampled candidates multiply by folds, with final refitting added.
• Bound the winner: best means best in the supplied experiment; it is not a universal optimum.
• Separate selection from evaluation: use outer folds when evaluating a search procedure and keep the final test outside tuning.

Before moving on: calculate the six-combination fit budget, explain why the 0.95 validation mean differs from 29/30 final-test accuracy, and identify what would leak if you fitted a scaler before making folds. Next, we will diagnose models that fit too little or fit the training data too closely.