K-nearest neighbors: classify a flower by nearby examples

You measure a flower’s petal length and width, but do not know its species. You already have measurements from flowers whose species are known. K-nearest neighbors, usually shortened to KNN, predicts by finding nearby training examples and combining their known answers. For classification, the simplest combination is a vote.

The number k says how many neighbors take part, not how many species exist. With k = 3, the three closest training flowers vote. If two are versicolor and one is virginica, uniform voting predicts versicolor, even if the single closest flower is virginica.

We will use the classic Iris dataset, with 150 samples across three species: setosa, versicolor and virginica. Petals are parts of a flower. A small selected subset will make the neighbor search inspectable, followed by a separate full-data evaluation. Review features and classification labels and training/test separation if needed.

The Iris dataset contains four features per flower. To make this first search understandable, we use only petal length and petal width, both in centimetres, and select three records from each species. The source row numbers identify positions in the packaged dataset; they are not prediction features.

Our query is a hypothetical flower with petal length 5.0 cm and width 1.8 cm. Its species is not supplied to the search. We chose this small subset to illustrate voting, not to represent the full population or measure model quality. Using only two measurements also discards information from the other features.

“Nearby” means nearby in these measurement values, not flowers growing near one another. This distinction is the limit of the everyday neighbor analogy: the model compares a numerical representation rather than understanding the flower as a person would.

With these two features we use Euclidean distance, the straight-line gap in a measurement plot. Subtract lengths and widths separately, square those differences, add them and take the square root. In one dimension it reduces to the absolute difference between the two values.

For query (5.0, 1.8) and source row 101 at (5.1, 1.9), the differences are −0.1 and −0.1 cm. Distance is the square root of 0.1² + 0.1², approximately 0.141. Row 52 at (4.9, 1.5) has distance √(0.1² + 0.3²), approximately 0.316. Row 50 at (4.7, 1.4) has distance √(0.3² + 0.4²) = 0.500.

Distances are calculated before labels are counted. Source row order is not neighbor order: a later reference record can be closest. No true query species is used to decide which points are nearest.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier

iris = load_iris()
# Selected teaching subset, not a representative evaluation split.
row_ids = np.array([0, 1, 2, 50, 51, 52, 100, 101, 102])
X = iris.data[row_ids, 2:4]  # Petal length and width in centimetres.
y = iris.target[row_ids]
query = [[5.0, 1.8]]
model = KNeighborsClassifier(n_neighbors=3).fit(X, y)
distances, indices = model.kneighbors(query)
print("Neighbor source rows:", row_ids[indices[0]].tolist())
print("Distances:", np.round(distances[0], 3).tolist())
print("Neighbor species:", iris.target_names[y[indices[0]]].tolist())
print("Prediction:", iris.target_names[model.predict(query)[0]])

Expected output

Neighbor source rows: [101, 52, 50]
Distances: [0.141, 0.316, 0.5]
Neighbor species: ['virginica', 'versicolor', 'versicolor']
Prediction: versicolor

load_iris() provides the measurements, numeric targets and matching species names. iris.data[row_ids, 2:4] selects the stated rows and the two petal columns. Python’s slice includes column 2 and stops before column 4. Selecting targets with the same row array preserves their pairing.

fit(X, y) retains reference observations for neighbor prediction; it does not learn a price-style slope or a logistic coefficient. The search implementation may build an index to speed access. kneighbors(query) returns distances and row positions within this selected training array. We map those positions through row_ids to recover the original source rows.

The three closest species give versicolor two votes and virginica one. The default weights are uniform: each selected neighbor contributes equally despite having a different distance. predict(query) returns a numeric species target, which target_names converts to its readable name. The classifier API describes these methods and defaults.

For this same query and reference subset, k = 1 selects only row 101 and predicts virginica. With k = 3, rows 101, 52 and 50 vote, producing versicolor. Neither choice is proved correct because our hypothetical query has no known true species.

A small k responds strongly to individual examples, including noisy or wrongly labelled records. A larger k spreads the decision across more references and can smooth local changes, but may overlook a small useful group. If you use all nine selected rows here, every query receives the same three-per-species vote tie: the locality has disappeared.

K is a setting you choose, rather than a learned number of classes. Compare candidate settings on validation data or through cross-validation, then evaluate the selected procedure on a protected test set. There is no universally correct value, and k must not exceed the available reference rows when predicting.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier

iris = load_iris()
row_ids = np.array([0, 1, 2, 50, 51, 52, 100, 101, 102])
X = iris.data[row_ids, 2:4]
y = iris.target[row_ids]
query = [[5.0, 1.8]]
for weighting in ["uniform", "distance"]:
    model = KNeighborsClassifier(n_neighbors=3, weights=weighting).fit(X, y)
    print(weighting, "->", iris.target_names[model.predict(query)[0]])

Expected output

uniform -> versicolor
distance -> virginica

This complete program reloads the same nine records and repeats the query. Uniform voting gives two votes to versicolor and one to virginica. Distance weighting instead gives each neighbor influence proportional to the inverse of its distance.

The closest virginica has distance about 0.141, giving weight about 7.071. The two versicolor neighbors have distances about 0.316 and 0.500, giving weights about 3.162 and 2.000. Their combined weight, about 5.162, is smaller, so distance-weighted voting predicts virginica.

This calculation uses the unrounded distances; the displayed approximations explain it. The numerical change is not evidence that weighting is more accurate. An unusually close mislabeled reference can receive strong influence. Exact zero-distance matches require special handling rather than literal division by zero; scikit-learn handles that case in its distance weighting implementation.

Imagine a query flower with petal length 5.0 cm and width 1.8 cm. Reference A differs only in length by 0.2 cm; reference B differs only in width by 0.1 cm. Their raw distances are 0.2 and 0.1, so B is nearer.

Now express widths in millimetres, consistently converting both references and the query. B’s width difference becomes 1 mm, while A’s length difference is still 0.2 cm. A raw calculation mixing those numerical columns now chooses A. The physical observations did not change; their representation changed the distance.

Feature scaling can give columns comparable numerical scales. Standardization uses a training-column mean and spread, so a transformed difference is expressed relative to that column’s training variation. This is a modelling choice, not proof that every feature deserves equal importance. See feature scaling and normalization for the transformations and the official KNN scaling example for their effect on a classifier.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=7, stratify=iris.target
)
# Teaching choice k=3, not selected by searching this test set.
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=3))
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Training and test rows:", len(y_train), len(y_test))
print("Test class counts:", np.bincount(y_test).tolist())
print("Correct predictions:", int(np.sum(predictions == y_test)))
print("Test accuracy:", round(accuracy_score(y_test, predictions), 3))

Expected output

Training and test rows: 120 30
Test class counts: [10, 10, 10]
Correct predictions: 30
Test accuracy: 1.0

This is a separate experiment using all 150 Iris records and all four measurements, not just the nine-record teaching subset. Stratification preserves the chosen class counts. The fixed seed makes this particular 120/30 split repeatable; it does not make every possible split equally easy.

make_pipeline places StandardScaler before the classifier. Fitting on X_train learns scaling statistics from the training portion, transforms those inputs and retains the transformed reference flowers. Prediction transforms test flowers using those same statistics before finding their neighbors. The test targets participate only in the accuracy comparison.

For this specified split and procedure, all 30 test species are predicted correctly. Accuracy is correct predictions divided by the number evaluated, here 30/30. That does not prove the classifier identifies every future flower or every other split correctly. The dataset covers three species, not all flowers, and our teaching choice k = 3 was not tuned by inspecting this test score.

To choose a scaler, k or weighting, compare candidates within the development data; fit preparation separately inside each validation fold. The model-setting selection lesson covers this workflow.

A uniform KNN classifier can report class probabilities as the fractions of selected neighbors carrying each label. For our three-neighbor query, the reported versicolor fraction is 2/3 and virginica fraction 1/3; none of the three is setosa. The probability columns follow the fitted model’s class order.

These fractions describe the chosen neighborhood. They are not automatically calibrated chances for a real unknown flower. Changing k, scaling or weighting changes the neighborhood or contributions, so it can change both the predicted class and its reported fractions.

With k = 1, the selected species gets probability 1 under uniform voting. That means the single neighbor has that label; it does not establish perfect certainty. A flower far from every training reference can still receive a confident-looking KNN answer. Inspect distances and evaluate reliability rather than treating a unanimous local vote as a guarantee.

Your task: In this complete program, change n_neighbors from 3 to 1. Predict which source row remains, its species and the final answer before running it. Explain why the result differs from the original three-neighbor vote.

Python · complete program to modify

import numpy as np
from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier

iris = load_iris()
# Selected teaching subset, not a representative evaluation split.
row_ids = np.array([0, 1, 2, 50, 51, 52, 100, 101, 102])
X = iris.data[row_ids, 2:4]  # Petal length and width in centimetres.
y = iris.target[row_ids]
query = [[5.0, 1.8]]
model = KNeighborsClassifier(n_neighbors=3).fit(X, y)
distances, indices = model.kneighbors(query)
print("Neighbor source rows:", row_ids[indices[0]].tolist())
print("Distances:", np.round(distances[0], 3).tolist())
print("Neighbor species:", iris.target_names[y[indices[0]]].tolist())
print("Prediction:", iris.target_names[model.predict(query)[0]])

Original output with three neighbors

Neighbor source rows: [101, 52, 50]
Distances: [0.141, 0.316, 0.5]
Neighbor species: ['virginica', 'versicolor', 'versicolor']
Prediction: versicolor

Worked answer

Use KNeighborsClassifier(n_neighbors=1). The query remains (5.0, 1.8) and all reference measurements and labels stay unchanged.

Neighbor source rows: [101]
Distances: [0.141]
Neighbor species: ['virginica']
Prediction: virginica

The nearest flower alone now supplies the answer. The two slightly farther versicolor references no longer participate, so their previous majority cannot influence this decision. Without the query’s actual species, this shows sensitivity to k rather than proving which policy is better.

Different tie situations need different care. A tie in class vote totals requires the classifier’s tie rule; an odd k avoids a two-class uniform vote tie but does not prevent ties among three or more classes. At the edge of a neighborhood, equally distant candidates with different labels can make the result depend on reference ordering. Our worked query avoids those distance ties.

A numerical student ID or source row number is an identifier, not a useful physical measurement. One-hot categories and other representations need a distance interpretation too. Irrelevant features can overwhelm useful ones, while many-dimensional data can make nearest and farther distances less distinguishable. More columns do not necessarily provide better neighbors.

KNN also supports regression: a uniform neighbor regressor averages nearby numerical targets, such as sale prices, instead of voting on categories. Choosing a classifier or regressor follows the target’s meaning. Arbitrarily encoded species numbers should not be averaged into a “middle species”.

The method retains reference data and must search it during prediction. Search indexes can help, but speed and storage depend on the dataset, feature dimension and search method. Large datasets can make predictions costly. The neighbor-methods guide covers distance rules, ties, regression and search approaches. Keep reference data appropriate for the intended use and review prediction errors on representative held-out observations.

KNN predicts from reference observations instead of fitting a slope or a logistic score. Our selected Iris query had a nearest virginica reference, but its three uniform neighbors gave versicolor the majority.

• Find neighbors first: compute distances from the same feature representation, then inspect the selected rows and targets.
• Choose the policy: k controls neighborhood size; weighting controls each selected record’s influence.
• Respect units: scales, identifiers and irrelevant columns can change what counts as close.
• Evaluate the whole procedure: learn preprocessing on training data, select settings with validation and check held-out observations.
• Read the limits: a unanimous vote, a high test score or a nearby point does not guarantee the next prediction.

Before moving on: explain the three distances, why k = 1 changes this flower’s answer and why converting feature units can change a raw search. Next, we will use decision trees and random forests to classify examples through learned rules.