Feature scaling and normalization: compare the methods

Imagine comparing samples using two measurements: one is usually around 13, while the other is around 1000. A distance calculation can be dominated by differences in the second column simply because its numbers are larger. That is not evidence that the measurement is more useful for the task.

Feature scaling changes the numerical representation of each input feature using a consistent rule. Standardization and min-max scaling are two common methods. Row normalization instead rescales each complete observation according to its own length. These operations answer different questions; the word “normalization” alone does not tell you which was used.

Our main example uses the Wine dataset used in scikit-learn’s scaling tutorial. Its alcohol and proline columns are recorded measurements with different numerical scales; proline is an amino acid measured in the samples. You do not need chemistry knowledge—we are comparing representations, not interpreting laboratory findings. Later small examples make the arithmetic easy to inspect.

Here are two actual rows from the packaged Wine dataset. The row position identifies a sample in the dataset; it is not a model input.

A Euclidean distance combines feature differences by squaring them, adding the results and taking a square root. These two rows contribute 1.03² = 1.0609 from alcohol, but 15² = 225 from proline. The proline difference contributes far more to this raw calculation because of the numerical scales we used.

If a unit conversion divides all recorded proline values by 1000, their numerical difference becomes 0.015. The physical information has not changed, but the raw distance now weights it very differently. A scale-sensitive method should not acquire an accidental feature preference from a convenient choice of units.

Scaling helps define the comparison, but does not discover the right importance for each feature. Equalized spreads are a modelling choice. Later, K-nearest neighbors will use distances to choose nearby examples; the representation directly affects those choices.

The mean is the arithmetic average. The standard deviation summarizes how far values typically lie from that mean. Standardization subtracts the fitted mean from each value, then divides by the fitted standard deviation: z = (x − mean) ÷ standard deviation.

A standardized zero is at the fitted mean. A positive value is above it; a negative value is below it. A result of two means two fitted standard-deviation units above the mean, not twice the original measurement.

For a non-constant training column, the transformation centers its mean at zero and scales its standard deviation to one, up to numerical precision. The later dataset need not have those statistics: its values are transformed with the original training rule. The transformation also preserves a skewed distribution’s shape rather than turning it into a normal, bell-shaped distribution.

Scikit-learn’s StandardScaler uses a variance calculation dividing by the number of training observations, equivalent to ddof=0. Other statistical conventions can give different scales. See the StandardScaler API for its definition and constant-column behavior.

Prepare and run

Run this complete Python program in the course environment with NumPy, pandas and scikit-learn. Input values are included or loaded from scikit-learn’s packaged dataset; no external file is needed.

Python · complete runnable example

import numpy as np
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler

# Three samples and two features: a mechanism demonstration.
train = load_wine(as_frame=True).data[["alcohol", "proline"]].iloc[:3]
scaler = StandardScaler().fit(train)
print("Original rows:", train.to_numpy().tolist())
print("Training means:", scaler.mean_.round(3).tolist())
print("Standardized rows:", scaler.transform(train).round(3).tolist())

# Change one measurement's numerical unit, then fit a separate scaler.
other_units = train.copy()
other_units["proline"] = other_units["proline"] / 1000
other_scaler = StandardScaler().fit(other_units)
print("Same standardized values after unit change:", np.allclose(
    scaler.transform(train), other_scaler.transform(other_units)
))

Expected output

Original rows: [[14.23, 1065.0], [13.2, 1050.0], [13.16, 1185.0]]
Training means: [13.53, 1100.0]
Standardized rows: [[1.413, -0.579], [-0.666, -0.828], [-0.747, 1.407]]
Same standardized values after unit change: True

The loader returns the packaged Wine table. We select two named features and the first three samples, then fit one scaler that learns separate statistics for each column. This tiny selected subset demonstrates the mechanism; it is not a sufficient or representative model-training experiment.

Alcohol’s fitted mean is 13.53, while proline’s is 1100. The first sample is above the alcohol mean but below the proline mean, so its transformed values have different signs. A standardized number is relative to that feature’s fitted statistics; its original physical unit has been replaced by a relative numerical scale.

The second part copies the table, divides proline by 1000 and fits a separate scaler to that alternate unit representation. Its mean and standard deviation change by the same factor, leaving equivalent standardized values within numerical tolerance. Allclose checks that equality; round and tolist make the displayed values readable.

This is a controlled comparison of two consistent unit systems. During actual prediction, do not refit a new scaler for every batch. Keep the training-fitted scaler and the original input units together.

Default min-max scaling computes (x − training minimum) ÷ (training maximum − training minimum). For training values 10, 20 and 30, the fitted minimum is 10 and the range is 20. The training endpoints become zero and one; 20 lies halfway between them and becomes 0.5.

The zero does not mean “no amount.” Original value 10 maps to zero because it is the fitted minimum, just as a standardized zero means the fitted mean. Keep this distinction when interpreting prepared inputs.

If a later measurement is 40, apply the same formula: (40 − 10) ÷ 20 = 1.5. A value below 10 can similarly become negative. Default min-max scaling preserves the fitted reference rather than guaranteeing every future value stays in the training interval.

MinMaxScaler can clip later values when explicitly configured with clip=True, but clipping loses information outside the endpoints. Do not mistake a clipped input for evidence that the underlying measurement stayed within range. The MinMaxScaler reference documents these choices.

Prepare and run

Run this complete Python program in the course environment with NumPy, pandas and scikit-learn. Input values are included or loaded from scikit-learn’s packaged dataset; no external file is needed.

Python · complete runnable example

from sklearn.preprocessing import MinMaxScaler, StandardScaler

train = [[10.0], [20.0], [30.0]]
new = [[40.0]]
minmax = MinMaxScaler().fit(train)
standard = StandardScaler().fit(train)
print("Training min-max:", minmax.transform(train).ravel().tolist())
print("New min-max:", minmax.transform(new)[0, 0])
print("Training mean:", standard.mean_[0])
print("Training standard deviation:", round(standard.scale_[0], 3))
print("New standardized:", round(standard.transform(new)[0, 0], 3))

Expected output

Training min-max: [0.0, 0.5, 1.0]
New min-max: 1.5
Training mean: 20.0
Training standard deviation: 8.165
New standardized: 2.449

These invented values isolate the arithmetic; they are not additional Wine records. The two scalers fit on the same three training measurements. The new value is transformed afterward and does not influence either set of statistics.

Min-max uses minimum 10 and maximum 30, so 40 becomes 1.5. Standardization uses mean 20. Its variance is ((10 − 20)² + (20 − 20)² + (30 − 20)²) ÷ 3 = 200 ÷ 3, giving standard deviation about 8.165. Therefore (40 − 20) ÷ 8.165 is about 2.449.

Fit learns the statistics; transform reuses them. Ravel flattens a one-column result into a sequence for display. Neither scaler is clipping the new measurement or predicting a target. Their different results use different reference scales.

A feature column holds one measurement across many samples. A row holds the full feature vector for one sample. Column scaling uses shared statistics for each feature. Row normalization instead divides every component of one row by that row’s own length.

For L2 normalization, a row [3, 4] has length √(3² + 4²) = 5. Dividing by five gives [0.6, 0.8], whose length is one. Row [6, 8] has length ten and becomes the same [0.6, 0.8]. The original magnitudes differ, but the component proportions match.

This can be useful for text vectors when a relative pattern of word use matters more than a document’s total count. It discards magnitude, so it is not a harmless default for every table. Normalizing an age/fare row can mix unrelated feature units and make one feature’s representation depend on the other.

Some tutorials call min-max scaling “normalization.” Here, row normalization means the explicit vector-length operation. Check the formula, axis and reference values whenever you encounter the term.

Prepare and run

Run this complete Python program in the course environment with NumPy, pandas and scikit-learn. Input values are included or loaded from scikit-learn’s packaged dataset; no external file is needed.

Python · complete runnable example

import numpy as np
from sklearn.preprocessing import Normalizer

# Illustrative counts for the words school and library in each document.
counts = np.array([[3.0, 4.0], [6.0, 8.0], [0.0, 0.0]])
normalized = Normalizer(norm="l2").transform(counts)
print("Normalized rows:", normalized.tolist())
print("Row lengths:", np.linalg.norm(normalized, axis=1).tolist())

Expected output

Normalized rows: [[0.6, 0.8], [0.6, 0.8], [0.0, 0.0]]
Row lengths: [1.0, 1.0, 0.0]

Each row is an illustrative document and the two columns count the words school and library. Both columns are counts with the same numerical unit. The second document has twice both counts, so L2 normalization gives it the same representation as the first.

The all-zero row has no counts in either selected word feature. There is no direction to rescale, so the normalizer leaves it zero rather than dividing by zero. Nonzero rows have unit length; the zero row does not.

Normalizer applies each row’s own rule without learning cross-row means or ranges. Its transform method can therefore operate here without fit; in a model pipeline it still follows the standard transformer interface. Np.linalg.norm with axis=1 checks each resulting row’s length. The Normalizer API describes this operation.

We have removed the distinction between three/four and six/eight counts. If document size itself matters to the task, retain or model that information deliberately rather than assuming normalization improved the inputs.

Distance-based methods such as KNN and K-means can change their comparisons when feature scales change. Many gradient-based and regularized models are also sensitive to representation. Standardization is a useful candidate for such workflows, but you still need to evaluate it on the task.

PCA seeks directions of variation, so columns with larger variance can dominate an unscaled analysis. Whether to standardize depends on whether those original differences in variation should matter. We will revisit this in PCA and feature variance.

Ordinary decision trees and random forests that split on individual feature thresholds generally do not need standardization merely to equalize feature scales. A positive affine rescaling preserves each feature’s ordering and corresponding split choices in the usual setting, subject to numerical details. This is not a statement about every tree-based implementation or every transformation.

Choose a feature representation consciously. Scaling an arbitrary category code does not remove its invented order, and scaling a passenger identifier does not make it useful. These decisions connect to categorical encoding and feature meaning.

Prepare and run

Run this complete Python program in the course environment with NumPy, pandas and scikit-learn. Input values are included or loaded from scikit-learn’s packaged dataset; no external file is needed.

Python · complete runnable example

from sklearn.preprocessing import MinMaxScaler, RobustScaler

# Invented values to isolate the effect of one unusually large observation.
train = [[10.0], [20.0], [30.0], [40.0], [1000.0]]
minmax = MinMaxScaler().fit(train)
robust = RobustScaler().fit(train)
print("Min-max values:", minmax.transform(train).ravel().round(4).tolist())
print("Robust median:", robust.center_[0])
print("Robust IQR:", robust.scale_[0])
print("Robust values:", robust.transform(train).ravel().tolist())

Expected output

Min-max values: [0.0, 0.0101, 0.0202, 0.0303, 1.0]
Robust median: 30.0
Robust IQR: 20.0
Robust values: [-1.0, -0.5, 0.0, 0.5, 48.5]

The invented final value 1000 stretches the min-max range to 990. The first four values are compressed into roughly 0–0.03. Scaling has not removed the unusual observation; it has changed how the ordinary values are represented relative to it.

Default RobustScaler centers by the training median and scales by the interquartile range. Here the median is 30, Q1 is 20 and Q3 is 40, so IQR is 20. Value 10 becomes (10 − 30) ÷ 20 = −1, while 1000 becomes 48.5.

Using central statistics makes this scale less sensitive to the extreme observation, but does not delete, cap or prove it invalid. Standardization’s mean and standard deviation can also be affected by extremes. Review outlier investigation before deciding that a numerical tail should disappear, and see the RobustScaler API for its settings.

The mean, standard deviation, endpoints and quartiles are learned preparation settings. For a model evaluation, estimate them from training inputs and apply the fitted transformation to both training and reserved test inputs. Inspecting test values to select the fitted scale lets evaluation records influence learning.

Do not independently standardize a prediction batch. Its zero would mean the batch’s mean rather than the mean the model learned to interpret. Similarly, fitting min-max to a single new row can map a measurement to an uninformative endpoint instead of its position on the training scale.

A pipeline keeps preparation with the model. During cross-validation, the scaler should be fitted within each fold’s training portion; scaling the entire dataset once beforehand leaks held-out feature information. The next lesson on training and test splits introduces the separation, and cross-validation extends it.

Row normalization is different: the rule uses a row’s own length rather than learned dataset statistics. It still needs to be an intentional, consistently applied representation. Different references mean different leakage and interpretation questions.

You fit a min-max scaler on 10, 20 and 30. Predict the output for a later value of 5, and explain why refitting on that later row would be a different procedure. The full code is included here.

from sklearn.preprocessing import MinMaxScaler

train = [[10.0], [20.0], [30.0]]
scaler = MinMaxScaler().fit(train)
print("Later value 5:", scaler.transform([[5.0]])[0, 0])

Worked answer

Later value 5: -0.25

The fitted range is 30 − 10 = 20. The unchanged formula gives (5 − 10) ÷ 20 = −0.25. This is a valid position below the fitted minimum, not a prediction error or a reason to silently refit.

A fit on only [[5.0]] has no observed variation and would not preserve the training reference. Likewise, fitting on training plus the later value would change the minimum and every transformed training position. Keep the fitted preparation rule stable, and investigate changing input ranges separately.

Second check: row [6, 8] becomes [0.6, 0.8] under L2 normalization. Would that let you recover the original total count? No: [3, 4] has the same result, so magnitude has been lost unless retained separately.

A constant training column has no observed spread. StandardScaler avoids division by zero by using a scale of one for a zero-variance column; default centering makes its training values zero. MinMaxScaler also handles a zero training range internally. Neither creates evidence of variation that the training data did not contain.

Missing-value handling is a separate policy. StandardScaler and MinMaxScaler can preserve NaN entries while learning from observed values, but that does not make every later model accept missing inputs. Use a deliberate imputation or estimator-specific policy, and keep the preparation sequence inside the training workflow.

Sparse matrices store mainly nonzero entries. Subtracting a mean would turn many implicit zeros into nonzeros, so default centering is unsuitable for sparse input. StandardScaler with with_mean=False can scale supported sparse data without centering; the representation and resource trade-offs need attention.

Scaling does not repair incorrect units, remove outliers, establish causal relationships or guarantee better accuracy. For ordinary unregularized least squares with an intercept, an invertible rescaling generally changes coefficient units rather than the set of possible predictions. Evaluation should justify the final choice.

The Wine example showed why raw units can dominate a distance calculation. We compared shared feature scales with a row-based operation, keeping their assumptions separate.

• Standardization: subtract a fitted mean and divide by a fitted standard deviation. Zero means the training mean, and distribution shape is retained.
• Min-max scaling: express a value relative to fitted endpoints. Later values can exceed the usual zero-to-one interval.
• Robust scaling: use median and IQR to reduce sensitivity of the fitted scale to extremes. Unusual observations remain present.
• Row normalization: divide a whole observation by its own length. It preserves direction while removing magnitude from nonzero rows.
• Evaluation: learn shared statistics on training data, reuse them consistently and choose according to the algorithm and task.

Before the next lesson: explain why calculating a scaler’s statistics from the test records would influence preparation. Next, we will build the training/test separation that keeps evaluation meaningful.