Principal component analysis (PCA)

A school records height and arm span in centimetres. Taller students often have longer arm spans, so the two columns may contain overlapping information. We want a smaller representation of those measurements, while knowing which differences it leaves out.

Principal component analysis (PCA) finds new directions along which the input records vary. It expresses each record using coordinates along those directions. Keeping fewer directions reduces the number of features: two input numbers can become one component coordinate. This is dimensionality reduction; a dimension here means one coordinate needed to represent a record.

PCA combines measurements rather than selecting a favourite original column. It is unsupervised learning: no correct target or student category is supplied. Our six measurements are invented to make the calculation inspectable; they establish nothing about real students or their abilities.

Before starting, be comfortable with rows, columns and array shapes and feature scaling. We will introduce the remaining mathematical terms as we use them.

Each row is one invented student; the columns are height and arm span, both in centimetres. The rows are (150, 152), (152, 150), (158, 160), (160, 158), (166, 168) and (168, 166). There are six observations and two features, so the array shape is (6, 2). No names or outcomes are needed to fit PCA.

Students differ mainly in their overall size: the small pair is near (151, 151), the middle pair near (159, 159), and the large pair near (167, 167). There is also a smaller difference within each pair: one measurement is two centimetres longer than the other.

Variation means how much recorded values differ across observations. Variance measures spread using squared deviations from a mean. PCA looks for directions with large projected variance, not for features with the most useful names or the strongest relationship to a target.

In the illustration, height increases to the right and arm span increases upward. Moving diagonally up and right changes both together. Moving across that diagonal changes their difference. Those two movements carry different amounts of variation in this particular table.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. All data is included or bundled with scikit-learn; no separate download is required.

Python · complete runnable example

import numpy as np
from sklearn.decomposition import PCA

X = np.array([[1., 1.], [2., 2.], [3., 3.], [4., 4.]])
pca = PCA(n_components=1).fit(X)
reduced = pca.transform(X)
restored = pca.inverse_transform(reduced)
print("Reduced shape:", reduced.shape)
print("Variance retained:", round(pca.explained_variance_ratio_[0], 3))
print("Reconstruction error:", round(float(np.mean((X - restored) ** 2)), 6))

Expected output

Reduced shape: (4, 1)
Variance retained: 1.0
Reconstruction error: 0.0

Here the input rows are (1, 1), (2, 2), (3, 3) and (4, 4). These are abstract numbers, not the student measurements. Both columns are always equal, so every point lies on one diagonal line. A single position along that line is enough to reconstruct both numbers.

np.array creates the numerical table. PCA(n_components=1) asks for one retained direction; fit learns the means and direction. transform creates the new coordinates, and inverse_transform maps them back into the original two-column space.

The shape (4, 1) still means four records; only the feature count changed. Retaining 1.0 of the input variance and displaying zero reconstruction error are special consequences of these exactly equal columns. Most real measurements do not fit perfectly on one line.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. All data is included or bundled with scikit-learn; no separate download is required.

Python · complete runnable example

import numpy as np
from sklearn.decomposition import PCA

# Invented height and arm-span measurements in centimetres.
X = np.array([[150., 152.], [152., 150.], [158., 160.],
              [160., 158.], [166., 168.], [168., 166.]])
pca = PCA(n_components=1, svd_solver="full").fit(X)
reduced = pca.transform(X)
print("Feature means:", pca.mean_.tolist())
print("Direction magnitudes:", np.round(np.abs(pca.components_[0]), 3).tolist())
print("Reduced shape:", reduced.shape)
print("Variance retained:", round(float(pca.explained_variance_ratio_[0]), 3))

Expected output

Feature means: [159.0, 159.0]
Direction magnitudes: [0.707, 0.707]
Reduced shape: (6, 1)
Variance retained: 0.977

The means are 159 cm for each feature. The direction has equal weight magnitudes, about 0.707, because the largest spread in this symmetric dataset runs along the diagonal. components_[0] contains two direction weights, not six student coordinates. reduced contains the six new coordinates.

For reporting we use np.abs, which takes absolute values. Either sign of a direction describes the same axis; the corresponding coordinates reverse sign too. The next sections choose the positive diagonal to make hand calculations consistent.

About 97.7% of the input variance remains in one component. That percentage describes these measurements, not student performance, classifier accuracy or a guarantee about future data. The second direction contains the remaining differences.

Reproduce the inputs before tracing them: (150, 152), (152, 150), (158, 160), (160, 158), (166, 168) and (168, 166), with columns height and arm span in centimetres. Each column has mean 159. Centering subtracts its learned mean from every value in that column.

The first student's (150, 152) becomes (−9, −7). The second student's (152, 150) becomes (−7, −9). These are offsets from the average measurements, not negative heights. The average centered point is (0, 0).

Centering lets PCA describe the spread around the observations' mean rather than around an arbitrary numerical origin. Scikit-learn's PCA does this automatically during fitting. It does not automatically divide each input column by its standard deviation; we will examine that separate scaling decision later.

For new records, subtract the same training means. Recomputing the means for every new batch would change the coordinate system and make its values inconsistent with the fitted representation.

Use the first student again: original measurements (150, 152), learned means (159, 159), centered offsets (−9, −7). Choose the positive diagonal with weights (1/√2, 1/√2), approximately (0.707, 0.707). Its length is one; this convention makes the coordinate a distance along the axis.

Projection measures position along a direction. Multiply each centered feature by its weight and add: (−9 × 0.707) + (−7 × 0.707) ≈ −11.314. The negative coordinate means the student lies below the mean along this chosen positive axis. It is not an error or a negative measurement.

The second centered student (−7, −9) gives the same coordinate. The two students differ in which measurement is longer, but their position along the retained diagonal is identical. That lost distinction is exactly why one coordinate cannot reconstruct both original records perfectly.

PCA chooses the first direction to maximize the variance of these projected coordinates. Later directions are perpendicular to earlier ones and capture the largest remaining variance under that constraint. In two dimensions, perpendicular means meeting at a right angle. PCA computes the directions from data; our symmetric example makes them easy to recognize by inspection.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. All data is included or bundled with scikit-learn; no separate download is required.

Python · complete runnable example

import numpy as np
from sklearn.decomposition import PCA

# Invented height and arm-span measurements in centimetres.
X = np.array([[150., 152.], [152., 150.], [158., 160.],
              [160., 158.], [166., 168.], [168., 166.]])
for count in [1, 2]:
    pca = PCA(n_components=count, svd_solver="full").fit(X)
    reduced = pca.transform(X)
    restored = pca.inverse_transform(reduced)
    print("Components:", count, "shape:", reduced.shape)
    print("First reconstruction:", np.round(restored[0], 3).tolist())
    print("Mean squared error:", round(float(np.mean((X - restored) ** 2)), 6))

Expected output

Components: 1 shape: (6, 1)
First reconstruction: [151.0, 151.0]
Mean squared error: 1.0
Components: 2 shape: (6, 2)
First reconstruction: [150.0, 152.0]
Mean squared error: 0.0

This program repeats the entire student table and fits separate one- and two-component representations. inverse_transform adds the learned means back after mapping the retained coordinates into the original feature space. It returns an approximation when a varying direction was discarded.

For one component, the first student becomes (151, 151): start with coordinate −11.314, multiply by each diagonal weight to obtain about (−8, −8), then add (159, 159). The reconstruction overestimates height by 1 cm and underestimates arm span by 1 cm. The second student reconstructs to the same point.

(X - restored) ** 2 squares each individual feature error. np.mean averages over all 12 entries, giving a mean squared reconstruction error of 1 cm². This is an input reconstruction metric, not a supervised prediction error.

Two components preserve both varying directions here: the shape remains (6, 2), the first reconstruction returns (150, 152), and the error rounds to zero. Toggle the introduction's component controls to compare the same positions visually. Keeping both directions transforms the representation without reducing its dimension.

For the six student measurements, the centered first-direction coordinates are approximately −11.314, −11.314, 0, 0, 11.314 and 11.314. Their squared values sum to 512. The perpendicular direction's squared coordinates sum to 12. Total squared variation is therefore 524, and the retained share is 512 ÷ 524 ≈ 0.977.

explained_variance_ratio_ reports the share associated with each fitted component. A cumulative share adds the shares of the first several components. Keeping both components gives 100% of this table's variation. In larger tables, inspect that cumulative share to understand what a chosen dimension count retains.

A 95% variance target does not mean 95% prediction accuracy, or 95% of every useful fact. Suppose a later target depended on whether arm span exceeds height. Our first component maps each opposite pair to the same position and discards precisely that distinction, even though it keeps almost all the input variance.

High variance can also reflect measurement errors or irrelevant variation. PCA does not know which changes matter to a later goal. Use appropriate evaluation metrics for that goal rather than treating variance retention as a task score.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. All data is included or bundled with scikit-learn; no separate download is required.

Python · complete runnable example

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Invented height and arm-span measurements in centimetres.
X = np.array([[150., 152.], [152., 150.], [158., 160.],
              [160., 158.], [166., 168.], [168., 166.]])
# Change only the arm-span column from centimetres to millimetres.
X_mm = X.copy()
X_mm[:, 1] *= 10
for name, values in [("Both cm", X), ("Mixed units", X_mm)]:
    pca = PCA(n_components=1, svd_solver="full").fit(values)
    print(name, "raw direction:", np.round(np.abs(pca.components_[0]), 3).tolist())
    standardized = StandardScaler().fit_transform(values)
    scaled_pca = PCA(n_components=1, svd_solver="full").fit(standardized)
    print(name, "standardized direction:",
          np.round(np.abs(scaled_pca.components_[0]), 3).tolist())

Expected output

Both cm raw direction: [0.707, 0.707]
Both cm standardized direction: [0.707, 0.707]
Mixed units raw direction: [0.095, 0.995]
Mixed units standardized direction: [0.707, 0.707]

The observations have not changed physically; only arm-span units changed. But its numbers become ten times larger, and its variance becomes 100 times larger. Raw PCA responds to numerical geometry, so its first direction in mixed units is dominated by arm span.

X.copy() prevents the unit conversion from modifying the original table. X_mm[:, 1] selects the entire second column. The displayed direction magnitudes belong to each representation's coordinate system; they are not feature-importance scores for a target.

StandardScaler centers each column and divides by its training standard deviation. In this constructed example, it cancels the positive unit conversion, yielding matching standardized directions. These coordinates are in standardized units; reconstruction with PCA alone returns standardized values, so apply the scaler's inverse transformation too if you need centimetres or millimetres.

Standardizing is a choice, not an automatic improvement. Equal numerical weight can be useful when units are arbitrary, but original variance differences may carry meaning when comparable measurements use the same units. Review how standardization differs from normalization and choose deliberately.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. All data is included or bundled with scikit-learn; no separate download is required.

Python · complete runnable example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# Four flower measurements; species labels are not used by PCA.
X = load_iris().data
X_train, X_test = train_test_split(X, test_size=0.2, random_state=7)
representation = make_pipeline(StandardScaler(), PCA(n_components=2, svd_solver="full"))
train_coordinates = representation.fit_transform(X_train)
saved_mean = representation.named_steps["standardscaler"].mean_.copy()
test_coordinates = representation.transform(X_test)
print("Training:", X_train.shape, "->", train_coordinates.shape)
print("Test:", X_test.shape, "->", test_coordinates.shape)
print("Training means unchanged:",
      bool(np.array_equal(saved_mean, representation.named_steps["standardscaler"].mean_)))

Expected output

Training: (120, 4) -> (120, 2)
Test: (30, 4) -> (30, 2)
Training means unchanged: True

This separate example uses the bundled Iris dataset: 150 flower records with four measurements. train_test_split separates 120 training rows and 30 test rows. No species labels are passed to the representation. make_pipeline connects scaling to PCA so the stages run in order.

fit_transform(X_train) learns both stages from training records and returns two coordinates per row. transform(X_test) uses the existing means, scales and directions. The final check confirms that transforming the test rows did not replace the scaler's training means.

The two shapes establish a reusable representation, not a successful classifier. To evaluate a supervised task, add the predictor to the pipeline, supply training targets and evaluate separate records. When comparing component counts with cross-validation, fit the entire pipeline inside each training fold. Transforming the full dataset before splitting would leak evaluation information into the learned geometry.

A new measurement far outside the training population can still receive coordinates; that does not make its reconstruction trustworthy. Distribution changes and unsuitable scales remain reasons to inspect the data.

For visualization, two or three components can create a plot from many measurements, but nearby points in the plot need not preserve every relationship from the original space. For compression or denoising, inspect reconstructions and confirm that discarded directions are acceptable; small variation is not automatically noise.

For a predictive model, compare candidate counts using the downstream validation score and practical cost. Reducing features can make some workflows cheaper, but fitting PCA adds work and can reduce accuracy. A full representation remains a useful comparison.

Scikit-learn also supports a retained-variance target with a full solver, such as PCA(n_components=0.95, svd_solver="full"). This selects enough components to exceed that share on the fitting data. It does not promise that future observations retain the same proportion or that a target relationship is preserved.

PCA is feature extraction: it creates combinations of features. Feature selection keeps a subset of original columns instead. New components can be harder to name or explain because several input meanings are mixed together. Keep the original measurements and fitted weights available for interpretation.

Your task: The fitted training table below has means (159, 159). A new invented student has measurements (170, 172). Predict the shape and one-component reconstruction. Will transforming this row replace the learned means?

Python · complete practice program

import numpy as np
from sklearn.decomposition import PCA

# Invented height and arm-span measurements in centimetres.
X = np.array([[150., 152.], [152., 150.], [158., 160.],
              [160., 158.], [166., 168.], [168., 166.]])
pca = PCA(n_components=1, svd_solver="full").fit(X)
new_measurements = np.array([[170., 172.]])
coordinates = pca.transform(new_measurements)
restored = pca.inverse_transform(coordinates)
print("New shape:", coordinates.shape)
print("Reconstructed:", np.round(restored[0], 3).tolist())
print("Feature means:", pca.mean_.tolist())

Worked answer

The centered offsets are (11, 13). Along the positive diagonal their coordinate is 24/√2 ≈ 16.971. Multiplying back along that diagonal gives (12, 12), then adding the training means gives (171, 171). One coordinate describes one row, so its shape is (1, 1). The learned means stay at (159, 159).

New shape: (1, 1)
Reconstructed: [171.0, 171.0]
Feature means: [159.0, 159.0]

The program may represent the axis with opposite signs; both the coordinate and weights reverse together, preserving reconstruction. An axis is not uniquely wrong because it points in the opposite direction. If you change n_components=1 to n_components=2, the shape becomes (1, 2) and this two-dimensional example reconstructs (170, 172), within numerical precision.

Ordinary PCA uses straight directions. Curved relationships can need several linear components even when another representation would describe them more compactly. Kernel PCA and other dimensionality-reduction methods make different choices; none should be assumed to preserve every relationship or provide causal explanations.

Outliers can dominate squared variation, so investigate unusual values before interpreting components. Missing values and categories also need an appropriate representation; assigning arbitrary numeric category codes invents geometry. Revisit missing values and outliers and categorical encoding where necessary.

Component signs can reverse, and equal or nearly equal variances can make individual axes ambiguous or unstable. Compare the retained space and reconstruction rather than demanding identical weights in every fit. Whitening is an additional option that rescales component coordinates; it is not the same as standardizing input columns, and this lesson uses the default without whitening.

Computation depends on the number of rows, columns and chosen solver. Incremental PCA can process batches and randomized approaches can approximate a smaller set of directions, with their own trade-offs. Our tiny arrays are not performance benchmarks. For implementation details, see the official PCA guide and PCA parameter and attribute reference.

• Components combine features: they are learned directions, not selected original columns or target labels.
• Trace the transformation: subtract training means, project onto retained axes and map back with the same means to reconstruct.
• Inspect loss: one component merges each student pair at its diagonal midpoint; two preserve both directions here.
• Read variance carefully: retaining most input spread does not guarantee preservation of a later prediction target.
• Fit responsibly: choose input scales deliberately and learn every representation stage from training records.

Before moving on: explain why (150, 152) reconstructs as (151, 151), why changing units can change raw PCA, and how a new row uses the existing fitted directions. Next, cross-validation will help us compare choices using multiple training and validation splits.