Model evaluation metrics: understand the mistakes behind a score

A spam filter that lets every message through can score 95% accuracy in an inbox where 95 of 100 messages are legitimate. It also misses every spam message. The number is arithmetically correct, but it does not answer whether the filter does its job.

An evaluation metric summarizes a particular aspect of predictions compared with known outcomes. Classification metrics compare categories; regression metrics compare numerical values. Choose the question first: are we measuring false alarms, missed spam, average price errors or the ranking of probability scores?

We will inspect invented spam decisions and house-price estimates with all values visible. These calculations do not train a model or establish real performance. Predictions being evaluated should come from held-out records, as in training/test separation. Review probabilities and thresholds or numerical prediction errors if needed.

Positive means spam in this lesson, not a desirable outcome. A true positive is caught spam; a false positive is a legitimate message wrongly flagged. A false negative is missed spam; a true negative is a legitimate message correctly unflagged.

Message four could be a genuine school notice sent to a spam folder: a false alarm. Messages two and three are spam left in the inbox: misses. Those consequences can have different costs, even though accuracy counts each mistake equally.

The actual label and prediction for each record must stay paired. Comparing a reordered prediction array with original labels can produce a plausible-looking but meaningless metric. Count identities and class definitions matter before any formula.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.metrics import (
    confusion_matrix, accuracy_score, precision_score, recall_score, f1_score
)

# Invented held-out labels and decisions; 1 means spam, 0 legitimate.
actual = [1, 1, 1, 0, 0, 0]
predicted = [1, 0, 0, 1, 0, 0]
print("Matrix [TN FP; FN TP]:",
      confusion_matrix(actual, predicted, labels=[0, 1]).tolist())
for name, metric in [("Accuracy", accuracy_score), ("Precision", precision_score),
                     ("Recall", recall_score), ("F1", f1_score)]:
    print(name + ":", round(metric(actual, predicted), 3))

Expected output

Matrix [TN FP; FN TP]: [[2, 1], [2, 1]]
Accuracy: 0.5
Precision: 0.5
Recall: 0.333
F1: 0.4

This complete Python program repeats the six message pairs. Label 1 is spam and 0 is legitimate. confusion_matrix(actual, predicted, labels=[0, 1]) places actual labels on rows and predicted labels on columns. The first row is TN, FP; the second is FN, TP. An explicit label order avoids interpreting a matrix backwards.

The matrix contains TN = 2, FP = 1, FN = 2 and TP = 1. Three of six decisions are correct, so accuracy is 0.5. Half the flagged messages are spam, but only one third of actual spam is caught. These measures are different because their denominators differ.

The loop calls one metric function at a time on the same paired arrays. Scikit-learn’s binary precision, recall and F1 defaults use label 1 as positive here; other labels require an appropriate pos_label. Round formats the display, not the underlying counts. The matrix API specifies the orientation.

Accuracy asks “What fraction of all decisions were correct?” Its formula is (TP + TN) / total. With our counts, (1 + 2) / 6 = 0.5. It treats each mistake as one wrong decision, regardless of its practical cost.

Precision asks “Of the messages flagged as spam, how many really were spam?” Its denominator is every flagged message: TP + FP. Here 1 / (1 + 1) = 0.5. Think of inspecting the spam folder: half its messages in this example belong there.

Recall asks “Of all actual spam messages, how many were caught?” Its denominator is TP + FN. Here 1 / (1 + 2) = 1/3, approximately 0.333. Think of starting with the three actual spam messages and checking which ones the filter found.

A filter can flag very few messages and achieve high precision while missing much spam. Another can flag almost everything and achieve high recall while wrongly flagging legitimate messages. Report the relevant counts and both questions rather than calling either score “the model’s correctness”. The precision reference documents the denominator and positive-class setting.

F1 combines precision and recall using their harmonic mean: 2 × precision × recall / (precision + recall), when that expression is defined. An equivalent binary count formula is 2TP / (2TP + FP + FN). Our example gives 2 / (2 + 1 + 2) = 0.4.

F1 is not the ordinary average. For precision 0.5 and recall 1/3, their ordinary average is about 0.417, while F1 is 0.4. The harmonic mean limits a high result when one of the two components is low.

F1 omits true negatives and does not encode the actual monetary or personal cost of each error. Two systems can have similar F1 but different error counts or practical consequences. It is useful when this precision/recall balance matches the task; it is not a universal replacement for accuracy or a complete deployment decision.

The F1 API explains the definition and averaging options. Always include the positive class and the data used for evaluation.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.metrics import accuracy_score, precision_score, recall_score

# Invented evaluation inbox: 95 legitimate messages and 5 spam messages.
actual = [0] * 95 + [1] * 5
predicted = [0] * 100  # Fixed policy: never flag spam.
print("Accuracy:", round(accuracy_score(actual, predicted), 3))
print("Spam recall:", recall_score(actual, predicted))
print("Precision with declared zero policy:",
      precision_score(actual, predicted, zero_division=0))

Expected output

Accuracy: 0.95
Spam recall: 0.0
Precision with declared zero policy: 0.0

The program creates an invented inbox with 95 legitimate messages and five spam messages. It applies a fixed never-flag policy rather than fitting a classifier or selecting a policy from these labels. Multiplication repeats the label values to form the arrays.

All 95 legitimate messages are correctly unflagged, so accuracy is 95/100. All five spam messages are missed, so recall is 0/5 = 0. No message is flagged; precision would require 0/0 and is undefined.

zero_division=0 deliberately reports zero for that undefined precision. It does not make the original fraction mathematically defined. The default may warn; declare the policy when reporting such results. A zero denominator must never be presented as perfect precision.

Compare a proposed filter with a relevant simple baseline and include class counts. A learned baseline, such as predicting the training majority class, must learn that choice from training labels rather than consulting the test distribution.

This separate six-message score set is invented. It is not the earlier six-decision example and is not an output from a trained spam model. Class one means spam. A message is predicted positive when its score is at least the selected threshold. Changing the threshold changes decisions without changing scores or fitting a model.

Read the trade-off

TP means caught spam, FP wrongly flagged legitimate messages, FN missed spam and TN correctly unflagged legitimate messages. At threshold 0.80, two messages are predicted spam: one true positive and one false positive. Recall falls to 1/3. At threshold 1.00, none are predicted spam, so precision is undefined and displayed as —. These particular scores are illustrative and are not claimed to be calibrated probabilities.

The lab kept scores fixed and changed the cutoff for flagging a message. A lower cutoff flags more messages; on a fixed set it cannot reduce the number of caught positives, but it can add false positives. Precision does not necessarily move monotonically as the cutoff changes.

Choose a cutoff using validation data and the consequences of errors. A spam-folder action may prefer fewer falsely flagged notices, while a separate review queue may accept more false alarms to miss less spam. State the action being evaluated rather than saying “higher recall is always better”.

After choosing the cutoff and fitted procedure, evaluate on a protected test set. Repeatedly changing the cutoff to improve its test score makes that set part of selection. A metric cannot correct leakage or an unsuitable split, even when its calculation is accurate.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.metrics import accuracy_score, f1_score

# Invented labels, not an output of the Iris model.
actual = ["setosa"] * 8 + ["versicolor", "virginica"]
predicted = ["setosa"] * 10
print("Accuracy:", round(accuracy_score(actual, predicted), 3))
print("Macro F1:", round(f1_score(actual, predicted, average="macro", zero_division=0), 3))
print("Weighted F1:", round(f1_score(actual, predicted, average="weighted", zero_division=0), 3))

Expected output

Accuracy: 0.8
Macro F1: 0.296
Weighted F1: 0.711

This invented ten-label example uses familiar Iris species names but is not a dataset model result. Eight actual labels are setosa, one versicolor and one virginica; every prediction is setosa. The number of actual examples for a class is its support.

Accuracy is 8/10, but neither rare species is recognized. Setosa has precision 8/10 and recall 8/8, giving F1 about 0.889. The other two species have F1 zero under the declared zero-division policy.

Macro F1 gives each class equal weight: (0.889 + 0 + 0) / 3, about 0.296. Weighted F1 weights by actual class support: (8 × 0.889 + 1 × 0 + 1 × 0) / 10, about 0.711. The latter looks more favorable because the dominant class contributes more.

State the averaging method and report per-class results when classes matter separately. A weighted score does not prove that rare classes are served well. Multilabel problems, where one record can carry several labels, require additional choices; these examples are ordinary single-label classification.

Classification counts do not describe how far a predicted price is from a sale price. For a regression task, subtract predictions from actual values and measure those numerical gaps. Absolute or squared errors prevent positive and negative misses from cancelling.

Mean absolute error (MAE) averages the absolute gaps. For errors of 0, 0 and 60 thousand dollars, it is (0 + 0 + 60) / 3 = 20 thousand dollars. The target’s units remain meaningful: an average absolute miss of $20,000.

Root mean squared error (RMSE) squares the gaps, averages them, then takes the square root. Here √((0² + 0² + 60²) / 3) is about 34.641 thousand dollars. Squaring gives large mistakes more influence, while the square root returns the result to the price unit.

Three errors of 20 have the same MAE of 20, but RMSE is only 20. Neither summary reveals every individual miss or its direction. Choose according to what matters and inspect the distribution, especially when large errors are costly.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score

# Invented actual and predicted prices, in thousands of US dollars.
actual = [100, 200, 300]
predictions = {
    "A": [100, 200, 360],
    "B": [120, 220, 320],
    "Constant 150": [150, 150, 150]
}
for name, predicted in predictions.items():
    print(name, "MAE:", round(mean_absolute_error(actual, predicted), 3))
    print(name, "RMSE:", round(root_mean_squared_error(actual, predicted), 3))
    print(name, "R2:", round(r2_score(actual, predicted), 3))

Expected output

A MAE: 20.0
A RMSE: 34.641
A R2: 0.82
B MAE: 20.0
B RMSE: 20.0
B R2: 0.94
Constant 150 MAE: 83.333
Constant 150 RMSE: 95.743
Constant 150 R2: -0.375

All values are invented prices in thousands of US dollars. Candidate A estimates 100, 200 and 360 against actual 100, 200 and 300: two exact estimates and one overestimate of 60. Candidate B estimates 120, 220 and 320: three overestimates of 20.

Both have MAE 20, but A’s concentrated large error gives a higher RMSE. B’s consistent overestimates matter too; a single average magnitude should not replace inspecting direction and individual records. These supplied estimates are not outputs from fitted models.

root_mean_squared_error is available in scikit-learn 1.4 and later, including the course environment. The RMSE API describes the calculation. If using another environment, check its version rather than assuming every older installation exposes this function.

R-squared, written R², is 1 − model squared-error sum / squared deviations from the evaluation targets’ mean. For the three actual prices 100, 200 and 300, their mean is 200 and squared deviations sum to 10,000 + 0 + 10,000 = 20,000.

Candidate A’s squared errors total 3,600, so R² = 1 − 3,600/20,000 = 0.82. The constant estimate 150 has squared errors 2,500 + 2,500 + 22,500 = 27,500, giving R² = −0.375. A negative value means more squared error than the reference of always using this evaluation set’s mean.

That evaluation mean is a mathematical reference for the metric, not a prediction policy allowed to learn from test labels. To compare a deployable baseline, fit its constant, such as a mean or median, from training data and evaluate it on the same reserved observations.

R² is not “82% of prices correct” and is not classification accuracy. A perfect fit gives one; negative values are possible. Constant evaluation targets and fewer than two records need special care because the reference variation is zero or the measure is undefined. The R² API documents these edge cases.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.metrics import roc_auc_score, average_precision_score, accuracy_score

# Invented spam scores and known labels. This program fits no model.
actual = [1, 0, 1, 0]
scores = [0.9, 0.8, 0.7, 0.1]
print("ROC-AUC:", round(roc_auc_score(actual, scores), 3))
print("Average precision:", round(average_precision_score(actual, scores), 3))
for threshold in [0.5, 0.85]:
    predicted = [int(score >= threshold) for score in scores]
    print("Accuracy at", threshold, ":", round(accuracy_score(actual, predicted), 3))

Expected output

ROC-AUC: 0.75
Average precision: 0.833
Accuracy at 0.5 : 0.75
Accuracy at 0.85 : 0.75

The four invented scores rank the messages as spam, legitimate, spam, legitimate. Scores are supplied directly; no model is fitted. At cutoff 0.5 the first three messages are flagged, including one false alarm. At cutoff 0.85 only the first is flagged, missing the other actual spam message. Both decision policies happen to have accuracy 3/4, despite different mistakes.

ROC-AUC summarizes ranking of positive versus negative records. There are four positive-negative pairs here; the positive score is higher in three, so AUC is 3/4. In the binary case, a tie contributes half. Use probability or decision scores for this purpose, not just the final 0/1 labels.

Average precision (AP) summarizes precision as recall increases over the ranked scores. Our two positives appear at positions one and three. Precision when each is reached is 1 and 2/3; each adds half the available recall, so AP = 0.5 × 1 + 0.5 × 2/3, about 0.833. AP uses this recall-weighted definition, not arbitrary trapezoidal area under a plotted curve.

Neither ranking score changes when this program changes only the action cutoff. Neither chooses the cutoff for you. See the ROC-AUC and average precision references for definitions and supported inputs.

ROC curves compare the true-positive rate, another name for recall, against false-positive rate across cutoffs. False-positive rate is FP / (FP + TN). Precision-recall curves compare precision against recall instead, which can make the cost of false alarms more visible when positives are rare.

ROC-AUC, AP and a chosen-cutoff metric answer different questions. When class proportions change, precision and AP can change too. Compare relevant populations and baselines rather than comparing scores across unrelated datasets as if they shared the same difficulty. Binary ROC-AUC needs both actual classes; very small evaluation samples give fragile evidence.

A model can rank spam above legitimate messages well while giving poorly calibrated probabilities. Calibration asks whether estimates around 0.8 correspond to roughly 80% observed positives on representative data. It is not implied by high AUC or accuracy. Probability-based measures such as log loss or Brier score inspect additional aspects of probability quality; they are not isolated proofs of calibration either.

The logistic regression lesson separates probability estimates from actions. Choose metrics and error costs before selecting the procedure, then document how they were measured.

Your task: The first example predicted message two as legitimate even though it was spam. Change only that decision to spam. Predict the new four counts, accuracy, precision, recall and F1 before running this complete program.

Python · complete program to modify

from sklearn.metrics import (
    confusion_matrix, accuracy_score, precision_score, recall_score, f1_score
)

# Invented held-out labels and decisions; 1 means spam, 0 legitimate.
actual = [1, 1, 1, 0, 0, 0]
predicted = [1, 0, 0, 1, 0, 0]
print("Matrix [TN FP; FN TP]:",
      confusion_matrix(actual, predicted, labels=[0, 1]).tolist())
for name, metric in [("Accuracy", accuracy_score), ("Precision", precision_score),
                     ("Recall", recall_score), ("F1", f1_score)]:
    print(name + ":", round(metric(actual, predicted), 3))

Worked answer

Use predicted = [1, 1, 0, 1, 0, 0]. Message two changes from a false negative to a true positive; the other five outcomes stay fixed.

Matrix [TN FP; FN TP]: [[2, 1], [1, 2]]
Accuracy: 0.667
Precision: 0.667
Recall: 0.667
F1: 0.667

Now TN = 2, FP = 1, FN = 1 and TP = 2. Accuracy is 4/6; precision is 2/3; recall is 2/3; F1 is 4/(4 + 1 + 1) = 2/3. Their values happen to agree for these counts. The formulas still answer different questions and will not agree for every prediction array.

Report the task, positive class or target unit, number of evaluated records, class supports, metric definition and cutoff or averaging method. Compare candidate procedures on the same observations. Include the baseline and explain whether settings were fixed beforehand or chosen using validation.

A score based on ten examples can change substantially after one additional error. Use an appropriate uncertainty assessment when it matters and respect related records or time structure. An aggregate score can hide poor results for a relevant subgroup; inspect such results where there are enough observations to interpret them responsibly.

Data quality, target availability and independent evaluation still determine what a score supports. Repeatedly selecting a model from a test metric makes that set part of development. A higher score cannot repair the evaluation design or guarantee performance after the data changes.

A useful metric answers a stated question on appropriate evaluation data. The spam examples distinguished false alarms from misses; the price examples showed why one average can hide an important error pattern.

• Start with counts: declare the positive class and keep actual labels paired with predictions.
• Read denominators: accuracy uses all decisions, precision uses positive predictions and recall uses actual positives.
• State the balance: F1 combines precision and recall; class averaging and zero-division policies need explanation.
• Keep price units: MAE and RMSE summarize numerical misses differently. R² is a relative squared-error measure, not percentage correctness.
• Separate score questions: ranking, calibration and chosen-cutoff behavior are distinct.
• Report the context: baseline, evaluation size, split, settings and limitations belong with the number.

Before moving on: recalculate the six-message metrics after correcting one missed spam message, then explain why equal MAE does not imply equally large worst errors. Next, we will explore grouping unlabelled observations with K-means.