Logistic regression: estimate spam probabilities and choose a threshold

An inbox needs a different answer from our house-price model. Instead of estimating an amount, it needs to classify each message as legitimate or spam. Logistic regression is a classification method despite the word “regression” in its name. In this two-class problem, it estimates a probability for each class, then a decision rule selects a label.

Suppose the model estimates a spam probability of 0.64. A rule that flags messages at 0.5 will flag it; a rule that flags at 0.7 will not. The estimate is identical in both cases. The threshold is the cutoff used by the decision rule, not another learned probability.

We will count occurrences of offer-related words as one simple message feature. All records and counts are invented; this deliberate simplification lets you inspect the steps and is not a useful real spam filter. Review classification versus regression and training and evaluation separation first if either distinction is unfamiliar.

For teaching, imagine a fixed word list such as “offer”, “prize” and “winner”. Each input row contains the number of occurrences from that list in one message. We supply these counts directly; this lesson does not include a text parser. The target is the known category, legitimate or spam.

These rows intentionally arrange the labels neatly. A real message can contain an offer and still be legitimate, or contain none of these words and still be spam. A feature offers evidence; it is not the target itself or a guaranteed rule.

The new message’s feature must be calculated using the same definition as training. Its actual label must not be supplied as an input. For a more realistic text workflow, the spam-detection project extracts features from messages rather than relying on this manually supplied single count.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.linear_model import LogisticRegression

# Invented messages: one manually counted feature per row.
# Counts represent occurrences of words from an offer-word list.
X = [[1], [2], [3], [4], [5], [6]]
y = ["legitimate", "legitimate", "legitimate", "spam", "spam", "spam"]
model = LogisticRegression().fit(X, y)
spam_column = list(model.classes_).index("spam")
probabilities = model.predict_proba([[4]])[0]
spam_probability = probabilities[spam_column]
print("Class order:", model.classes_.tolist())
print("Probabilities:", [round(float(p), 3) for p in probabilities])
print("Spam probability:", round(spam_probability, 3))
print("Flag at 0.5:", bool(spam_probability >= 0.5))
print("Flag at 0.7:", bool(spam_probability >= 0.7))

Expected output

Class order: ['legitimate', 'spam']
Probabilities: [0.363, 0.637]
Spam probability: 0.637
Flag at 0.5: True
Flag at 0.7: False

X contains six rows, each holding one feature. The nested brackets preserve the rows-and-columns shape expected by the estimator. y supplies the corresponding labels. fit(X, y) learns parameters from those paired examples.

predict_proba([[4]]) returns one row with two probabilities. The first [0] selects that row. classes_ identifies each column; looking up “spam” gives its column index explicitly. Do not assume every dataset calls the positive class 1 or places your intended class in the second column.

For this four-occurrence query, the spam estimate is about 0.637 and the legitimate estimate about 0.363. They sum to one before display rounding. The query shares a feature value with a training row, so this is an API demonstration, not an independent performance check.

bool converts each comparison to True or False. The two cutoffs give different flags without calling fit again. Decisions use the original probability, not its rounded display. Small solver/version differences may affect final displayed digits; the probability is not exactly 0.637.

The LogisticRegression API documents the class ordering and probability method. Float converts NumPy scalar values to ordinary Python numbers before displaying the list.

Binary logistic regression first combines features into a score: intercept plus coefficient × feature for one input. That score can be negative, zero or positive; it is not itself a probability. The sigmoid transformation converts it to a number between zero and one.

The formula is p = 1 / (1 + exp(−score)). Exp is the exponential function. You do not need to compute it by hand to use the model: a score of zero maps to 0.5, negative scores map below 0.5 and positive scores map above 0.5. Increasing the score increases the estimate smoothly rather than adding the same amount of probability each time.

For binary scikit-learn logistic regression, the positive score corresponds to the second entry in classes_. In our example that entry is “spam”. Our fitted score at four occurrences is about 0.56, which maps to a spam probability around 0.637. These numbers come from fitted settings, not a direct rule that four words mean 64% spam.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from math import exp

# Supplied scores illustrate the transformation; no model is fitted.
for score in [-2, 0, 2]:
    probability = 1 / (1 + exp(-score))
    print("Score", score, "-> probability", round(probability, 3))

Expected output

Score -2 -> probability 0.119
Score 0 -> probability 0.5
Score 2 -> probability 0.881

This standalone Python example supplies three scores to inspect the transformation. It does not fit coefficients, use messages or evaluate a classifier. The loop calculates the formula once for each score and rounds only for display.

The outputs remain in the probability range even when a score is negative. That is one reason logistic regression differs from predicting an amount with a straight line: an ordinary linear prediction has no inherent zero-to-one bound. Logistic regression is not a linear price-style model followed by rounding.

Training adjusts coefficients and the intercept using the known labels. Its probability-based error measure is commonly called log loss. Assigning high probability to the observed category produces a smaller loss; confidently assigning low probability to it produces a larger loss.

For a message known to be spam, a spam estimate of 0.9 is rewarded more than 0.6. An estimate of 0.01 receives a much larger penalty. For a legitimate message, the relevant observed-class probability is the legitimate probability, not the spam probability. Training considers all the labelled records rather than tuning each message separately.

The scikit-learn estimator uses regularization by default: a penalty discourages excessively large coefficients. Its strength is a model setting selected with validation. Regularization helps control a fit; it does not make six invented messages sufficient evidence for deployment.

The official model guide describes this objective and regularization. Choosing a cutoff is a later decision-policy step; it does not change the fitting loss or automatically retrain the coefficients.

A binary decision policy can be “flag as spam when its probability is at least 0.7”. The phrase “at least” matters: a value equal to the threshold is flagged in this lesson. We consistently use probability >= threshold.

Scikit-learn’s standard binary predict uses its default decision rule, ordinarily a 0.5 probability cutoff. To use an application-specific cutoff, obtain the intended class probability and compare it explicitly. Exact ties in library predict follow its implementation; do not assume that a value displayed as 0.500 is an exact tie or necessarily uses our custom equality convention.

The four-occurrence query had probability about 0.637. It is flagged at 0.5 and unflagged at 0.7. Neither choice changes that estimate. A stricter cutoff is not automatically better: it flags fewer messages, including fewer legitimate ones but potentially fewer actual spam messages too.

Choose the policy from the intended use and validation evidence. The threshold-selection guide distinguishes estimating probabilities from taking actions, and explains why thresholds must not be selected using the final test data.

Here, a positive decision means flagging spam. It does not mean a good outcome. Keep that definition fixed when reading error counts.

A false positive could send an important school notice to a spam folder. A false negative could leave a junk message in the inbox. The preferable trade-off depends on what flagging actually does: a reversible folder move and permanent deletion are different actions with different consequences.

Precision asks what fraction of flagged messages really are spam. Recall asks what fraction of actual spam messages were flagged. Accuracy alone can conceal the trade-off, especially if spam is rare. We will inspect counts here; the metrics lesson develops these measures further.

Positive means “flag as spam”: a false positive wrongly flags a legitimate message, while a false negative misses actual spam. These are six invented spam probabilities, not scores from the later dataset model. Class one means spam. A message is predicted positive when its score is at least the selected threshold. Changing the threshold changes decisions without changing scores or fitting a model.

Try it, then read the trade-off

At threshold 0.80, two messages are predicted spam: one true positive and one false positive. Recall falls to 1/3. At threshold 1.00, none are predicted spam, so precision is undefined and displayed as —. These particular scores are illustrative and are not claimed to be calibrated probabilities.

Prepare and run

Use Python with NumPy and scikit-learn in the course environment. The invented data is included; no file or download is needed.

Python · complete runnable example

from sklearn.metrics import confusion_matrix

# Fixed hypothetical validation probabilities, not model outputs.
# Actual 1 means spam; actual 0 means legitimate.
scores = [0.95, 0.80, 0.60, 0.55, 0.30, 0.10]
actual = [1, 0, 1, 0, 0, 1]
for threshold in [0.5, 0.8]:
    decisions = [int(score >= threshold) for score in scores]
    tn, fp, fn, tp = confusion_matrix(actual, decisions, labels=[0, 1]).ravel()
    print("Threshold:", threshold)
    print("Decisions:", decisions)
    print("TP, FP, FN, TN:", int(tp), int(fp), int(fn), int(tn))

Expected output

Threshold: 0.5
Decisions: [1, 1, 1, 1, 0, 0]
TP, FP, FN, TN: 2 2 1 1
Threshold: 0.8
Decisions: [1, 1, 0, 0, 0, 0]
TP, FP, FN, TN: 1 1 2 2

The six probabilities and actual labels are repeated in this complete example. These fixed hypothetical validation scores are independent of the fitted six-message demonstration; no model generated them. The comparison isolates the effect of changing a cutoff.

confusion_matrix with labels=[0, 1] returns rows for actual labels and columns for predicted labels, in that explicit order. Flattening with ravel() gives true negatives, false positives, false negatives and true positives. The print statement rearranges them into the stated TP, FP, FN, TN order.

At 0.5, four messages are flagged and two are genuinely spam. At 0.8, two are flagged and one is genuinely spam. Raising the cutoff reduces false positives from two to one, but raises false negatives from one to two. In this particular illustration, precision stays at one half; threshold changes do not guarantee that every metric improves.

The confusion_matrix reference specifies the row and column order used here.

Your task: Reproduce the six-score comparison below with a single threshold of 0.6. Predict the decisions before running it, then identify the missed spam and wrongly flagged messages. Message positions are numbered 1–6.

Python · complete program to modify

from sklearn.metrics import confusion_matrix

# Fixed hypothetical validation probabilities, not model outputs.
# Actual 1 means spam; actual 0 means legitimate.
scores = [0.95, 0.80, 0.60, 0.55, 0.30, 0.10]
actual = [1, 0, 1, 0, 0, 1]
for threshold in [0.5, 0.8]:
    decisions = [int(score >= threshold) for score in scores]
    tn, fp, fn, tp = confusion_matrix(actual, decisions, labels=[0, 1]).ravel()
    print("Threshold:", threshold)
    print("Decisions:", decisions)
    print("TP, FP, FN, TN:", int(tp), int(fp), int(fn), int(tn))

Worked answer

Replace the loop’s threshold list with [0.6]. Scores 0.95, 0.80 and 0.60 meet the cutoff; the others do not. Equality is included by the comparison.

Threshold: 0.6
Decisions: [1, 1, 1, 0, 0, 0]
TP, FP, FN, TN: 2 1 1 2

Messages 1 and 3 are caught spam. Message 2 is legitimate but flagged, the single false positive. Message 6 is spam but unflagged, the single false negative. Messages 4 and 5 are correctly unflagged. All six scores stayed fixed; we changed only their decision cutoff.

A predicted probability of 0.8 is not a guarantee that one particular message is spam. A well-calibrated model would, on representative evaluation data, have roughly 80% actual spam among messages receiving estimates around 0.8. A sigmoid keeps numbers in range; it does not by itself establish that this frequency interpretation is accurate. Consult the probability calibration guide for how to assess it.

With several features, the model adds one weighted contribution per feature before the sigmoid. Word frequencies and other available message measurements can be inputs, provided their definitions and column order remain consistent. Scaling can matter because the penalty acts on coefficient size; fit preprocessing using only the permitted training data.

The model assumes a linear relationship between the features and log-odds, the score before the sigmoid. Odds compare a probability with its complement; the logarithm of those odds gives this score. The probability curve itself is not a straight line. Some feature interactions or more complicated patterns need different features or another model; the method also supports more than two classes, though the binary sigmoid explanation alone does not cover that case.

Spam changes over time, class proportions vary and labels can contain mistakes. Measure the fitted pipeline and selected cutoff on held-out data that matches the intended use. Choose model settings and cutoff on validation data, not by repeatedly improving the final test score. Neither high accuracy nor a plausible-looking probability proves that a new inbox is well served.

Our invented spam example taught a two-stage process: logistic regression fits a probability model; a threshold determines the action taken from that estimate.

• Keep inputs and labels distinct: features supply evidence; known categories teach the classifier.
• Read the right column: probability entries follow the model’s class order.
• Understand the transformation: a weighted score passes through the sigmoid in the binary model.
• Separate the policy: a new cutoff changes decisions without changing the fitted probabilities.
• Check consequences: false positives wrongly flag legitimate messages; false negatives miss spam. Use validation for selection and held-out data for evaluation.
• Interpret estimates carefully: bounded probabilities, calibration and useful real-world performance are different claims.

Before moving on: explain why the same 0.64 estimate can produce two decisions and identify what happens to the six validation messages at cutoff 0.6. Next, we will classify an input by comparing it with nearby examples.