Spam detection

Your phone receives an ordinary school reminder and an unsolicited prize message. A filter should let the reminder through while identifying unwanted messages. Rather than write a rule for every phrase, we can learn patterns from messages that already have ordinary/spam labels.

This is text classification: turning message text into numerical features, then predicting a category. We use 0 for ham, the dataset’s term for ordinary messages, and 1 for spam. The model predicts a label from patterns in its training data; it does not understand a sender’s intentions.

The worked source is the UCI SMS Spam Collection. SMS messages differ from email, so this project does not establish real email-filter performance. Review training/test separation and classification errors if needed.

The downloaded file has no header. Each line contains a label, a tab separator and the message text. pd.read_csv reads it using sep="\t", header=None and explicit names ["label", "message"]. The text is the input; the supplied label is the answer used for training and evaluation.

The checked file parses into 5,572 rows. It contains repeated message texts, so we inspect those before splitting. A real import should also check missing text, unexpected labels and conflicting answers for the same message. The expression label == "spam" must not silently turn an unrecognized label into ham.

Credit: Almeida, T. and Hidalgo, J. (2011), SMS Spam Collection, UCI Machine Learning Repository, DOI 10.24432/C5CC84. UCI lists it under CC BY 4.0. This historical collection combines several sources; it is not a random sample of all messages received today.

Prepare and run

Download the UCI SMS Spam Collection archive, unzip it and place SMSSpamCollection in the script or notebook’s working directory. Use Python with pandas and scikit-learn in the course environment. Each program runs independently.

Python · complete runnable example

import pandas as pd

data = pd.read_csv("SMSSpamCollection", sep="\t", header=None,
                   names=["label", "message"])
print("Parsed rows:", len(data))
print("Missing labels:", int(data.label.isna().sum()))
print("Missing messages:", int(data.message.isna().sum()))
print("Label names:", sorted(data.label.unique().tolist()))
conflicts = data.groupby("message").label.nunique()
print("Message texts with conflicting labels:", int((conflicts > 1).sum()))
clean = data.drop_duplicates(subset=["message"]).copy()
print("Removed repeated message texts:", len(data) - len(clean))
print("Remaining rows:", len(clean))
print("Ham rows:", int((clean.label == "ham").sum()))
print("Spam rows:", int((clean.label == "spam").sum()))

Expected output

Parsed rows: 5572
Missing labels: 0
Missing messages: 0
Label names: ['ham', 'spam']
Message texts with conflicting labels: 0
Removed repeated message texts: 403
Remaining rows: 5169
Ham rows: 4516
Spam rows: 653

The checked labels are exactly “ham” and “spam”, with no missing labels or messages. There are no exact message texts with conflicting labels. Removing repeated texts keeps 5,169 rows: 4,516 ham and 653 spam. These are counts for the displayed import, not a promise that every future download or parser produces the same table.

drop_duplicates(subset=["message"]) retains the first occurrence of each exact text. Checking conflicts first avoids silently resolving contradictory labels by file order. If conflicts existed, investigate them instead of accepting whichever answer appears first.

Exact deduplication reduces one source of overlap across the train/test split. It does not merge punctuation variants, repeated campaigns with small edits or messages from related senders. Those relationships can still make an ordinary random split optimistic.

After exact deduplication, a stratified 80/20 split gives 4,135 training messages and 1,034 test messages. Training contains 3,613 ham and 522 spam; testing contains 903 ham and 131 spam. stratify=y approximately preserves the label proportions. random_state=7 fixes this shuffled split for repeatable comparison.

Fit both vocabulary and classifier on training messages only. Test messages are transformed with that existing vocabulary and classified with the fitted model. Learning a vocabulary from all messages first would allow held-out text to influence the feature representation, even without using its labels.

This dataset is not chronologically ordered. Holding out the last lines is not automatically a future-period test. Sender- or campaign-aware evaluation also requires suitable provenance that this two-column file does not provide.

A token is a text unit produced by a chosen splitting rule. With the default CountVectorizer, text is lowercased and word-like tokens of at least two characters are counted. Punctuation separates tokens rather than becoming its own feature. In “Win a free prize now”, the one-letter “a” is discarded and the tokens are win, free, prize and now.

The vectorizer learns a vocabulary from training text and assigns one column per known token. A message becomes a row of counts: a known word used twice contributes 2 in its column, and absent words contribute 0. Most columns are zero in a short message, so the library uses a sparse matrix that stores the nonzero entries efficiently.

This is a bag-of-words representation. It largely discards order and context: “win a prize” and “a prize win” give the same counts under these rules. Counting a word is not understanding its meaning. See the text feature-extraction documentation for tokenization and representation choices.

Multinomial Naive Bayes learns how token counts are associated with each class. It also learns a class prior: how common ham and spam are in training. For a new message, it combines that prior with evidence from its counted tokens to compare the classes.

The “naive” assumption treats token evidence as conditionally independent given the class. Real language does not satisfy that neatly: words depend on nearby words and context. The simplifying model can still be useful, but a good score does not mean it has captured the full structure of language.

Default smoothing adds a small amount to token counts when estimating class-related token probabilities, avoiding an automatic zero simply because a known token was not observed in one class. The model combines evidence in log space, turning probability products into stable sums. A larger resulting log score wins; negative scores are normal and are not negative probabilities. See the model’s assumptions and smoothing.

Prepare and run

Download the UCI SMS Spam Collection archive, unzip it and place SMSSpamCollection in the script or notebook’s working directory. Use Python with pandas and scikit-learn in the course environment. Each program runs independently.

Python · complete runnable example

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import precision_score, recall_score, f1_score

# Download and unzip UCI's SMS Spam Collection before running.
data = pd.read_csv("SMSSpamCollection", sep="\t", header=None, names=["label", "message"])
data = data.drop_duplicates(subset=["message"]).copy()
y = (data["label"] == "spam").astype(int)
train_X, test_X, train_y, test_y = train_test_split(data["message"], y, stratify=y, test_size=0.2, random_state=7)
model = make_pipeline(CountVectorizer(), MultinomialNB()).fit(train_X, train_y)
prediction = model.predict(test_X)
for name, metric in [("Precision", precision_score), ("Recall", recall_score), ("F1", f1_score)]:
    print(name + ":", round(metric(test_y, prediction, zero_division=0), 3))
print("Example class:", int(model.predict(["Win a free prize now"])[0]))

Expected output

Precision: 0.975
Recall: 0.885
F1: 0.928
Example class: 1

make_pipeline(CountVectorizer(), MultinomialNB()) fits the vocabulary first, transforms the training text, then fits the classifier on the resulting counts. Predicting test messages applies the same vocabulary and learned classifier without fitting them again.

The supplied example “Win a free prize now” receives class 1, spam. That is one decision by this model, not a general rule that every message containing “free” is spam. The next trace includes the full setup again and exposes the four counted tokens and both class scores.

Precision 0.975, recall 0.885 and F1 0.928 describe spam as the positive class on this split. We will derive their meaning from actual counts rather than treat them as interchangeable percentages. zero_division=0 specifies a reporting convention when a metric’s denominator is zero; no such zero denominator occurs for this fitted model’s reported spam metrics.

Use the same illustrative message, “Win a free prize now”. The trained vocabulary recognizes all four retained tokens. Each occurs once, so the classifier receives four nonzero counts among 7,687 vocabulary columns. Other vocabulary columns remain zero.

The next program reconstructs those scores from the fitted token log probabilities and class priors, then checks the prediction. It uses the full message and training code within this section, so you do not need variables left over from the previous example.

Prepare and run

Download the UCI SMS Spam Collection archive, unzip it and place SMSSpamCollection in the script or notebook’s working directory. Use Python with pandas and scikit-learn in the course environment. Each program runs independently.

Python · complete runnable example

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import precision_score, recall_score, f1_score

# Download and unzip UCI's SMS Spam Collection before running.
data = pd.read_csv("SMSSpamCollection", sep="\t", header=None, names=["label", "message"])
data = data.drop_duplicates(subset=["message"]).copy()
y = (data["label"] == "spam").astype(int)
train_X, test_X, train_y, test_y = train_test_split(data["message"], y, stratify=y, test_size=0.2, random_state=7)
model = make_pipeline(CountVectorizer(), MultinomialNB()).fit(train_X, train_y)
prediction = model.predict(test_X)

import numpy as np

vectorizer = model.steps[0][1]
classifier = model.steps[1][1]
message = "Win a free prize now"
row = vectorizer.transform([message])
print("Vocabulary size:", len(vectorizer.vocabulary_))
print("Tokens:", vectorizer.build_analyzer()(message))
print("Nonzero counts:")
words = vectorizer.get_feature_names_out()
for index in row.indices:
    print(words[index] + ":", int(row[0, index]))
# Log-space addition combines prior and token evidence stably.
scores = row @ classifier.feature_log_prob_.T + classifier.class_log_prior_
for label, score in zip(classifier.classes_, scores[0]):
    print("Class:", int(label), "log score:", round(float(score), 3))
manual = classifier.classes_[np.argmax(scores[0])]
print("Manual class:", int(manual))
print("Pipeline class:", int(model.predict([message])[0]))
unknown = "zzzxqvtoken"
print("Unknown-token nonzero counts:", vectorizer.transform([unknown]).nnz)
print("Unknown-token class:", int(model.predict([unknown])[0]))

Expected output

Vocabulary size: 7687
Tokens: ['win', 'free', 'prize', 'now']
Nonzero counts:
free: 1
now: 1
prize: 1
win: 1
Class: 0 log score: -32.334
Class: 1 log score: -23.658
Manual class: 1
Pipeline class: 1
Unknown-token nonzero counts: 0
Unknown-token class: 0

The vectorizer’s analyzer exposes its tokenization rules. The transformed row stores nonzero column indexes and counts; get_feature_names_out() maps those indexes back to tokens. The displayed token order follows matrix storage, not necessarily sentence order.

The expression row @ feature_log_prob_.T + class_log_prior_ sums count-weighted token evidence and adds the learned prior for each class. np.argmax chooses the larger score; classes_ maps its position to label 0 or 1. The reconstructed class agrees with the pipeline.

The invented word “zzzxqvtoken” is absent from this vocabulary, so its row contains no nonzero counts. The classifier still predicts ham, using only its learned priors. This is not evidence that the unknown message is safe. Unknown words do not update the vocabulary during prediction; unfamiliar language or spelling may weaken the representation.

Ham is the majority label in training. A majority-class baseline predicts ham for every test message, ignoring the text. It gets 903 of 1,034 test labels correct: about 87.3% accuracy. It also misses all 131 spam messages.

That baseline shows why accuracy alone is misleading when labels are imbalanced. A filter can appear accurate by mostly predicting the common class while failing to do the job we care about. Fit the baseline’s chosen label from training targets, then evaluate it on the same test rows as the model.

For this fixed model, 900 ham messages are correctly retained, three are wrongly flagged, 116 spam messages are caught and 15 are missed. These counts distinguish the inconvenience of false alarms from missed unwanted messages. Their practical costs depend on how the filter is used.

Prepare and run

Download the UCI SMS Spam Collection archive, unzip it and place SMSSpamCollection in the script or notebook’s working directory. Use Python with pandas and scikit-learn in the course environment. Each program runs independently.

Python · complete runnable example

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import precision_score, recall_score, f1_score

# Download and unzip UCI's SMS Spam Collection before running.
data = pd.read_csv("SMSSpamCollection", sep="\t", header=None, names=["label", "message"])
data = data.drop_duplicates(subset=["message"]).copy()
y = (data["label"] == "spam").astype(int)
train_X, test_X, train_y, test_y = train_test_split(data["message"], y, stratify=y, test_size=0.2, random_state=7)
model = make_pipeline(CountVectorizer(), MultinomialNB()).fit(train_X, train_y)
prediction = model.predict(test_X)

from sklearn.dummy import DummyClassifier
from sklearn.metrics import confusion_matrix, accuracy_score

# Numeric placeholder inputs: this baseline ignores message content.
baseline = DummyClassifier(strategy="most_frequent").fit(
    [[0]] * len(train_y), train_y)
baseline_prediction = baseline.predict([[0]] * len(test_y))
print("Training messages:", len(train_X))
print("Test messages:", len(test_X))
print("Actual test ham:", int((test_y == 0).sum()))
print("Actual test spam:", int((test_y == 1).sum()))
print("Rows: actual ham/spam; columns: predicted ham/spam")
print(confusion_matrix(test_y, prediction, labels=[0, 1]))
print("Model accuracy:", round(accuracy_score(test_y, prediction), 3))
print("Always-ham accuracy:", round(accuracy_score(test_y, baseline_prediction), 3))
print("Always-ham spam recall:", round(recall_score(test_y, baseline_prediction, zero_division=0), 3))

Expected output

Training messages: 4135
Test messages: 1034
Actual test ham: 903
Actual test spam: 131
Rows: actual ham/spam; columns: predicted ham/spam
[[900   3]
 [ 15 116]]
Model accuracy: 0.983
Always-ham accuracy: 0.873
Always-ham spam recall: 0.0

The matrix rows are actual ham/spam labels and its columns are predicted ham/spam labels. The first row, [900, 3], describes ordinary messages: 900 correct and three false alarms. The second, [15, 116], describes spam: 15 missed and 116 caught.

Spam precision is 116 / (116 + 3) ≈ 0.975: among messages flagged as spam, how many have a spam label? Spam recall is 116 / (116 + 15) ≈ 0.885: among actual spam messages, how many are caught? High precision can coexist with lower recall because the denominators answer different questions.

F1 combines precision and recall using their harmonic mean: 2 × precision × recall / (precision + recall), giving about 0.928. It is not an average of all correct predictions or a measure of spam intent. The filter’s accuracy is 0.983, but its separate false-alarm and missed-spam counts carry information that number conceals.

Your task: The filter flags 119 test messages, of which three are actually ham. The test set contains 131 spam messages, of which 15 are missed. Calculate spam precision and recall. Then name the error when a school reminder is flagged, and explain why automatically deleting flagged messages is a separate design decision.

Worked answer

Correctly flagged spam is 119 − 3 = 116. Precision is 116 / 119 ≈ 0.975; recall is (131 − 15) / 131 = 116 / 131 ≈ 0.885. The flagged school reminder is a false positive, while missed spam is a false negative.

A classifier label does not dictate deletion. Moving messages to a reviewable spam folder permits recovery from a false alarm; automatic deletion may have a different cost. Choose the action and any score threshold using suitable development evidence and the intended use, rather than repeatedly adjusting them to this test result.

The bag of words loses sentence order, negation and much context. A legitimate message discussing a scam may contain suspicious vocabulary. An altered spelling or new language may contain few known tokens. Counts alone also leave out email-specific information such as headers and attachment behavior.

Possible experiments include word pairs, character features, different weighting or another classifier. Compare complete pipelines through training validation; specify any decision-threshold policy there as well. Naive Bayes probability outputs should not automatically be read as calibrated chances of spam.

Exact deduplication does not resolve every related campaign, and this file lacks the sender/time information needed for some more realistic evaluation protocols. New tactics, formats and time periods can change performance. Representative, appropriately separated email data is needed before making an email-performance claim.

Record the source and attribution, imported file version, deduplication policy, label mapping, split, vocabulary rules, classifier settings and error counts. If test-error analysis changes the design, use a fresh evaluation protocol for the revision rather than calling repeated test tuning untouched evidence.

You have built a text-classification pipeline: inspect labels and duplicates, split the messages, learn a vocabulary from training text, count known tokens and fit Naive Bayes. New messages pass through the same fitted representation and classifier.

• Representation: the checked vocabulary has 7,687 tokens. Unknown tokens contribute no count, and word counts do not preserve full sentence meaning.
• Test errors: 900 correctly retained ham messages, three false alarms, 15 missed spam messages and 116 correctly flagged spam messages.
• Metrics: spam precision 0.975, recall 0.885 and F1 0.928 describe different aspects of this fixed test result.
• Scope: this is an attributed historical SMS benchmark. It does not establish future or real email-filter performance.

Before using a filter’s output, explain how the message became features, what a false alarm costs and what evidence supports the proposed action. A model decision and a deployment policy are related decisions that need their own justification.