Handling missing values and outliers: learn with Titanic data

You have loaded Titanic passenger records into a table. One passenger’s age is absent, while another has a much larger recorded fare than most passengers in a small excerpt. Both deserve attention, but they are different problems.

A missing value means a value is not available in the field as currently interpreted. An outlier is an observed value or record that is unusual relative to a chosen comparison. Missing does not mean zero, and unusual does not mean wrong. Handling either requires understanding the source and the task before changing the data.

We will use an attributed excerpt of Seaborn’s Titanic teaching dataset. Each row represents a passenger. Age is recorded in years, fare is a recorded ticket-fare field, and pclass identifies ticket class. Our excerpts teach preparation; they are too small to establish a reliable survival model. First review loading and inspecting datasets if reading a CSV or selecting a column is unfamiliar.

In a CSV, an empty age cell can be read as missing. Pandas can represent missingness using NaN, pd.NA or other markers depending on the column type. In our numerical examples, np.nan is NumPy’s marker for an absent numerical value. It is not a normal age and does not mean zero.

Use data["age"].isna() to ask which ages are absent, rather than comparing values with NaN. Comparing an entry to np.nan with equality is not a dependable missingness test. In pandas, isna handles the supported missing markers; isna().sum() counts them.

Source conventions still matter. A dataset might use -999 or the text “unknown” as a placeholder. Unless configured or converted, those can remain ordinary numbers or text. Conversely, zero may be a genuine recorded value. Keep the source description alongside your reader settings; the pandas missing-data guide explains the supported markers and checks.

Prepare and run

Use Python in the course environment with pandas, NumPy and scikit-learn. The Titanic values needed for this example are embedded in the program; no download is required.

Python · complete runnable example

from io import StringIO
import pandas as pd

csv_text = """source_row,age,fare,pclass
0,22,7.25,3
1,38,71.2833,1
2,26,7.925,3
3,35,53.1,1
4,35,8.05,3
5,,8.4583,3
258,35,512.3292,1
"""
data = pd.read_csv(StringIO(csv_text))
print("Excerpt rows:", len(data))
print("Missing ages:", data["age"].isna().sum())
print("Missing fares:", data["fare"].isna().sum())
print("Rows with missing age:", data.loc[data["age"].isna(), "source_row"].tolist())
print("Recorded high fare:", data.loc[data["source_row"] == 258, "fare"].iloc[0])

Expected output

Excerpt rows: 7
Missing ages: 1
Missing fares: 0
Rows with missing age: [5]
Recorded high fare: 512.3292

Read the evidence

The program embeds age, fare and ticket-class fields from source rows 0–5 and 258. Source_row is a label we added to identify positions in the CSV; it is not an original passenger identifier. This deliberately selected excerpt is not a random sample, and its counts do not describe the full dataset.

The blank age in row 5 becomes missing. The Boolean mask from isna selects that row, and summing the mask counts one absent age. Every fare in this excerpt is present, including 512.3292. A missing-value check therefore does not flag the high fare.

The loc expression finds source row 258 by its added label; iloc[0] then selects the first matching fare. The file can have no missing fares and still contain values worth investigating. These two checks answer different questions.

The triple quotes let the CSV string span visible lines. Its first line supplies column names; the empty age field on the line starting with 5 is the absent value.

A blank age tells us that the field is unavailable in this record. It does not tell us why. The measurement might never have been collected, might have been lost in an export, or might be absent more often for particular passenger groups. Do not invent a historical explanation from the blank alone.

The reason affects what a preparation policy can achieve. If we remove every passenger without a recorded age, we may change which kinds of passengers remain. If absence is related to a group or outcome, the remaining rows can give a misleading picture. Compare missingness across relevant fields while respecting what the source actually establishes.

Missing values can occur in inputs or in the target we want to predict. For a supervised survival task, a passenger without a known survival outcome cannot simply receive a guessed target and be treated as labelled evidence. Missing input preparation and obtaining reliable labels are separate decisions.

Imputation means supplying a replacement for an absent value. A simple age policy might use the training median: the middle observed age when ages are sorted. The mean is the arithmetic average; it is more affected by very large or small values. Neither statistic recovers a particular passenger’s real age.

• Keep or recover the source value: if a documented export problem can be corrected, repair it from evidence and record the correction.
• Remove a row or column: this can simplify preparation but discard useful evidence and change the population represented. A heavily missing column is not automatically useless.
• Supply a replacement: mean, median or a documented constant can keep rows available, while changing the feature’s distribution.
• Preserve absence: an extra indicator can record whether the original value was missing. Some estimators also handle missing values directly; check the chosen estimator.

For text categories, a most-frequent category or an explicit “unknown” category may be appropriate. A numerical median cannot meaningfully replace a port name. We will handle categories in the next lesson on categorical encoding. More complex imputation methods can use other columns, but require additional assumptions and evaluation.

When evaluating a prediction model, reserve some records for checking how it performs on data it did not learn from. Those are test records; the records used for learning are training records. We will build a proper split in the training and test split lesson.

A median computed from all records lets reserved test data influence preparation. This is a form of data leakage: using information outside the intended learning portion to shape a fitted step. Even without seeing the survival labels, the procedure no longer matches learning solely from the training data.

Fit the imputer on training inputs, then transform training and later inputs with that same fitted rule. In scikit-learn, fit learns the replacement and transform applies it. For our trace, four selected passenger ages are the demonstration training subset; two later rows are transformed. This ordered, tiny subset is for understanding the mechanism, not a suitable experimental split. See scikit-learn’s data-leakage guidance.

Prepare and run

Use Python in the course environment with pandas, NumPy and scikit-learn. The Titanic values needed for this example are embedded in the program; no download is required.

Python · complete runnable example

import numpy as np
from sklearn.impute import SimpleImputer

# Ages from source rows 0 to 3: a tiny demonstration training subset.
train_age = np.array([[22.0], [38.0], [26.0], [35.0]])
# Ages from source rows 4 and 5: apply the already-fitted rule.
new_age = np.array([[35.0], [np.nan]])

imputer = SimpleImputer(strategy="median")
imputer.fit(train_age)
print("Training median:", imputer.statistics_[0])
print("Prepared training ages:", imputer.transform(train_age).ravel().tolist())
print("Prepared new ages:", imputer.transform(new_age).ravel().tolist())
print("New age was missing:", np.isnan(new_age).ravel().astype(int).tolist())

Expected output

Training median: 30.5
Prepared training ages: [22.0, 38.0, 26.0, 35.0]
Prepared new ages: [35.0, 30.5]
New age was missing: [0, 1]

Trace each change

The training array has four rows and one age column; the nested brackets retain the two-dimensional shape that SimpleImputer expects. Sort the observed ages to get 22, 26, 35 and 38. With four values, the median averages the two middle ages: (26 + 35) ÷ 2 = 30.5.

Fit stores 30.5 in statistics_. Transform leaves the observed training ages unchanged. In new_age, 35 is already present, so it remains 35; the missing entry becomes 30.5. Transform does not calculate a new median from these later records.

The last line calculates an indicator from the original new_age: zero means observed and one means missing. It remains separate from the prepared age values in this example. It can be supplied as an additional model input when appropriate, but it is not a prediction or a survival label. The SimpleImputer API also offers an indicator option; its generated columns depend on missingness seen during fitting.

For display, ravel flattens the one-column result into a sequence and tolist converts it to a Python list. Astype(int) converts the original true/false missingness flags into 1 and 0; it does not estimate any age.

In our small training subset, the sorted recorded ages are 22, 26, 35 and 38, producing median 30.5. Replacing a missing age with 30.5 says “use this chosen preparation value.” It does not say that a historical record established the passenger was 30.5 years old.

Filling many missing ages with the same value creates a repeated pile at the median and can reduce variation. Relationships involving age can also change. Keep the original column or raw file, and record how the prepared version was produced. A missingness indicator preserves the fact of absence, but cannot restore the unknown measurement.

If every training age is missing, there is no observed median to learn. Choose an explicit, reviewed alternative rather than pretending a median exists. Similarly, do not calculate a group replacement from a survival outcome you would not know when making a prediction. A useful preparation procedure must be available under the actual prediction conditions.

An outlier depends on the reference group and the rule used. A high fare relative to all passengers may be less surprising among first-class tickets. Before calling a value an error, check the units, ticket context and how the fare field was recorded. An impossible negative age and an unusually large positive fare are different kinds of evidence.

A common screening rule uses the interquartile range (IQR). Q1 is the 25th percentile and Q3 is the 75th percentile; they mark the middle half of an ordered distribution. IQR = Q3 − Q1. Values below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR receive a flag.

A flag is a request for review, not an automatic delete command. Quartile calculations can use different interpolation conventions, especially for tiny samples; the next program uses pandas’ default linear interpolation. The NIST box-plot explanation describes quartiles and screening fences.

Prepare and run

Use Python in the course environment with pandas, NumPy and scikit-learn. The Titanic values needed for this example are embedded in the program; no download is required.

Python · complete runnable example

import pandas as pd

# Seven recorded fares: source rows 0 to 5, then row 258.
fares = pd.Series([7.25, 71.2833, 7.925, 53.1, 8.05, 8.4583, 512.3292])
q1 = fares.quantile(0.25)
q3 = fares.quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
flagged = (fares < lower) | (fares > upper)
print("Q1:", round(q1, 4))
print("Q3:", round(q3, 4))
print("Upper screening boundary:", round(upper, 4))
print("Flagged fares:", fares[flagged].tolist())
print("Rows still present:", len(fares))

Expected output

Q1: 7.9875
Q3: 62.1916
Upper screening boundary: 143.4979
Flagged fares: [512.3292]
Rows still present: 7

This program repeats all seven fares, so it can run independently of the inspection example. The source-row labels are 0–5 and 258 in that order. The fares are not seven evenly spaced or randomly sampled tickets; the high-fare record was selected deliberately to make the screening mechanism visible.

Sorted fares are 7.25, 7.925, 8.05, 8.4583, 53.1, 71.2833 and 512.3292. Pandas interpolates Q1 halfway between 7.925 and 8.05, giving 7.9875, and Q3 halfway between 53.1 and 71.2833, giving 62.19165 before rounding. IQR is 54.20415, so the upper fence is 62.19165 + 1.5 × 54.20415 = 143.497875, displayed as 143.4979.

The Boolean mask selects 512.3292 for inspection; the original Series still has seven rows. With this excerpt, the lower fence is negative and flags none of the positive fares. Using the full dataset or a different reference group changes the quartiles and boundaries. We have demonstrated a screening calculation, not shown that a particular fare is invalid.

If investigation confirms a transcription error and the correct value is available, repair it and record the evidence. If a measurement is impossible but the correct value is unavailable, mark it as missing or exclude it under a documented policy. If the value is a genuine rare case relevant to the task, removing it can erase exactly the evidence you need.

Other choices include keeping the value, using a method less sensitive to extremes, transforming a skewed feature, or capping values at justified bounds. A cap changes a recorded value; a transformation changes its representation. Neither establishes that the original value was wrong. Compare the effect on a validation set rather than assuming “cleaner-looking” data leads to better predictions.

A z-score is another screen: it measures how far a value is from the mean in standard-deviation units. Extreme values affect those statistics, and a skewed fare distribution makes a universal cutoff difficult to justify. Use distribution and domain context rather than treating every value beyond three standard deviations as erroneous.

When an outlier policy learns quantiles, caps or other statistics for a model, learn those settings from training data and reuse them later. Do not remove difficult test cases merely to make reported accuracy look better. A real deployment can still encounter rare passengers or inputs.

Use this complete Python program. The four training ages come from the Titanic excerpt. The incoming values are deliberately invented for practice: one missing age and one observed age of 80. They are not additional historical passenger records.

import numpy as np
from sklearn.impute import SimpleImputer

train_age = np.array([[22.0], [38.0], [26.0], [35.0]])
incoming = np.array([[np.nan], [80.0]])
imputer = SimpleImputer(strategy="median").fit(train_age)
print("Prepared ages:", imputer.transform(incoming).ravel().tolist())

Your tasks: predict the output; explain whether the observed age of 80 changes the replacement; and decide whether an IQR flag on a recorded fare of 512.3292 is enough to delete the passenger.

Worked answer

Prepared ages: [30.5, 80.0]

The training median is (26 + 35) ÷ 2 = 30.5. The imputer replaces only the missing input; 80 remains 80 and does not refit the rule. SimpleImputer is not an outlier detector. If you incorrectly included the 80 in fitting, the five observed ages would have median 35; the incoming record would have influenced preparation.

The high fare requires investigation. Confirm the source, ticket context and comparison group before choosing a policy. A statistical flag alone does not justify deletion. Keep a record of the chosen action and assess its consequences for the task.

A pipeline connects preparation and prediction steps so they are fitted in sequence. With an imputer followed by a classifier, fitting the pipeline on training inputs learns the replacement and then trains the classifier on prepared inputs. Predicting later applies the fitted replacement before the classifier.

During cross-validation, a pipeline helps ensure each fold’s replacement is learned from that fold’s training records. Computing the median once from the whole dataset before cross-validation leaks held-out feature information. See cross-validation without data leakage and the Pipeline API for the combined mechanism.

This lesson prepares values rather than training a survival classifier. In the later Titanic prediction project, the imputer is fitted inside the model pipeline after the split. Preparation policies still need evaluation: a median, an indicator or an outlier rule is a choice to test, not a guarantee of accurate or fair predictions.

In our Titanic excerpt, an absent age and a high recorded fare need different decisions. We inspected both before changing either, and kept the original evidence available.

• Detect absence: use isna with the source’s missing-value conventions. A missing age does not mean age zero.
• Choose a policy: removal, replacement and retaining missingness have different effects. A replacement is a preparation value, not a recovered fact.
• Fit on training inputs: learn the median or other preparation settings there, then reuse them on later records.
• Review unusual observations: an IQR or other flag asks for investigation. It does not establish that a fare is wrong.
• Document and evaluate: preserve the raw source, explain changes and test the policy within the model’s validation procedure.

Before the next lesson: explain why a median can replace an absent numerical age but cannot meaningfully replace a port name. Next, we will prepare categorical inputs without inventing an order between their names.