Loading datasets: read and inspect data with pandas

A teacher gives you a file containing students’ study hours and exam scores. Before you can explore a relationship or train a model, Python needs a way to read those records. Loading a dataset means bringing its recorded values into a structure your program can work with. Here we will use pandas, a Python library whose DataFrame represents a table with named columns.

Reading the file successfully does not prove the data is suitable. A score might be recorded out of 50 rather than 100; a blank study-time cell might mean “not recorded,” rather than zero. We will load a small fictional mark sheet, inspect what arrived and investigate mistakes before changing any values.

If tables and row labels are unfamiliar, first review arrays, DataFrames and selecting cells. This lesson uses ordinary Python and pandas; loading itself does not train a machine learning model.

In this mark sheet, one row describes one student’s result for one exam. The student column names that record; hours is the reported study time before the exam; score is the exam result out of 100. A different file might contain one row per attendance day, so “one row per student” is not a universal rule.

If our later task is to estimate an exam score from study time, hours is an input feature and score is the target—the outcome to predict. The name is useful for checking records, but is not automatically a useful prediction input. This distinction connects to features and targets in the house-price example.

Ask the source what units and time period each column uses. “Hours” could mean hours per day or total hours before the exam. A column name alone cannot settle that. Also ask whether the study time would really be available when the prediction is requested.

Prepare and run

Run this complete Python program in the course environment with pandas installed. Its small school dataset is included; no download is required.

Python · complete runnable example

from io import StringIO
import pandas as pd

csv_text = "student,hours,score\nDiya,2,60\nSaira,3,70\n"
data = pd.read_csv(StringIO(csv_text))
print("Rows:", len(data))
print("Columns:", ", ".join(data.columns))
print("First score:", data.loc[0, "score"])
print("Shape:", data.shape)

Expected output

Rows: 2
Columns: student, hours, score
First score: 60
Shape: (2, 3)

Follow the load

The import statements make StringIO and pandas available; pd is simply the short name we choose for pandas. csv_text contains three lines. The escape sequence \n starts a new line inside the Python string. StringIO presents that string as a readable, file-like stream, and read_csv returns a DataFrame.

With these default settings, the first line supplies the column names. The next two lines become data rows. That is why len(data) is two rather than three. Joining the column names with a comma produces the displayed list. The shape tuple reports rows first and columns second: two records, three columns.

data.loc[0, "score"] selects row label 0 and the score column, giving Diya’s 60. Pandas supplies row labels 0 and 1 here; those labels are not additional student records. A differently indexed table can have different labels.

After loading, inspect both the values and the table’s structure. The small school file has the columns student, hours and score, with two data rows. The following Python inspection commands operate on a DataFrame named data. They are inspection snippets, not a standalone program; use them after a load.

print(data.head())
print(data.shape)
print(data.columns.tolist())
print(data.dtypes)
data.info()

head shows the first five rows by default, or fewer when the table is smaller. It is a preview, not proof that every later row is valid. Shape checks the number of rows and columns; columns shows the headings; dtypes shows how pandas represents each column. For example, an integer column can represent the score values 60 and 70, but an integer type cannot tell you whether those scores are out of 100.

info prints a summary including column names, non-missing counts and types. Call it directly rather than wrapping it in print: it prints its own summary and returns None. Text-type names can differ across pandas versions and settings, so inspect the meaning rather than expecting every installation to display the same type label. See the pandas info reference.

Prepare and run

Run this complete Python program in the course environment with pandas installed. Its small school dataset is included; no download is required.

Python · complete runnable example

from io import StringIO
import pandas as pd

csv_text = "student,hours,score\nDiya,2,60\nSaira,,70\nDiya,2,60\n"
data = pd.read_csv(StringIO(csv_text))
print("Rows:", len(data))
print("Missing hours:", data["hours"].isna().sum())
print("Duplicate rows after the first:", data.duplicated().sum())
print("Hours are numeric:", pd.api.types.is_numeric_dtype(data["hours"]))
print("Scores outside 0 to 100:", ((data["score"] < 0) | (data["score"] > 100)).sum())

Expected output

Rows: 3
Missing hours: 1
Duplicate rows after the first: 1
Hours are numeric: True
Scores outside 0 to 100: 0

This program reproduces its own CSV. Saira’s hours field is empty, and Diya’s complete record appears twice. The file still loads: successfully reading a file and checking its quality are separate jobs.

isna marks missing cells; summing that column counts one missing hours value. Duplicated marks exact repeats after their first occurrence, so the second Diya row contributes one. The original row is not counted as a repeat. A numeric type check confirms hours is represented numerically, including a missing entry; it does not make that missing entry equal to zero.

The last line counts scores below zero or above 100. Each comparison produces true/false values per row, and the vertical-bar operator combines the conditions. The parentheses keep each comparison together. This check finds no violations in these three scores, but it only checks the rule we wrote. We have not checked whether the scores belong to the right students.

Pause before deleting the repeat: is this an accidental copy or two genuinely distinct exam records with missing exam identifiers? Likewise, confirm why Saira’s time is absent before replacing it. The next lesson covers missing values and unusual measurements.

CSV means comma-separated values, but exports commonly use other separators. A school spreadsheet may export the line Diya;2;60 with semicolons. A reader expecting commas can interpret that entire line as one field. The file may load without an exception and still have the wrong shape.

Choose the separator from the source format, not by randomly trying settings until the output looks plausible. For a semicolon export, read_csv accepts sep=";". If the file has no header, supply header=None and documented column names; otherwise the first student record can be mistaken for headings.

Quoted cells can contain the separator. In CSV, "Diya, A",2,60 can contain a single student name with a comma. A proper CSV reader understands quoting; splitting every line yourself on commas would break that record.

Blank cells and some common text markers are interpreted as missing by default. Source-specific markers can be configured with na_values. Do not guess that a dash or zero means “missing,” and remember that a real text value such as an identifier may need different handling. The pandas read_csv reference describes separator, header, type and missing-value settings.

Prepare and run

Run this complete Python program in the course environment with pandas installed. Its small school dataset is included; no download is required.

Python · complete runnable example

from io import StringIO
import pandas as pd

csv_text = "student_id;hours;score\n001;2;60\n002;3;70\n"
wrong = pd.read_csv(StringIO(csv_text))
print("Default reader columns:", len(wrong.columns))

data = pd.read_csv(StringIO(csv_text), sep=";", dtype={"student_id": "string"})
print("Configured reader columns:", len(data.columns))
print("First student ID:", data.loc[0, "student_id"])
print("First score:", data.loc[0, "score"])

Expected output

Default reader columns: 1
Configured reader columns: 3
First student ID: 001
First score: 60

The CSV input is repeated in full. The default comma reader sees no commas in the header, so it creates one column. Supplying sep=";" produces the intended three columns. Checking the shape and headings reveals the problem more clearly than merely checking whether a load returned a result.

The dtype mapping tells pandas to treat student_id as text. The characters 001 identify a student; they are not a quantity to calculate with. Without that instruction, numeric inference can discard the leading zeros and produce 1. Reading the identifier correctly at the source is safer than trying to reconstruct lost formatting afterward.

A type setting is not a cleaning strategy for every column. If a numeric field contains “two hours,” first inspect the source and decide what it means. Silently forcing bad cells to missing values can hide an export error.

For your own school export, replace the StringIO input with a file path. The following Python snippet reads an existing file; it will only run once you have created or received that file.

from pathlib import Path
import pandas as pd

print("Working folder:", Path.cwd())
data = pd.read_csv("students.csv")
print(data.shape)

A relative path such as students.csv is resolved from the process’s current working folder, which is not always the folder containing your script. Notebook tools also choose a working folder. If you get FileNotFoundError, inspect the printed folder, the actual filename and the extension before editing the dataset.

You can supply a relative subfolder path, such as "data/students.csv", or the actual full path to your file. The output depends on that file, so this snippet has no universal expected shape. Keep the original raw file unchanged and save cleaned versions separately.

If names show garbled characters or the reader raises a decoding error, check the file’s documented character encoding. The encoding setting controls how bytes become text; changing it does not resolve a wrong separator or incorrect units. Do not suppress errors simply to get an output.

Pandas can read several formats. These Python snippets are alternatives for existing sources, not extra lines to run against our embedded CSV. An Excel workbook can contain several sheets; name the intended one explicitly and install the required engine, such as openpyxl for common .xlsx workbooks.

data = pd.read_excel("students.xlsx", sheet_name="Class A")
# Requires the workbook and a suitable Excel engine, such as openpyxl.

JSON stores structured objects rather than comma-delimited records. A flat list of objects can form a table; nested objects may need deliberate normalization. Do not rename a JSON or Excel file to .csv and expect read_csv to understand it.

data = pd.read_json("students.json")
# Requires a JSON structure suitable for this reader.

read_csv can also accept a supported direct data URL. A webpage displaying a table or asking you to sign in is not necessarily a downloadable CSV. Confirm the source and access requirements, and keep an appropriate fixed version for a repeatable experiment when the source terms allow it.

Scikit-learn has small packaged teaching datasets, such as Iris through load_iris. Fetch functions may download and cache larger datasets. These are convenient sources, but you must still read the dataset description, feature units and limitations. We use documented sources in the later Titanic project.

A school says its export contains two exam records and three columns. It sends the following source text:

student_id;hours;score
001;2;60
002;;70

You read it with the default comma reader and see one column. Someone proposes splitting the result manually, converting student IDs to numbers and replacing the blank hours cell with zero. What should you do instead? Predict the row count, column count and missing-hours count after a correctly configured load.

Worked answer

Use a semicolon separator and a text type for student_id, then inspect the parsed table. The header supplies three column names, and the two following lines supply two data rows. There should be one missing hours value. Identifier 001 should remain text, and the blank entry should remain missing until its meaning is investigated.

This complete Python answer repeats the source, so it can be run independently.

from io import StringIO
import pandas as pd

csv_text = "student_id;hours;score\n001;2;60\n002;;70\n"
data = pd.read_csv(StringIO(csv_text), sep=";", dtype={"student_id": "string"})
print("Shape:", data.shape)
print("Missing hours:", data["hours"].isna().sum())
print("First ID:", data.loc[0, "student_id"])

Expected output

Shape: (2, 3)
Missing hours: 1
First ID: 001

Manual splitting risks breaking quoted fields. Numeric identifiers can lose leading zeros. Replacing a blank with zero would claim that the second student studied for zero hours, which the source does not establish.

Write down the source, retrieval date or version, licence or permitted use, selected sheet and reader settings. Also record the expected columns, row definition and units. This is a small data contract: an explicit description of what you expect, so a changed export can be noticed.

Large files may exceed available memory. read_csv can select needed columns with usecols, read an initial number of rows with nrows or return chunks for processing. A small preview can help diagnose the format, but it does not inspect the whole dataset. Chunking also requires a plan for calculations spanning chunks; it is not automatic model training.

Look beyond exact duplicate rows: repeated observations of one student can matter when you later create a training and test split. A successful load does not establish that students are representative, that outcomes are accurate or that future predictions will work. Those questions remain part of preparation and evaluation.

Use the reader that matches the source format. With our school CSV, the header becomes column names and each following record becomes a row. Pandas returns a DataFrame; it does not train a model or certify the records.

• Confirm meaning: define one record, each column, its units and when the value was available.
• Inspect the result: check sample rows, headings, shape, types, missing values, repeated records and relevant ranges.
• Investigate surprises: a one-column table can signal a wrong separator; identifiers may require text types; blanks do not automatically mean zero.
• Keep the load repeatable: preserve the raw source and record its version, path, reader settings and expected structure.

Before the next lesson: explain why the blank study-time cell should remain missing during inspection. Next, we will decide how to handle missing values and unusual measurements with reasons for each choice.