Data Science with Python / Section 2 — Pandas: Data Analysis Powerhouse / Lesson 9
🐼 Section 2 · Pandas 🟢 Beginner MODULE 09

Reading, Cleaning & Exploring Data

⏱️ 27 min read
📖 Theory + Code
🧩 5 Quiz Questions
🏗️ 1 Challenge
Your progress in Section 233%
🎯 What you'll learn: How to load a real CSV file with pd.read_csv() and its most common parameters, how to find and handle missing data with .isnull(), .dropna(), and .fillna(), how to detect and remove duplicate rows with .duplicated() and .drop_duplicates(), how to fix a column's data type with .astype(), and how to rename columns.
Section 1

Reading a CSV File

Almost every real dataset starts life as a CSV file. pd.read_csv() loads it straight into a DataFrame in one line — but real-world files are messy, so it accepts a long list of parameters to handle the edge cases.

reading_csv.py
PYTHON 
import pandas as pd

# The basic case — first row becomes the header by default
df = pd.read_csv("sales.csv")

# Common parameters you'll reach for constantly:
df = pd.read_csv(
    "sales.csv",
    sep=",",                  # the delimiter — use "\t" for tab-separated files
    header=0,                 # row number to use as column names (0 = first row)
    usecols=["date", "sales"],  # load only these columns — saves memory on wide files
    dtype={"sales": "float64"}, # force a specific dtype for a column while reading
    na_values=["N/A", "missing", "--"],  # extra strings to treat as missing
    parse_dates=["date"],     # parse this column as datetime instead of plain text
    nrows=1000,                # load only the first 1000 rows — handy for a quick preview
)
📝
Related readers exist too
Pandas has matching functions for other formats: pd.read_excel(), pd.read_json(), pd.read_sql(). They all follow the same idea — a file (or connection) in, a DataFrame out — with format-specific parameters.
Section 2

Handling Missing Data

Real datasets almost always have gaps — a sensor that didn't report, a survey question left blank. Pandas represents missing values as NaN ("Not a Number"), and gives you tools to find, drop, or fill them.

missing_data.py
PYTHON 
# --- Finding missing values ---
print(df.isnull())          # same-shape DataFrame of True/False
print(df.isnull().sum())   # count of missing values PER COLUMN — the most useful one-liner
print(df.isnull().sum().sum())  # total missing values in the whole DataFrame

# --- Dropping missing values ---
df_clean = df.dropna()                    # drop any ROW that has at least one NaN
df_clean = df.dropna(subset=["sales"])   # only drop rows where "sales" specifically is NaN
df_clean = df.dropna(axis=1)              # drop COLUMNS that contain any NaN instead of rows

# --- Filling missing values ---
df["sales"] = df["sales"].fillna(0)               # replace NaN with a fixed value
df["sales"] = df["sales"].fillna(df["sales"].mean())  # replace NaN with the column's mean
df["region"] = df["region"].fillna("Unknown")        # works for text columns too
df.isnull().sum() — one number per column
dateproductregionsales
count of NaN0103
⚠️
dropna() and fillna() don't change the original by default
Like most Pandas methods, .dropna() and .fillna() return a new DataFrame instead of modifying df in place — you have to assign the result back, as shown above, or pass inplace=True. Forgetting to reassign is one of the most common early Pandas mistakes.
Section 3

Detecting and Removing Duplicates

Duplicate rows sneak in from merged exports, repeated form submissions, or double-counted imports. Pandas makes them easy to find and drop.

duplicates.py
PYTHON 
# --- Detecting duplicates ---
print(df.duplicated())              # True for every row that's an exact repeat of an earlier one
print(df.duplicated().sum())       # how many duplicate rows exist
print(df.duplicated(subset=["date", "product"]))  # duplicate based on specific columns only

# --- Removing duplicates ---
df_clean = df.drop_duplicates()                       # keeps the FIRST occurrence of each duplicate by default
df_clean = df.drop_duplicates(keep="last")         # keep the LAST occurrence instead
df_clean = df.drop_duplicates(subset=["date", "product"])  # dedupe by a subset of columns
✨
Always check duplicate counts before and after
Run df.duplicated().sum() before and after cleaning, and compare len(df) to len(df_clean). It's an easy sanity check that catches you accidentally dropping far more rows than you intended.
Section 4

Fixing Data Types & Renaming Columns

CSV files carry no type information, so Pandas has to guess — and it sometimes guesses wrong (a "sales" column full of numbers stored as text, for example). Fixing dtypes early prevents confusing bugs later.

dtypes_and_renaming.py
PYTHON 
# --- Checking and fixing dtypes ---
print(df.dtypes)                        # dtype of every column
df["sales"] = df["sales"].astype("float64")   # force numeric
df["quantity"] = df["quantity"].astype("int32")   # force integer
df["date"] = pd.to_datetime(df["date"])       # dedicated helper for dates — safer than astype

# --- Renaming columns ---
df = df.rename(columns={"nm": "name", "sc": "score"})  # rename a specific subset
df.columns = ["date", "product", "region", "sales", "quantity"]  # replace ALL column names at once
📝
astype() raises an error on bad data
If a column has stray text mixed into a numeric column (like "unknown" in a price column), .astype("float64") raises a ValueError. Clean or fill the bad values first — that's exactly the order the workflow in this lesson follows: handle missing values and duplicates before locking in dtypes.
Section 5

A Typical Cleaning Workflow

There's no single "correct" order, but this sequence works well for most tabular datasets and avoids common pitfalls.

1
Load and take a first look
pd.read_csv(), then .head(), .info(), and .shape to understand what you're working with.
2
Check for missing data
.isnull().sum() to see where the gaps are, then decide row-by-row whether to dropna() or fillna() per column.
3
Check for duplicates
.duplicated().sum(), then .drop_duplicates() if the count is non-zero and unexpected.
4
Fix dtypes and rename columns
.astype()/pd.to_datetime() for types, .rename() for clearer column names — now that the data is clean enough to convert safely.
Section 6

Lesson Summary

Let's recap everything you learned in this lesson:

✅pd.read_csv() loads tabular data, with parameters like sep, usecols, dtype, na_values, and parse_dates.
✅.isnull().sum() counts missing values per column; .dropna()/.fillna() handle them.
✅.duplicated() flags repeat rows; .drop_duplicates() removes them.
✅.astype() converts a column's dtype; pd.to_datetime() is the safer choice for dates.
✅.rename(columns={...}) renames specific columns; df.columns = [...] replaces all of them at once.
🧩 Knowledge Check — Lesson 9
Answer all 5 questions to test your understanding. Instant feedback on every answer.
1. Which method counts how many missing values are in each column?
2. What happens if you call df.dropna() without assigning the result?
3. Which method removes duplicate rows from a DataFrame?
4. What's the safer way to convert a column of date strings to actual dates?
5. What should generally happen BEFORE converting a numeric column's dtype with .astype()?
💪
Coding Challenge — Lesson 9
Apply what you learned · Beginner Level

Practice a full mini cleaning pass on a small, messy DataFrame.

Challenge: Clean the Survey Data 🧹

Build this DataFrame by hand: pd.DataFrame({"name": ["Ali","Sara","Ali","Omar", None], "age": [25, None, 25, 31, 40], "city": ["Lahore","Karachi","Lahore","Multan","Lahore"]}). Then: (1) print how many missing values exist per column, (2) fill missing age values with the column's mean, (3) fill the missing name with "Unknown", (4) drop any fully duplicated rows, and (5) print the final .shape.

Rules: Use .isnull().sum(), .fillna(), and .drop_duplicates() — don't loop through rows manually.
💡 Show hints if you're stuck
  • Missing counts: df.isnull().sum()
  • Fill numeric with mean: df["age"] = df["age"].fillna(df["age"].mean())
  • Drop duplicates last, after filling, so the "Ali" rows can be recognized as true duplicates: df = df.drop_duplicates()
Finished this lesson?
Mark it complete to track your progress.
🎉

Lesson 9 Complete!

You can load real CSV files and clean them — missing data, duplicates, dtypes, and column names. Next up: filtering, sorting, and selecting exactly the rows you need.

Module 09 of 13 Section 2 — Pandas: Data Analysis Powerhouse