Project 1 — Exploratory Data Analysis on a Real Dataset
.shape/.dtypes/.isnull().sum(), clean up the messy parts, then use groupby() and sort_values() to answer real business questions: which region sells the most, and which products are the top performers. We'll close by writing up a few findings — the actual point of doing EDA in the first place.
The Project Brief
Imagine you've been handed a CSV export from a small retail chain's point-of-sale system — one row per line item sold, across a handful of regions, over about a year. Nobody has cleaned it for you. Your job, like on any real data project, is to turn it into something trustworthy enough to draw conclusions from.
We'll use a plausible, illustrative dataset called retail_sales.csv with five columns. Everything you see in this lesson — the exact numbers, the missing values, the findings — is a worked example built to teach the workflow, not a real published dataset. The methods are 100% real pandas; the numbers are a stand-in for whatever CSV lands on your desk.
.isnull().sum() to see exactly which columns and how many rows are affected.sort_values() and nlargest() to rank regions and products.Load the Data and Take a First Look
Every EDA starts the same way: read the file, then immediately check its size and preview a few rows before doing anything else. Never trust a dataset you haven't looked at yet.
import pandas as pd df = pd.read_csv("retail_sales.csv") print(df.shape) # (1450, 5) print(df.head())
| order_date | product | region | sales | quantity | |
|---|---|---|---|---|---|
| 0 | 2024-01-03 | Laptop | West | 1200.0 | 1.0 |
| 1 | 2024-01-03 | Mouse | West | 45.0 | 3.0 |
| 2 | 2024-01-04 | Monitor | NaN | 910.0 | 2.0 |
| 3 | 2024-01-05 | Keyboard | South | 65.0 | NaN |
| 4 | 2024-01-06 | Laptop | East | 1150.0 | 1.0 |
Already there are two things worth noticing: row 2 is missing its region, and row 3 is missing its quantity. Next, check the dtypes pandas inferred for each column — this is where you often catch a second problem before you even start cleaning.
print(df.dtypes) # order_date object # product object # region object # sales float64 # quantity float64 # dtype: object print(df.info()) # RangeIndex: 1450 entries, 0 to 1449 # Data columns (total 5 columns): # order_date 1450 non-null object # product 1450 non-null object # region 1432 non-null object # sales 1450 non-null float64 # quantity 1424 non-null float64
object (plain text), not a real date — pandas can't do date math or sort chronologically on it until we convert it. quantity is float64 instead of a whole-number int64 — that's not a mistake in the file, it's a side effect of missing values, which you'll see explained in the next section.Find the Missing Data
.info() already hinted at missing values through the non-null counts, but .isnull().sum() gives you the exact picture, column by column, in a format you can act on directly.
print(df.isnull().sum()) # order_date 0 # product 0 # region 18 # sales 0 # quantity 26 # dtype: int64 # As a share of all rows, so you know if it's worth worrying about print((df.isnull().sum() / len(df) * 100).round(2)) # region 1.24 # quantity 1.79
NaN, pandas silently upgrades the whole column to float64 so the NaNs fit. You'll get int64 back once every missing value is filled in and you explicitly convert with .astype(int).Under 2% missing in either column is small enough to handle with simple fills rather than dropping rows outright — dropping would throw away real revenue data in the sales column just because a neighboring cell was blank.
Clean the Data
Two different missing-value problems get two different fixes. A missing region is categorical — there's no sensible number to compute, so it gets an explicit placeholder label. A missing quantity is numeric, so a sensible statistic (the median) fills the gap without distorting the total too much.
# Missing region: label it rather than guess or drop the row df["region"] = df["region"].fillna("Unknown") # Missing quantity: fill with the column's median, a robust "typical" value df["quantity"] = df["quantity"].fillna(df["quantity"].median()) # Now that there are no NaNs left in quantity, it can safely become int df["quantity"] = df["quantity"].astype(int) # Fix the dtype problem: parse order_date into real datetime values df["order_date"] = pd.to_datetime(df["order_date"])
.astype(int) while NaNs are still present raises an error — NaN is a float concept and simply has no integer equivalent. Always fill or drop the missing values first, then convert the dtype.Verify the cleanup actually worked before moving on — this habit catches typos in column names and forgotten steps early.
print(df.isnull().sum().sum()) # 0 ← zero missing values left, anywhere in the DataFrame print(df.dtypes) # order_date datetime64[ns] # product object # region object # sales float64 # quantity int64 # dtype: object
Summarize with groupby()
With clean data in hand, it's time to actually answer questions. First: how does total revenue break down by region, and how many orders and what average order value sit behind that total? Named aggregation with .agg() computes all three at once, one row per region.
region_summary = (
df.groupby("region")
.agg(
total_sales=("sales", "sum"),
orders=("sales", "count"),
avg_order_value=("sales", "mean"),
)
.round(2)
.sort_values("total_sales", ascending=False)
)
print(region_summary)
| region | total_sales | orders | avg_order_value |
|---|---|---|---|
| West | 198450.0 | 402 | 493.66 |
| East | 176220.0 | 388 | 454.18 |
| South | 151300.0 | 361 | 419.11 |
| North | 139560.0 | 281 | 496.66 |
| Unknown | 9840.0 | 18 | 546.67 |
Next, the same idea applied to product instead of region — this time to see which products carry the highest average order value, not just the highest total.
product_aov = (
df.groupby("product")["sales"]
.mean()
.round(2)
.sort_values(ascending=False)
.rename("avg_order_value")
)
print(product_aov)
| product | avg_order_value |
|---|---|
| Laptop | 1178.35 |
| Monitor | 897.50 |
| Keyboard | 68.20 |
| Mouse | 44.90 |
avg_order_value than West despite a much lower total_sales — West simply has far more orders. Reporting only the total, or only the average, would each hide half the picture. That's exactly why region_summary computes both side by side.Sort to Find Top Performers
The region_summary table above is already sorted, but sometimes you want the single biggest rows out of the raw, un-grouped data — for example, the five highest-value individual orders. .nlargest() is a shortcut for exactly that, faster and shorter to write than a full sort_values().head().
# Top 5 individual orders by revenue top_orders = df.nlargest(5, "sales")[["order_date", "product", "region", "sales"]] print(top_orders) # Equivalent, longer way of writing the same thing: # df.sort_values("sales", ascending=False).head(5) # High-value orders worth a closer look — anything over $1,000 big_orders = df[df["sales"] > 1000] print(f"{len(big_orders)} orders were over $1,000")
| order_date | product | region | sales | |
|---|---|---|---|---|
| 812 | 2024-08-14 | Laptop | West | 1420.0 |
| 204 | 2024-03-02 | Laptop | East | 1390.0 |
| 1103 | 2024-11-21 | Laptop | North | 1360.0 |
| 67 | 2024-01-19 | Monitor | West | 1290.0 |
| 955 | 2024-10-03 | Laptop | South | 1275.0 |
df.nlargest(5, "sales") and df.sort_values("sales", ascending=False).head(5) return the same rows. nlargest() is worth reaching for as the shorter, slightly more efficient option when you only need the top N and don't need the whole DataFrame sorted.The Complete Script, Start to Finish
Here's every step from this lesson combined into one script you could actually run against a CSV shaped like retail_sales.csv.
import pandas as pd # 1. Load and take a first look df = pd.read_csv("retail_sales.csv") print("Shape:", df.shape) print(df.head()) print(df.dtypes) # 2. Find the missing data print(df.isnull().sum()) # 3. Clean it up df["region"] = df["region"].fillna("Unknown") df["quantity"] = df["quantity"].fillna(df["quantity"].median()) df["quantity"] = df["quantity"].astype(int) df["order_date"] = pd.to_datetime(df["order_date"]) assert df.isnull().sum().sum() == 0, "Still missing values!" # 4. Summarize with groupby() region_summary = ( df.groupby("region") .agg( total_sales=("sales", "sum"), orders=("sales", "count"), avg_order_value=("sales", "mean"), ) .round(2) .sort_values("total_sales", ascending=False) ) product_aov = ( df.groupby("product")["sales"] .mean() .round(2) .sort_values(ascending=False) .rename("avg_order_value") ) # 5. Sort to find top performers top_orders = df.nlargest(5, "sales")[["order_date", "product", "region", "sales"]] print("\n=== Total Sales by Region ===") print(region_summary) print("\n=== Avg Order Value by Product ===") print(product_aov) print("\n=== Top 5 Individual Orders ===") print(top_orders)
Writing Up Findings
Numbers on their own aren't an analysis — the last step is turning them into plain statements someone could act on. Here's what this particular (illustrative) run of the workflow suggests. Treat these as an example of the kind of finding EDA produces, not as claims about any real business.
total_sales and the most orders in this sample — its lead comes mainly from volume, not a higher price per order.avg_order_value — a handful of high-ticket items pull the per-order average up a lot.total_sales of the four named regions but the highest avg_order_value — worth investigating whether that's fewer, larger-ticket customers.quantity column show up as float64 instead of int64, even though every actual value was a whole number?.fillna() called on quantity before .astype(int), and not the other way around?The base project works end to end — now push it further. Use the cleaned df from Section 7 as your starting point for each task below.
Now that
order_date is a real datetime column, extract the month with df["order_date"].dt.month (or build a "2024-01"-style label with .dt.to_period("M")), add it as a new column, then groupby() that column and sum sales to find which month had the highest total revenue.
Sort
region_summary by total_sales and separately by avg_order_value. Do the top regions match in both rankings? Write one sentence explaining, in your own words, why they might not.
Filter
df down to a single region with boolean indexing (e.g. df[df["region"] == "West"]), then re-run the product group-by summary from Section 5 on just that subset. Does the top product change compared to the full-dataset result?
💡 Show hints if you're stuck
- Task 1:
df["month"] = df["order_date"].dt.to_period("M"), thendf.groupby("month")["sales"].sum().sort_values(ascending=False) - Task 2:
region_summary.sort_values("avg_order_value", ascending=False)— compare its top row toregion_summary's existing total_sales order - Task 3:
west = df[df["region"] == "West"], thenwest.groupby("product")["sales"].mean().sort_values(ascending=False)