Data Science with Python / Section 4 — Statistics & Probability for Data Science / Lesson 22
📈 Section 4 · Statistics 🟡 Intermediate MODULE 22

Correlation & Regression Analysis

⏱️ 50 min
📖 Pearson r & Line of Best Fit
🧩 3 Quiz Questions
🏗️ 1 Challenge · 3 tasks
Your progress in Section 480%
🎯 The last piece before Section 5. Descriptive stats summarize one variable at a time. This lesson looks at two variables together — do they move in sync? Can you predict one from the other? Correlation measures how strongly two variables move together, and regression fits a line that predicts one from the other. This is also where the course's single most important warning lives: correlation is not causation, and it's worth understanding exactly why.
Section 1

The Pearson Correlation Coefficient

The Pearson correlation coefficient, written r, measures the strength and direction of the linear relationship between two numeric variables.

Pearson Correlation Coefficient
r = Σ[(xᵢ − x̄)(yᵢ − ȳ)] / √(Σ(xᵢ − x̄)² · Σ(yᵢ − ȳ)²)
The covariance of x and y, normalized by the product of their individual spreads — this normalization is what keeps r always between −1 and 1, regardless of the variables' original units.
📈
r close to +1
Strong positive relationship — as x increases, y tends to increase too, in a roughly straight-line pattern.
📉
r close to −1
Strong negative relationship — as x increases, y tends to decrease.
➖
r close to 0
Little to no LINEAR relationship. There could still be a strong non-linear pattern that r completely misses.
📏
r is unitless
Whether x is in dollars and y is in years, or any other units, r is always a plain number between −1 and 1.
pearson_correlation.py
PYTHON 
import numpy as np
import pandas as pd
from scipy import stats

# Hours studied vs. exam score, for 10 students
hours_studied = np.array([1, 2, 2, 3, 4, 4, 5, 6, 7, 8])
exam_score    = np.array([52, 58, 55, 64, 68, 70, 75, 80, 85, 92])

# NumPy: full correlation MATRIX (r between every pair of variables)
corr_matrix = np.corrcoef(hours_studied, exam_score)
print("Correlation matrix:\n", corr_matrix)
print("r (hours vs score):", corr_matrix[0, 1].round(3))

# scipy.stats: r AND a p-value testing whether r is significantly different from 0
r, p_value = stats.pearsonr(hours_studied, exam_score)
print(f"r = {r:.3f}, p-value = {p_value:.5f}")
Output
r (hours vs. score)≈ 0.99 — a very strong positive linear relationship
p-valuewell below 0.05 — the correlation is statistically significant, not likely due to chance in this sample
📝
np.corrcoef() returns a matrix, not a single number
np.corrcoef(a, b) returns a 2×2 matrix: the diagonal is always 1 (a variable perfectly correlates with itself), and the off-diagonal values, [0,1] and [1,0], both hold the r between a and b — the matrix is symmetric. scipy.stats.pearsonr() is often more convenient for two variables since it directly returns just (r, p_value).
Section 2

Correlation Is Not Causation

This is the single most important idea in this lesson. A high r tells you two variables move together — it tells you nothing about why. There are several ways two variables can be strongly correlated without one causing the other.

1
Reverse causation
Maybe y actually causes x, not the other way around — the correlation is real, but the direction you assumed is backwards.
2
A confounding variable
A third factor drives both. Ice cream sales and drowning deaths correlate — both rise in summer heat, which drives both independently. Neither causes the other.
3
Coincidence
With enough variables, some will correlate strongly by pure chance — especially in small samples, or when testing many pairs at once.
4
Genuine causation
Sometimes x really does cause y — but correlation alone can never distinguish this case from the three above. That requires a controlled experiment (like the A/B test and t-test from Lesson 21), or careful causal-inference methods beyond this course.
⚠️
The hours-studied example, revisited
The r ≈ 0.99 between hours studied and exam score above is highly suggestive — but even here, "studying more causes higher scores" is a reasonable interpretation, not something the correlation coefficient itself proves. A student's baseline preparation, access to a tutor, or general motivation could all be confounding variables driving both. Correlation is a strong first clue that points you toward a hypothesis — it's not the proof itself.
Section 3

df.corr() and Visualizing with a Heatmap

For a whole DataFrame with several numeric columns, pandas computes every pairwise correlation at once with df.corr() — and a Seaborn heatmap from Lesson 15 turns that grid of numbers into something you can read at a glance.

corr_heatmap.py
PYTHON 
import seaborn as sns
import matplotlib.pyplot as plt

df = pd.DataFrame({
    "hours_studied": [1, 2, 2, 3, 4, 4, 5, 6, 7, 8],
    "exam_score":    [52, 58, 55, 64, 68, 70, 75, 80, 85, 92],
    "sleep_hours":   [8, 7, 8, 6, 7, 6, 6, 5, 6, 5],
})

corr = df.corr()
print(corr)

fig, ax = plt.subplots(figsize=(6, 5))
sns.heatmap(corr, annot=True, cmap="coolwarm", vmin=-1, vmax=1, ax=ax)
ax.set_title("Correlation Matrix")
plt.show()
Illustrative correlation matrix — sns.heatmap(annot=True) output
hours_studiedexam_scoresleep_hours
hours_studied1.000.99-0.85
exam_score0.991.00-0.81
sleep_hours-0.85-0.811.00
✨
Reading a diverging colormap like "coolwarm"
cmap="coolwarm" maps values near +1 to warm red/orange, values near −1 to cool blue, and values near 0 to a neutral color in between. Setting vmin=-1, vmax=1 explicitly is important — without it, Seaborn scales the color range to the data's own min/max, which can make a moderate 0.5 correlation look artificially "maxed out" red.
⚠️
The sleep_hours correlation is a reminder, not a conclusion
sleep_hours correlates negatively with exam_score in this illustrative data — but that doesn't mean sleeping less causes better grades. It's far more plausible that busier, higher-achieving students both study more AND sleep less — a confounding pattern exactly like Section 2's ice cream example, not evidence that cutting sleep helps.
Section 4

Simple Linear Regression — Fitting a Line

Where correlation measures how strongly two variables relate, regression goes a step further: it fits an actual straight line through the data, which you can then use to predict y from a new x.

Line of Best Fit
ŷ = mx + b
m is the slope (how much y changes per unit of x); b is the intercept (predicted y when x = 0). The "best fit" line is the one that minimizes the sum of squared vertical distances between the line and every actual data point — ordinary least squares.
linear_regression.py
PYTHON 
# scipy.stats.linregress — slope, intercept, r, p-value, and std error, all at once
result = stats.linregress(hours_studied, exam_score)

print(f"slope (m):     {result.slope:.3f}")
print(f"intercept (b): {result.intercept:.3f}")
print(f"r-value:       {result.rvalue:.3f}")
print(f"r-squared:     {result.rvalue**2:.3f}")
print(f"p-value:       {result.pvalue:.5f}")

# Predict the exam score for a student who studies 5.5 hours
predicted = result.slope * 5.5 + result.intercept
print(f"Predicted score at 5.5 hours: {predicted:.1f}")

# np.polyfit(x, y, degree) is an equivalent, more general alternative
m, b = np.polyfit(hours_studied, exam_score, deg=1)
print(f"np.polyfit slope/intercept: {m:.3f}, {b:.3f}")

# Plot the data with the fitted line on top
fig, ax = plt.subplots(figsize=(7, 5))
ax.scatter(hours_studied, exam_score, color="#4f9eff", label="Actual scores")
x_line = np.linspace(hours_studied.min(), hours_studied.max(), 100)
ax.plot(x_line, result.slope * x_line + result.intercept, color="#2de8c0", linewidth=2.5, label="Line of best fit")
ax.set_xlabel("Hours Studied")
ax.set_ylabel("Exam Score")
ax.legend()
plt.show()
✨
r-squared: how much variance the line explains
Squaring the correlation coefficient gives r² ("r-squared"), interpreted as the proportion of variance in y that's explained by the linear relationship with x. An r-squared of 0.98 means the line explains about 98% of the variation in exam scores — very high, though again, "explains" describes the statistical fit, not a proven causal mechanism.
📝
linregress() vs. polyfit() — when to use which
scipy.stats.linregress() is specialized for exactly one x and one y, and bundles in r-value, p-value, and standard error for free — ideal for this kind of statistical analysis. np.polyfit(x, y, deg=1) is more general — pass deg=2 or higher for a curved polynomial fit instead of a straight line — but it only returns the fitted coefficients, none of the statistical extras.
🧩 Knowledge Check — Lesson 22
3 questions on correlation and regression before you move on.
1. A dataset shows a Pearson correlation of r = -0.92 between "hours of TV watched" and "exam score." What can you correctly conclude?
2. In the line of best fit ŷ = mx + b, what does the slope m represent?
3. Ice cream sales and drowning incidents both rise in summer and correlate strongly. What best explains this?
💪
Try It Yourself — Lesson 22
Fit your own line · Intermediate Level

Use this illustrative dataset relating advertising spend (in thousands of PKR) to weekly units sold: ad_spend = [10, 15, 20, 25, 30, 35, 40, 45], units_sold = [102, 118, 135, 149, 168, 180, 195, 210].

Task 1: Compute r 📊

Use stats.pearsonr(ad_spend, units_sold) to get the correlation coefficient and its p-value. How strong is the relationship?
Task 2: Fit and interpret the line 📈

Use stats.linregress(ad_spend, units_sold) to get the slope and intercept. In plain English, what does the slope mean in terms of "units sold per extra 1,000 PKR spent on ads"?
Task 3: Predict, and consider the caveat ⚠️

Use the fitted line to predict units sold at an ad spend of 50 (thousand PKR) — a value beyond the original data range. Then write one sentence on why extrapolating a line beyond the range it was fit on is riskier than predicting within that range.
💡 Show hints if you're stuck
  • Task 1: r, p = stats.pearsonr(ad_spend, units_sold) — expect r to be very close to 1 for this clean, illustrative data.
  • Task 2: result = stats.linregress(ad_spend, units_sold), then result.slope is the predicted increase in units_sold per 1 unit increase in ad_spend (i.e. per 1,000 PKR).
  • Task 3: result.slope * 50 + result.intercept — the caveat: the line was only fit on spend values from 10 to 45, so a prediction at 50 assumes the same linear trend continues beyond what was actually observed, which may not hold.
Finished this lesson?
Mark it complete to track your progress.
🎉

Lesson 22 Complete!

You can now measure correlation with Pearson's r, visualize it with a heatmap, fit a line of best fit, and — most importantly — explain why correlation isn't causation. One lesson left in Section 4: the checkpoint quiz.

Module 22 of 23 Section 4 — Statistics & Probability for Data Science