Data Science with Python / Section 4 — Statistics & Probability for Data Science / Lesson 21
📈 Section 4 · Statistics 🟡 Intermediate MODULE 21

Hypothesis Testing & p-values

⏱️ 50 min
📖 t-tests with SciPy
🧩 3 Quiz Questions
🏗️ 1 Challenge · 3 tasks
Your progress in Section 460%
🎯 The question this lesson answers: You ran an A/B test and Group B converted better than Group A — but is that a real effect, or could it just be random noise from small samples? Hypothesis testing is the formal framework for answering that, and the p-value is its most famous — and most misunderstood — output. This lesson builds the concept from the ground up and ends with a real two-sample t-test using scipy.stats.ttest_ind.
Section 1

Null Hypothesis vs. Alternative Hypothesis

Every hypothesis test starts by writing down two competing claims about the world, then using data to decide which one the evidence favors.

H₀ — Null Hypothesis
The "nothing interesting is happening" claim. There's no real difference or effect — any difference observed in the sample is just random variation. This is the assumption you start by defaulting to, and try to find evidence against.
H₁ — Alternative Hypothesis
The claim you're actually interested in — that there IS a real effect or difference. You never directly "prove" H₁; you only ever gather enough evidence to reject H₀ in its favor, or fail to.
📝
Example: does a new website design increase conversion rate?
H₀: the new design's true conversion rate equals the old design's — the difference you measured in your sample is just noise. H₁: the new design's true conversion rate is different from the old design's. The t-test later in this lesson is exactly this scenario, worked with real numbers.
Section 2

What a p-value Actually Means

This is the single most misunderstood number in applied statistics, so it's worth being precise.

The Correct Definition
p-value = P(data this extreme or more extreme | H₀ is true)
The probability of observing a result at least as extreme as what you got, ASSUMING the null hypothesis is true.

A small p-value means: "if there really were no effect, seeing data this extreme would be unlikely." That's evidence against H₀ — but it is not a statement about how likely H₀ or H₁ is to be true.

⚠️
The #1 misinterpretation to avoid
A p-value of 0.03 does NOT mean "there's a 3% chance the null hypothesis is true" or "there's a 97% chance the effect is real." The p-value is computed by assuming H₀ is true in the first place — it can't simultaneously tell you the probability that assumption is wrong. It only answers: "how surprising is this data, if H₀ were true?" Confusing these two questions is one of the most common statistical errors in published research, and it's worth internalizing the difference early.
✨
A rough mental model
Think of the p-value as a measure of surprise, not proof. Small p-value → "this would be a surprising coincidence if there were truly no effect, so maybe there is one." Large p-value → "this result is unremarkable under the assumption of no effect — the data doesn't give us a strong reason to abandon H₀."
Section 3

The Significance Level — α = 0.05

A p-value alone doesn't tell you whether to reject H₀ — you need a threshold to compare it against, decided before looking at the results. That threshold is the significance level, written α (alpha), and the overwhelmingly common convention is α = 0.05.

1
Choose α before the test
Most commonly 0.05 (a 5% threshold) — occasionally 0.01 for stricter fields, or 0.10 for exploratory work.
2
Run the test, get a p-value
Using a function like scipy.stats.ttest_ind(), covered next.
3
Compare p-value to α
If p ≤ α: reject H₀ — the result is "statistically significant." If p > α: fail to reject H₀ — not enough evidence against it.
📝
"Fail to reject" is not "accept"
If p > α, that doesn't prove H₀ is true — it just means this particular dataset didn't provide strong enough evidence against it. A larger sample, or a real effect that's simply small, could both produce the same "fail to reject" result. Statistical language is deliberately asymmetric here: you can reject H₀, but you never "accept" it outright.
Section 4

A Worked Example: Two-Sample t-test

A t-test compares the means of two groups and asks: is the difference between them larger than you'd expect from random sampling alone? Here's the A/B test scenario from Section 1, worked with scipy.stats.ttest_ind — "ind" for "independent samples."

ab_test_ttest.py
PYTHON 
import numpy as np
from scipy import stats

# Time on page (seconds) for visitors shown the OLD design vs. the NEW design
group_a_old = np.array([32, 28, 35, 30, 27, 33, 29, 31, 26, 34])
group_b_new = np.array([38, 41, 36, 44, 39, 37, 42, 35, 40, 38])

print("Mean A (old):", group_a_old.mean().round(2))
print("Mean B (new):", group_b_new.mean().round(2))

# H0: the two designs have the same true mean time-on-page
# H1: the two designs have different true mean time-on-page
t_stat, p_value = stats.ttest_ind(group_a_old, group_b_new)

print(f"t-statistic: {t_stat:.3f}")
print(f"p-value:     {p_value:.5f}")

alpha = 0.05
if p_value <= alpha:
    print(f"p ({p_value:.5f}) <= alpha ({alpha}) -> reject H0. Difference is statistically significant.")
else:
    print(f"p ({p_value:.5f}) > alpha ({alpha}) -> fail to reject H0. Not enough evidence of a difference.")
Output
Mean A (old) / Mean B (new)30.5 / 39.0
t-statistic≈ −6.4 (exact value depends on the SciPy version's floating-point computation)
p-valuewell below 0.05
Conclusionp ≤ alpha → reject H0. The new design's higher time-on-page in this sample is unlikely to be pure chance.
✨
ttest_ind() assumes equal variances by default
scipy.stats.ttest_ind(a, b) defaults to equal_var=True — Student's t-test, which assumes both groups have similar spread. If that assumption looks shaky (e.g. one group's std is much larger than the other's), pass equal_var=False to run Welch's t-test instead, which doesn't require it and is often the safer default in practice.
📝
The sign of the t-statistic just reflects argument order
Passing group_a_old first and group_b_new second produces a negative t-statistic because A's mean is smaller than B's — swap the argument order and the sign flips, but the p-value (and the conclusion) stays exactly the same. The magnitude of t, not its sign, is what matters for significance.
Section 5

Type I vs. Type II Errors

Every hypothesis test can be wrong in two different ways, and it's worth knowing which is which — because they're a direct trade-off, not independent risks.

Type I Error (False Positive)
Rejecting H₀ when it was actually true — concluding there's an effect when there isn't one. The significance level α IS the Type I error rate you've chosen to tolerate: at α = 0.05, you accept a 5% chance of a false positive on any single test where H₀ is actually true.
Type II Error (False Negative)
Failing to reject H₀ when it was actually false — missing a real effect that was there. Its probability is called β (beta); a test's "power" (1 − β) is its ability to detect a real effect, which improves with larger sample sizes.
⚠️
Lowering α to reduce false positives increases false negatives
Choosing a stricter α (say 0.01 instead of 0.05) makes it harder to falsely reject a true H₀ — but it also makes it harder to detect a real effect that IS there, raising the Type II error rate. There's no free lunch: for a fixed sample size, reducing one error type increases the other. The usual way to lower both simultaneously is to collect more data.
🧩 Knowledge Check — Lesson 21
3 questions on hypothesis testing before you move on.
1. A test returns a p-value of 0.02. What does this correctly mean?
2. Using scipy.stats.ttest_ind(group_a, group_b) with alpha = 0.05, if the returned p-value is 0.15, what's the correct conclusion?
3. Rejecting the null hypothesis when it was actually true is called:
💪
Try It Yourself — Lesson 21
Run your own t-test · Intermediate Level

Two versions of an email subject line were sent to different customer segments. Click-through counts (out of many sends, summarized as a per-day rate) were logged over 8 days each: subject_a = [12, 15, 11, 14, 13, 16, 12, 15] and subject_b = [18, 20, 17, 22, 19, 21, 18, 23].

Task 1: State the hypotheses 📝

Before writing any code, write out H₀ and H₁ in plain English for this comparison — what would "no real difference" mean here?
Task 2: Run the t-test 🧪

Use stats.ttest_ind(subject_a, subject_b) to get the t-statistic and p-value. At α = 0.05, do you reject or fail to reject H₀?
Task 3: Try Welch's t-test 🔬

Re-run the test with equal_var=False. Does the conclusion change? Compute both groups' standard deviations with np.std(..., ddof=1) first to see how different their spreads actually are.
💡 Show hints if you're stuck
  • Task 1: H0 — subject A and subject B produce the same true average click-through rate. H1 — they produce different true average rates.
  • Task 2: t_stat, p_value = stats.ttest_ind(subject_a, subject_b), then compare p_value to 0.05.
  • Task 3: stats.ttest_ind(subject_a, subject_b, equal_var=False) — with data this clean, the conclusion is unlikely to flip, but the exact p-value will differ slightly.
Finished this lesson?
Mark it complete to track your progress.
🎉

Lesson 21 Complete!

You can now state hypotheses, interpret a p-value correctly, and run a real t-test with scipy.stats.ttest_ind. Next: correlation and regression — measuring how two variables move together, and fitting a line through them.

Module 21 of 23 Section 4 — Statistics & Probability for Data Science