📐 Section 2 · Supervised Learning 🟡 Intermediate MODULE 09

Naive Bayes — Probabilistic Classification

⏱️ 22 min read

📖 Probability-Based Learning

🧩 4 Quiz Questions

🏗️ 1 Challenge · 3 tasks

Your progress in Section 257%

🎯 What you'll learn: Naive Bayes takes an entirely different route to classification than anything so far in Section 2 — instead of geometry (SVM's margins, KNN's distances) or an optimized weighted sum (linear regression), it's built directly on probability theory. You'll see Bayes' theorem stated plainly, understand the "naive" assumption that gives the algorithm its name, and use GaussianNB and MultinomialNB from sklearn.naive_bayes — including why this family of models is a classic choice for spam filtering and text classification.

Section 1

Bayes' Theorem, Stated Plainly

Bayes' theorem is a rule for updating a belief once new evidence comes in. In classification terms: given some features (the evidence), what's the probability that a data point belongs to a particular class?

Bayes' Theorem

P(class | features) = P(features | class) · P(class) / P(features)

Read "P(A | B)" as "the probability of A, GIVEN that B is true."

🎯

P(class | features)

The "posterior" — what we actually want: the probability of each class, GIVEN the observed features. This is what decides the prediction.

📊

P(features | class)

The "likelihood" — how probable these particular feature values are, IF the point really belongs to that class. Estimated from training data.

📈

P(class)

The "prior" — how common that class is overall, before looking at any features at all (e.g. what fraction of all emails are spam).

➗

P(features)

A normalizing constant — the same for every class being compared, so it can often be ignored when just comparing which class is most likely.

📧

The spam-filter framing

"Given that this email contains the word 'free', what's the probability it's spam?" Bayes' theorem answers this by combining: how often 'free' appears in KNOWN spam emails (likelihood), how common spam is overall (prior), and normalizing against how often 'free' appears across all emails. Naive Bayes classifiers do exactly this kind of calculation — just across many words at once, and comparing spam vs. not-spam to see which comes out more probable.

Section 2

The "Naive" Assumption

Computing P(features | class) exactly would require knowing how every feature interacts with every other feature, given the class — that gets computationally expensive (and hard to estimate from limited data) very fast as the number of features grows. Naive Bayes sidesteps this with a bold simplification:

The Naive (Conditional Independence) Assumption

P(x₁, x₂, ..., xₙ | class) ≈ P(x₁|class) · P(x₂|class) · ... · P(xₙ|class)

Assume every feature is independent of every other feature, once you already know the class.

This is "naive" because it's almost certainly not literally true — in a spam email, the word "free" and the word "winner" probably DO tend to show up together more than pure independence would predict. The assumption is a simplification made for tractability, not a claim that features are truly unrelated.

✨

Why the "wrong" assumption still works well in practice

Naive Bayes doesn't need a perfectly accurate PROBABILITY estimate to get the CLASSIFICATION right — it just needs to rank the correct class highest among the candidates, even if the exact probability numbers are somewhat off. In practice, this makes Naive Bayes surprisingly competitive, especially for text classification where there are thousands of features (words) and estimating their true joint interactions would be hopeless anyway. It's also extremely fast to train — computing each P(feature | class) independently is cheap — which matters when working with large, high-dimensional, sparse feature sets like word counts.

Section 3

GaussianNB and MultinomialNB

scikit-learn provides several Naive Bayes variants, differing in what assumption they make about HOW each feature's likelihood P(feature | class) is distributed. The two you'll reach for most often:

🔔

GaussianNB

Assumes continuous numeric features follow a normal (bell-curve) distribution within each class. Good default for continuous features like measurements or sensor readings.

🔢

MultinomialNB

Designed for discrete count data — classically, word counts in a document. The standard choice for text classification with a bag-of-words representation.

gaussian_nb.py

PYTHON

from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split

# GaussianNB works well for continuous numeric features (no scaling required)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = GaussianNB()
model.fit(X_train, y_train)

print("Test accuracy:", model.score(X_test, y_test))
print("Predicted class probabilities:", model.predict_proba(X_test)[:3])

For text data, the workflow first converts raw text into word counts (a "bag of words") before MultinomialNB ever sees it — typically with scikit-learn's CountVectorizer:

multinomial_nb_text.py

PYTHON

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split

# emails: list of raw email text strings. labels: 0 = not spam, 1 = spam
emails_train, emails_test, y_train, y_test = train_test_split(
    emails, labels, test_size=0.2, random_state=42
)

# Convert text into word-count vectors — fit vocabulary on train only
vectorizer = CountVectorizer(stop_words="english")
X_train = vectorizer.fit_transform(emails_train)
X_test = vectorizer.transform(emails_test)

model = MultinomialNB()
model.fit(X_train, y_train)

print("Test accuracy:", model.score(X_test, y_test))

new_email = ["Congratulations, you have WON a free prize, click now!"]
new_email_vec = vectorizer.transform(new_email)
print("Prediction (0=not spam, 1=spam):", model.predict(new_email_vec))

📝

CountVectorizer must be fit on training data only

Just like StandardScaler in Lesson 3 and Lesson 8, vectorizer.fit_transform() is only called on the training text — it learns the vocabulary from those documents. The test set uses .transform() only, mapping onto that SAME vocabulary, to avoid leaking test-set information into preprocessing.

Section 4

Where Naive Bayes Shines

Despite its simplifying assumption, Naive Bayes remains a genuinely useful, widely-used algorithm — particularly in a specific niche.

📧

Spam filtering

The textbook use case — fast, effective, and interpretable word-level probabilities.

📰

Text/document classification

Topic tagging, sentiment analysis baselines, news categorization.

⚡

Fast baselines

Trains almost instantly even on huge feature counts — a great sanity-check model before reaching for something heavier.

✨

Why high-dimensional, sparse data is Naive Bayes' comfort zone

Text data, represented as word counts, typically has THOUSANDS of features (one per vocabulary word) where any single document only uses a small fraction of them — mostly zeros (sparse). Estimating true feature interactions in that setting is essentially impossible with realistic amounts of data. The naive independence assumption, which would be a real liability on a small dataset with a handful of genuinely correlated features, becomes much less costly when there are simply too many features to model interactions between them anyway.

⚠️

Where it tends to struggle

When features are few and STRONGLY correlated with each other, the independence assumption costs more accuracy, and other algorithms from this section (SVM, gradient boosting) often outperform it. Naive Bayes is best thought of as a fast, strong baseline for high-dimensional problems like text — not a universal default.

Section 5

Lesson Summary

✅Bayes' theorem combines a prior P(class) with a likelihood P(features|class) to compute P(class|features).

✅The naive assumption treats features as conditionally independent given the class — technically wrong, but effective in practice, especially for ranking classes correctly.

✅GaussianNB suits continuous features; MultinomialNB suits discrete counts, like word frequencies via CountVectorizer.

✅Naive Bayes shines on high-dimensional, sparse data like text — spam filtering and text classification are its classic strengths.

🧩 Knowledge Check — Lesson 9

4 questions on Bayes' theorem and Naive Bayes classifiers.

1. In Bayes' theorem, what does P(class) represent?

2. What is the "naive" assumption in Naive Bayes?

3. Which Naive Bayes variant is the standard choice for word-count based text classification?

4. Why does Naive Bayes often perform well on text classification despite its simplifying assumption?

💪

Try It Yourself — Lesson 9

Build a spam classifier and reason about the assumption · Intermediate Level

These tasks put both Naive Bayes variants to work.

Task 1: Fit a GaussianNB and read probabilities 🔔

Using Section 3's code, fit a GaussianNB on any numeric classification dataset. Print model.predict_proba(X_test)[:5] and explain in a sentence what each row of numbers represents.

Task 2: Build a tiny spam classifier 📧

Using Section 3's CountVectorizer + MultinomialNB pattern, create a small list of ~10 example texts (some "spam-like," some not), with matching labels. Fit the model and test it on 2–3 new sentences you write yourself.

Task 3: Argue both sides of the naive assumption ⚖️

Pick a real-world classification scenario with strongly correlated features (e.g. predicting house price category from square footage AND number of bedrooms, which usually move together). Write 2–3 sentences on why the naive independence assumption might hurt Naive Bayes here more than it would for spam text classification.

💡 Show hints if you're stuck

Task 1: Each row is the predicted probability of each class for one test row — the columns sum to 1.0, and .predict() would return whichever class has the highest probability.
Task 2: Try phrases with words like "free," "win," "click now" for spam-like examples vs. ordinary sentences for the other class — with only 10 examples, don't expect great accuracy, but the mechanics should work end to end.
Task 3: With square footage and bedroom count, treating them as independent given the price category ignores that bigger houses almost always have more bedrooms — the model may double-count that shared signal or mis-weight it, unlike in text where thousands of largely unrelated words dilute any single dependency.

Finished this lesson?

Mark it complete to track your progress.

Module 09 of 24 Section 2 — Supervised Learning Algorithms