Data Science with Python / Section 5 — Machine Learning with Scikit-Learn / Lesson 29
🤖 Section 5 · Machine Learning 🟡 Intermediate MODULE 29

Natural Language Processing Basics

⏱️ 45 min
📖 Turning Text Into Numbers
🧩 3 Quiz Questions
🏗️ 1 Challenge · 3 tasks
Your progress in Section 586%
🎯 Everything so far has been numbers. Every model in this section — LinearRegression, LogisticRegression, DecisionTreeClassifier, RandomForestClassifier, KMeans — needs its input as numeric arrays. Most real-world text (reviews, tweets, support tickets, emails) isn't numeric at all. Natural Language Processing (NLP) is the field concerned with getting computers to work with human language — and the first, most essential skill is turning raw text into the kind of numeric feature matrix every scikit-learn model already knows how to use.
Section 1

What Is NLP?

Natural Language Processing is the branch of AI/data science focused on enabling computers to read, interpret, and generate human language. It powers a huge range of everyday tools.

📧
Spam Detection
Classifying emails as spam or legitimate based on their text content.
😊
Sentiment Analysis
Classifying a review or comment as positive, negative, or neutral — this lesson's worked example.
🤖
Chatbots
Understanding a user's message well enough to generate a relevant response.

This lesson focuses on the foundational technique behind classical (pre-deep-learning) NLP: converting text into numbers a standard classifier can use, then reusing exactly the same LogisticRegression workflow from Lesson 25.

Section 2

Text Preprocessing

Before any numbers get involved, raw text usually goes through a few cleanup steps to reduce noise and make similar words match each other.

1
Lowercasing
"Great" and "great" are the same word — lowercasing prevents the model from treating them as two different tokens.
2
Tokenization
Splitting a string of text into individual units ("tokens") — usually words. "I love this movie" becomes ["i", "love", "this", "movie"].
3
Stopword removal
Extremely common words ("the", "is", "a", "and") carry little distinguishing information for many tasks and are often removed to reduce noise.

Here's what preprocessing does to a sample sentence, step by step:

IabsolutelyLOVEthisproduct

↓ lowercase + tokenize

iabsolutelylovethisproduct

↓ remove stopwords ("i", "this")

iabsolutelylovethisproduct
📝
scikit-learn's vectorizers handle most of this automatically
Rather than writing manual preprocessing code, scikit-learn's CountVectorizer and TfidfVectorizer (Section 4) both accept a lowercase=True parameter (the default) and a stop_words='english' parameter that handles lowercasing, tokenization, and stopword removal for you as part of fitting the vectorizer. Dedicated NLP libraries like NLTK or spaCy offer more advanced preprocessing (like stemming/lemmatizing "running" → "run"), but scikit-learn's built-in options are enough for the bag-of-words approach this lesson covers.
Section 3

The Bag-of-Words Concept

Bag-of-words is the simplest way to turn text into numbers: represent each document as a count of how many times each word appears, completely ignoring word order and grammar — as if the words were dumped into a bag and only their counts mattered.

Bag-of-words illustration — 3 tiny documents
Documentgoodmoviebadgreat
"good movie"1100
"bad movie"0110
"good movie, great movie"1201

Each row becomes a numeric feature vector, exactly like the rows in every dataset from earlier lessons — the "features" are simply "how many times did this specific word appear." That vector is what actually gets passed into LogisticRegression.fit() or any other scikit-learn model.

⚠️
Word order is thrown away entirely
"The dog bit the man" and "the man bit the dog" produce the identical bag-of-words vector — same words, same counts, completely different meaning. This is a genuine limitation of the bag-of-words approach; more advanced NLP techniques beyond this course (like word embeddings and transformer-based models) capture word order and context, at the cost of much more complexity.
Section 4

CountVectorizer and TfidfVectorizer

scikit-learn implements bag-of-words directly with CountVectorizer from sklearn.feature_extraction.text.

count_vectorizer.py
PYTHON 
from sklearn.feature_extraction.text import CountVectorizer

docs = [
    "good movie",
    "bad movie",
    "good movie great movie",
]

vec = CountVectorizer()
X = vec.fit_transform(docs)     # returns a sparse matrix

print("Vocabulary:", vec.get_feature_names_out())
print("Dense matrix:\n", X.toarray())
📝
Sparse matrices — why .toarray() is only for looking, not for real use
fit_transform() returns a sparse matrix, a memory-efficient format that only stores the non-zero entries — essential once vocabularies grow into the thousands of words and most documents only use a tiny fraction of them. .toarray() converts it to a normal dense NumPy array for printing/inspection, but for real datasets that conversion can use huge amounts of memory — models like LogisticRegression accept the sparse matrix directly, so there's rarely a need to call .toarray() outside of teaching examples.

TfidfVectorizer improves on plain counts with TF-IDF (Term Frequency-Inverse Document Frequency) weighting — words that appear in almost every document (even after stopword removal) get down-weighted, while words that are frequent in one document but rare across the whole collection get boosted, since they're more distinctive.

tfidf_vectorizer.py
PYTHON 
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words='english', lowercase=True)
X_tfidf = tfidf.fit_transform(docs)

print("Vocabulary:", tfidf.get_feature_names_out())
print("TF-IDF matrix:\n", X_tfidf.toarray().round(3))
🔢
CountVectorizer
Raw word counts. Simple, fast, and a fine baseline. A word that appears everywhere still gets a high value.
⚖️
TfidfVectorizer
Weighted by distinctiveness. A word common across nearly every document gets down-weighted, even if it appears often in one document.
📚
Both share the same interface
.fit_transform() to learn the vocabulary and transform training text; .transform() (not .fit_transform()) for new text afterward — same pattern as StandardScaler in Lesson 28.
🚀
TF-IDF is usually the stronger default
For most classification tasks, TfidfVectorizer tends to be a solid first choice — though CountVectorizer paired with certain models (like Naive Bayes) is also a very common, effective combination.
Section 5

Worked Example — Sentiment Classification

This ties everything together: TF-IDF vectorizing a small set of illustrative sentences, then feeding the result straight into LogisticRegression from Lesson 25 — the exact same classifier, just fed text-derived features instead of numeric measurements.

sentiment_classifier.py
PYTHON 
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Illustrative, hand-labeled sentiment examples — 1 = positive, 0 = negative
texts = [
    "I absolutely love this movie",
    "This was a fantastic experience",
    "Best purchase I have made all year",
    "Amazing quality and great service",
    "Pretty good overall, happy with it",
    "I hate this so much",
    "Terrible, worst purchase ever",
    "This made me really angry",
    "Awful, would not recommend",
    "Disappointing and frustrating experience",
]
labels = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]

# Vectorize FIRST, then split — the vectorizer's vocabulary is learned from all text
vec = TfidfVectorizer(stop_words='english', lowercase=True)
X = vec.fit_transform(texts)

X_train, X_test, y_train, y_test = train_test_split(
    X, labels, test_size=0.3, random_state=42
)

clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)

print(f"Test accuracy: {clf.score(X_test, y_test):.2f}")

# Classify brand-new text — must use .transform(), never .fit_transform() again
new_review = vec.transform(["I really enjoyed this, would buy again"])
prediction = clf.predict(new_review)[0]
probability = clf.predict_proba(new_review)[0]

print("Predicted sentiment:", "Positive" if prediction == 1 else "Negative")
print(f"P(negative)={probability[0]:.3f}, P(positive)={probability[1]:.3f}")
⚠️
10 examples is a toy dataset, not a benchmark
With only 10 hand-picked sentences (and just 3 held out for testing), this example exists purely to demonstrate the mechanics of the text → vectorize → classify pipeline correctly. Any accuracy number this specific script produces is not a meaningful measure of real-world sentiment classification performance — a genuine sentiment model needs thousands of diverse, realistic examples, exactly the same lesson Lesson 24 taught about small illustrative datasets in general.
✨
Everything from Lessons 25 and 27 still applies here
Once text is vectorized into a numeric (sparse) matrix, it's just a feature matrix like any other — train_test_split, .predict_proba(), classification_report, and cross_val_score from earlier lessons all work identically on it. NLP's special ingredient is entirely in how the features get created, not in how the model afterward is trained or evaluated.
🧩 Knowledge Check — Lesson 29
3 questions on NLP basics before you move on.
1. What does the bag-of-words model deliberately throw away?
2. What does TfidfVectorizer do differently from plain CountVectorizer?
3. After fitting a TfidfVectorizer on training text, what should you call on brand-new text at prediction time?
💪
Try It Yourself — Lesson 29
Build your own tiny text classifier · Intermediate Level

Extend Section 5's texts/labels sentiment example for all three tasks.

Task 1: Add your own examples ✍️

Add 4 new sentences to texts (2 positive, 2 negative) with matching entries in labels, then re-run the full pipeline from Section 5. Does the vocabulary size (len(vec.get_feature_names_out())) grow as expected?
Task 2: Compare CountVectorizer vs. TfidfVectorizer 🔬

Repeat the Section 5 pipeline, but swap TfidfVectorizer for CountVectorizer (same stop_words='english' setting). Does the test accuracy on your expanded dataset change?
Task 3: Inspect the vocabulary 📖

Print vec.get_feature_names_out() for your fitted vectorizer and manually check: did stopword removal actually drop words like "this," "was," and "and" from the vocabulary?
💡 Show hints if you're stuck
  • Task 1: Keep the same 1/0 label convention — 1 for positive, 0 for negative — and make sure len(texts) == len(labels) still holds.
  • Task 2: vec = CountVectorizer(stop_words='english'); X = vec.fit_transform(texts) — everything else in the pipeline stays identical.
  • Task 3: print(sorted(vec.get_feature_names_out())) — common English stopwords should be absent from the printed list.
Finished this lesson?
Mark it complete to track your progress.
🎉

Lesson 29 Complete!

You can now turn raw text into numeric features with bag-of-words and TF-IDF, and feed the result into a standard classifier. One lesson left in this course — the capstone project, combining everything from Sections 1 through 5.

Module 29 of 30 Section 5 — Machine Learning with Scikit-Learn