🧠 Classical ML · Tier 2
🟡 Intermediate
MODULE 06
Feature Engineering
Classical ML — Module 6 of 100%
🎯 What you'll learn: Creating new features from raw data (ratios, aggregations, interactions), every major encoding strategy for categorical data (one-hot, ordinal, target/mean encoding), extracting features from dates and text, generating polynomial interactions automatically, and pruning irrelevant features with SelectKBest and RFE.
6.1 · Creating Features from Raw Data
The Most Impactful Step in Real ML
Kaggle grandmasters consistently report that feature engineering drives more improvement than model selection. A simple linear model with 20 well-crafted features often outperforms XGBoost on raw features. The goal is to express domain knowledge numerically — give the model patterns a human would recognise.
The most productive feature types are ratios (debt-to-income), aggregations (mean spend by customer), differences (days since last purchase), and flags (is_weekend, is_new_customer).
Creating features with PandasPYTHON
import pandas as pd import numpy as np # ── Ratios and interactions ───────────────────────────── df['debt_to_income'] = df['debt'] / (df['income'] + 1e-5) df['area_per_room'] = df['total_area'] / df['rooms'].replace(0, np.nan) df['price_x_quantity'] = df['price'] * df['quantity'] # ── Log transform skewed distributions ───────────────── df['log_income'] = np.log1p(df['income']) # log(x+1), safe for 0 # ── Binary flags ──────────────────────────────────────── df['is_high_value'] = (df['total_spend'] > df['total_spend'].quantile(.9)).astype(int) df['has_missing'] = df['age'].isna().astype(int) # missingness as a signal # ── Group aggregations ────────────────────────────────── group_stats = df.groupby('customer_id')['amount'].agg(['mean', 'max', 'count']) group_stats.columns = ['cust_mean_amt', 'cust_max_amt', 'cust_txn_count'] df = df.merge(group_stats, on='customer_id', how='left')
💡
Missingness as a feature
If a value is missing non-randomly — e.g., customers who don't report income often have lower income — then the fact it's missing is informative. Create a
has_missing_income binary flag before imputing. This preserves the signal that would be lost after imputation.6.2 · Encoding Categorical Variables
Numbers Models Can Actually Use
Most models require numerical input. Categorical features need to be encoded — but the choice of encoding significantly impacts performance. Using the wrong encoding can introduce false ordinal relationships or cause target leakage.
One-Hot
When: <15 categories, no order
Creates N-1 binary columns. Use
drop='first' to avoid multicollinearity. Default choice for linear models.Ordinal
When: meaningful order exists
Maps "low"→0, "medium"→1, "high"→2. Preserves ordering. Use
OrdinalEncoder(categories=[...]).Target (Mean)
When: high-cardinality (100+ cats)
Replaces category with mean target value. Requires cross-fold application to avoid leakage. Very powerful for tree models.
Encoding strategies — sklearnPYTHON
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder from sklearn.preprocessing import TargetEncoder # sklearn ≥ 1.3 # ── One-Hot: city (5 unique values) ───────────────────── ohe = OneHotEncoder(drop='first', sparse_output=False, handle_unknown='ignore') city_encoded = ohe.fit_transform(df[['city']]) # ── Ordinal: education level (ordered) ───────────────── edu_enc = OrdinalEncoder( categories=[['high school', 'bachelor', 'master', 'phd']], handle_unknown='use_encoded_value', unknown_value=-1 ) df['edu_ord'] = edu_enc.fit_transform(df[['education']]) # ── Target Encoding: postal_code (thousands of codes) ── te = TargetEncoder(cv=5, smooth='auto') # cv folds prevent leakage df['postal_te'] = te.fit_transform(df[['postal_code']], df['target']).ravel()
⚠️
Target encoding leakage risk
Naively computing the mean target per category on the full training set and then using it as a feature is data leakage — the current row's label contributes to its own encoding. Always use cross-fold target encoding (sklearn's TargetEncoder with cv=5) or compute encodings on out-of-fold predictions only. sklearn ≥ 1.3 handles this correctly by default.
6.3 · Date and Text Features
Extracting Signal from Timestamps and Strings
Dates are almost never useful as raw timestamps. Models can't learn "Mondays have higher churn" from a Unix timestamp — you need to explicitly extract the day-of-week, hour, month, or flags like is_weekend. Text similarly needs transformation: bag-of-words or TF-IDF converts sentences to sparse numeric matrices.
Date feature extraction from datetime columnPYTHON
df['created_at'] = pd.to_datetime(df['created_at']) # Core temporal features df['year'] = df['created_at'].dt.year df['month'] = df['created_at'].dt.month df['dayofweek'] = df['created_at'].dt.dayofweek # 0=Monday df['hour'] = df['created_at'].dt.hour df['quarter'] = df['created_at'].dt.quarter # Higher-signal engineered flags df['is_weekend'] = (df['dayofweek'] >= 5).astype(int) df['is_business'] = (df['hour'].between(9, 17)).astype(int) # Elapsed time features reference_date = df['created_at'].max() df['days_since_signup'] = (reference_date - df['created_at']).dt.days df['days_since_last_txn'] = (reference_date - df['last_transaction']).dt.days
Text features — TF-IDF and simple bowPYTHON
from sklearn.feature_extraction.text import TfidfVectorizer tfidf = TfidfVectorizer( max_features=5000, # top 5000 terms by TF-IDF score ngram_range=(1, 2), # unigrams + bigrams min_df=3, # ignore terms in < 3 docs sublinear_tf=True # log(1+tf) — reduces dominance of common terms ) X_text = tfidf.fit_transform(df['description']) # sparse matrix # Cheap text-length features often beat TF-IDF alone df['text_len'] = df['description'].str.len() df['word_count'] = df['description'].str.split().str.len() df['has_question'] = df['description'].str.contains('?', regex=False).astype(int)
6.4 · Polynomial Features & Feature Selection
Expand Then Prune
If you suspect non-linear relationships but want to keep using a linear model, PolynomialFeatures automatically generates all interaction and power terms (e.g., from A, B → A², AB, B²). This can dramatically improve linear model accuracy at the cost of exponential feature growth — prune aggressively afterwards.
PolynomialFeatures + SelectKBest pipelinePYTHON
from sklearn.preprocessing import PolynomialFeatures from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif from sklearn.feature_selection import RFE from sklearn.linear_model import LogisticRegression # ── Polynomial expansion (degree 2) ──────────────────── poly = PolynomialFeatures(degree=2, include_bias=False) X_poly = poly.fit_transform(X_scaled) # 10 features → 65 features # ── SelectKBest: pick top K by F-score ───────────────── selector = SelectKBest(score_func=f_classif, k=20) X_best = selector.fit_transform(X_poly, y) # ── RFE: recursive feature elimination with model ────── rfe = RFE(estimator=LogisticRegression(max_iter=1000), n_features_to_select=15) X_rfe = rfe.fit_transform(X_scaled, y) print("Selected features:", list(df.columns[rfe.support_])) # ── Full pipeline: scale → poly → select → model ────── from sklearn.pipeline import Pipeline pipe = Pipeline([ ('scaler', StandardScaler()), ('poly', PolynomialFeatures(degree=2, include_bias=False)), ('select', SelectKBest(f_classif, k=20)), ('model', LogisticRegression(max_iter=1000)) ])
📌
Feature selection methods compared
SelectKBest — fast, univariate (tests each feature independently). Misses correlated groups. Use for quick baseline.
RFE — trains a model, removes weakest features, repeats. Captures feature interactions but slow for large datasets.
L1 regularisation (Lasso) — built-in feature selection: coefficients of irrelevant features shrink to exactly 0. Best when you want to keep the linear model itself.
Tree feature importance — use
RFE — trains a model, removes weakest features, repeats. Captures feature interactions but slow for large datasets.
L1 regularisation (Lasso) — built-in feature selection: coefficients of irrelevant features shrink to exactly 0. Best when you want to keep the linear model itself.
Tree feature importance — use
SelectFromModel with a RandomForest for non-linear feature ranking.🧩 Module 6 Check
5 questions · instant feedback
1. You have a "city" column with 4 unique values: London, Paris, Berlin, Tokyo. Which encoder should you use for a logistic regression model?
2. What is the risk of naïvely computing mean target per category on the full training set and using it as a feature?
3. You apply PolynomialFeatures(degree=2) to 10 features. Approximately how many features will result?
4. Which feature selection method is best suited for a dataset with 50 features and a moderate-size training set (~5,000 rows)?
5. A transaction timestamp is stored as a datetime. You want to predict whether a transaction is fraudulent. Which derived features are most likely to be useful?