Tutorial datasets are clean: all-numeric, no missing values, ready to feed straight into model.fit(). Real data never looks like this. A typical dataset has:
Each column type needs different preprocessing — and it all needs to happen correctly inside cross-validation, without leakage, and be saveable as one object. This is exactly what ColumnTransformer + Pipeline solves.
ColumnTransformer lets you apply different transformers to different columns, then combines the outputs side by side into one array.
from sklearn.compose import ColumnTransformer from sklearn.preprocessing import StandardScaler, OneHotEncoder from sklearn.impute import SimpleImputer from sklearn.pipeline import Pipeline numeric_features = ['age', 'salary', 'tenure_years'] categorical_features = ['city', 'department'] # Numeric pipeline: fill missing values, THEN scale numeric_pipeline = Pipeline([ ('imputer', SimpleImputer(strategy='median')), ('scaler', StandardScaler()) ]) # Categorical pipeline: fill missing values, THEN encode categorical_pipeline = Pipeline([ ('imputer', SimpleImputer(strategy='most_frequent')), ('encoder', OneHotEncoder(handle_unknown='ignore')) ]) # Combine both into one ColumnTransformer preprocessor = ColumnTransformer(transformers=[ ('num', numeric_pipeline, numeric_features), ('cat', categorical_pipeline, categorical_features), ])
Pipeline (impute → scale, impute → encode). This is normal and powerful — pipelines can be nested. The numeric branch never touches categorical columns and vice versa.
Real data has gaps. SimpleImputer fills them using a strategy you choose:
| Strategy | Fills with | Use for |
|---|---|---|
mean | Average of the column | Numeric, roughly normal distribution |
median | Middle value | Numeric with outliers (safer default) |
most_frequent | Mode (most common value) | Categorical columns |
constant | A fixed value you specify | When missing itself is meaningful (e.g. fill with 'Unknown') |
Now attach the preprocessor to a model. The whole thing becomes one fittable, predictable, saveable object:
from sklearn.ensemble import GradientBoostingClassifier from sklearn.model_selection import train_test_split, cross_val_score import pandas as pd import numpy as np # Realistic messy dataset data = pd.DataFrame({ 'age': [25, 32, np.nan, 19, 41, 38], 'salary': [40000, 75000, 95000, 30000, np.nan, 88000], 'tenure_years': [1, 5, 8, 0.5, 12, 6], 'city': ['London', 'Paris', 'London', np.nan, 'Berlin', 'Paris'], 'department': ['Eng', 'HR', 'Eng', 'Finance', 'Eng', 'HR'], }) y = [1, 0, 1, 0, 1, 0] # (preprocessor built exactly as in the previous example) # Full pipeline: preprocessing + model in ONE object full_pipeline = Pipeline([ ('preprocessor', preprocessor), ('model', GradientBoostingClassifier(random_state=42)) ]) X_train, X_test, y_train, y_test = train_test_split( data, y, test_size=0.33, random_state=42 ) # One call does: impute → scale/encode → fit model full_pipeline.fit(X_train, y_train) # One call does: impute → scale/encode → predict predictions = full_pipeline.predict(X_test) # Cross-validate the WHOLE pipeline — no leakage between folds scores = cross_val_score(full_pipeline, data, y, cv=3) print(f"CV scores: {scores}")
Because everything is one object, deployment is one line:
import joblib # Save the entire fitted pipeline — preprocessing AND model joblib.dump(full_pipeline, 'model_pipeline.joblib') # Later, in production, load and use directly loaded_pipeline = joblib.load('model_pipeline.joblib') # New raw data — no manual preprocessing needed new_data = pd.DataFrame({ 'age': [29], 'salary': [52000], 'tenure_years': [2], 'city': ['Madrid'], # unseen city — handled by handle_unknown='ignore' 'department': ['Eng'] }) prediction = loaded_pipeline.predict(new_data) print(prediction)
new_data is raw, unprocessed data with a missing-value-free but never-seen city ("Madrid"). The pipeline handles imputation, encoding (including the unknown category), scaling, and prediction — all from one .predict() call. This is exactly how a FastAPI endpoint would use it.
Using a dataset of your choice (or the example above), build a complete pipeline that:
data.select_dtypes().handle_unknown='ignore' is set on the encoder.cross_val_score with cv=5 on the full pipeline. Report mean and std.handle_unknown='ignore' do in OneHotEncoder?handle_unknown='ignore' prevents crashes on unseen categories in production