Free lesson · ML System Design Interviews
Feature Engineering Patterns
What Is Feature Engineering?
Feature engineering is the art of turning raw data into numbers that ML models can learn from effectively.
Think of it like preparing ingredients for cooking:
- Raw data = whole chickens, bags of flour, raw eggs
- Features = diced chicken, sifted flour, beaten eggs
- The model = the chef who turns prepared ingredients into a great dish
A chef won't use raw ingredients - they prepare them. Similarly, ML models work best when you give them carefully prepared features.
💡 Tip: 80% of ML success comes from feature engineering, not from choosing the fanciest model.
Common Feature Engineering Patterns
Pattern 1: Bucketing (Binning)
Convert continuous numbers into categories (bins).
Why? Some ML models work better with categories, and bucketing can capture non-linear relationships.
Example: Age groups
Raw data:
user_age: 23, 45, 67, 19, 55, ...
After bucketing:
age_bucket: "18-25", "45-54", "65+", "18-25", "45-54", ...
Code Example:
import pandas as pd
data = pd.DataFrame({
'user_age': [23, 45, 67, 19, 55, 32, 71, 21]
})
# Method 1: Custom bins
data['age_bucket'] = pd.cut(
data['user_age'],
bins=[0, 18, 25, 35, 50, 65, 100],
labels=['<18', '18-25', '25-35', '35-50', '50-65', '65+']
)
print(data)
# age_bucket: "<18", "18-25", "65+", "18-25", "50-65", "25-35", "65+", "18-25"
📝 Note: Bucketing is especially useful for age, income, price, and other continuous variables that have threshold effects (e.g., "over 65" might qualify for senior discounts).
Pattern 2: One-Hot Encoding
Convert categorical data into binary features.
Why? Most ML models need numbers, not text categories.
Example: Favorite color
Raw data:
color: "red", "blue", "red", "green", ...
After one-hot encoding:
color_is_red: [1, 0, 1, 0, ...]
color_is_blue: [0, 1, 0, 0, ...]
color_is_green: [0, 0, 0, 1, ...]
Code Example:
import pandas as pd
data = pd.DataFrame({
'favorite_color': ['red', 'blue', 'red', 'green', 'blue']
})
# One-hot encode
encoded = pd.get_dummies(data['favorite_color'], prefix='color')
print(encoded)
# color_blue color_green color_red
# 0 0 0 1
# 1 1 0 0
# 2 0 0 1
# 3 0 1 0
# 4 1 0 0
⚠️ Watch out: With 1000 categories, you'll create 1000 columns! Use careful filtering for high-cardinality features.
Pattern 3: Normalization (Scaling)
Convert features to a standard scale so they're comparable.
Why? Some models get confused when features have wildly different ranges (e.g., age 0-100 vs income 0-1,000,000).
Example: Scaling to 0-1
Raw data:
age: 25, 45, 67
income: 50000, 120000, 200000
After normalization:
age_normalized: 0.0, 0.63, 1.0
income_normalized: 0.0, 0.35, 1.0
Now both features are on the same 0-1 scale.
Code Example:
from sklearn.preprocessing import MinMaxScaler
data = pd.DataFrame({
'age': [25, 45, 67],
'income': [50000, 120000, 200000]
})
scaler = MinMaxScaler() # Scales to 0-1
normalized = pd.DataFrame(
scaler.fit_transform(data),
columns=['age_norm', 'income_norm']
)
print(normalized)
# age_norm income_norm
# 0 0.0 0.0
# 1 0.63 0.35
# 2 1.0 1.0
💡 Tip: Use StandardScaler if your data is normally distributed (bell curve). Use MinMaxScaler if you need fixed 0-1 range.
Pattern 4: Log Transforms
Take the logarithm of a number to compress large ranges and make distributions more normal.
Why? Some data is highly skewed (like user spending: most people spend
0, few spend 0,000). Log transform makes it more balanced.Example: User spending
Raw data:
spending: [10, 15, 20, 50, 100, 500, 5000]
After log transform:
log_spending: [1.0, 1.18, 1.30, 1.70, 2.0, 2.70, 3.70]
Now the range is compressed (1.0 to 3.70 instead of 10 to 5000).
Code Example:
import numpy as np
spending = np.array([10, 15, 20, 50, 100, 500, 5000])
# Log transform (natural log)
log_spending = np.log(spending)
print(log_spending)
# [2.30, 2.71, 2.996, 3.91, 4.61, 6.21, 8.52]
Time-Based Features
Extract meaningful features from timestamps.
Why? Time matters. People behave differently at 9 AM vs 9 PM, on weekdays vs weekends.
Common time features:
import pandas as pd
from datetime import datetime
# Sample data with timestamp
events = pd.DataFrame({
'timestamp': pd.to_datetime([
'2024-03-15 09:30:00',
'2024-03-15 22:45:00',
'2024-03-20 14:20:00',
])
})
# Extract features
events['hour'] = events['timestamp'].dt.hour
events['day_of_week'] = events['timestamp'].dt.dayofweek # 0=Monday, 6=Sunday
events['day_of_month'] = events['timestamp'].dt.day
events['is_weekend'] = events['day_of_week'].isin([5, 6]).astype(int)
events['is_business_hours'] = events['hour'].between(9, 17).astype(int)
print(events[['timestamp', 'hour', 'day_of_week', 'is_weekend']])
# timestamp hour day_of_week is_weekend
# 0 2024-03-15 09:30:00 9 4 0
# 1 2024-03-15 22:45:00 22 4 0
# 2 2024-03-20 14:20:00 14 2 0
💡 Tip: "Recency" (how long ago something happened) is a powerful feature in recommendations. A user who viewed a product 1 hour ago is more likely to buy it than someone who viewed it 30 days ago.
Aggregation Features
Compute summaries from multiple rows (count, sum, average, etc.).
Example: User activity features
import pandas as pd
# Raw clicks data
clicks = pd.DataFrame({
'user_id': [1, 1, 1, 2, 2, 3],
'timestamp': pd.date_range('2024-03-01', periods=6),
'product_id': [101, 102, 101, 103, 104, 101]
})
# Aggregate: counts in last 7 days
user_features = clicks.groupby('user_id').agg({
'product_id': 'count', # Total clicks
'timestamp': ['min', 'max'] # First and last click
}).rename(columns={'product_id': 'total_clicks'})
print(user_features)
# total_clicks
# user_id
# 1 3
# 2 2
# 3 1
# More sophisticated: clicks in different time windows
clicks['days_ago'] = (pd.Timestamp.now() - clicks['timestamp']).dt.days
clicks_7d = clicks[clicks['days_ago'] <= 7].groupby('user_id').size()
clicks_30d = clicks[clicks['days_ago'] <= 30].groupby('user_id').size()
print(f"Clicks last 7 days: {clicks_7d.to_dict()}")
print(f"Clicks last 30 days: {clicks_30d.to_dict()}")
📝 Note: Aggregation features are powerful but expensive to compute at serving time. This is where feature stores come in (see below).
Cross Features
Combine two features to create a new signal.
Why? Sometimes the combination matters more than individual features.
Example: Age group + Income level
data = pd.DataFrame({
'age_group': ['young', 'young', 'old', 'old'],
'income_level': ['low', 'high', 'low', 'high']
})
# Create cross feature
data['age_income_combo'] = data['age_group'] + '_' + data['income_level']
print(data['age_income_combo'])
# 0 young_low
# 1 young_high
# 2 old_low
# 3 old_high
# Then one-hot encode if needed
Why this matters:
- Young + low income → probably price-sensitive
- Young + high income → probably willing to pay for premium features
- Old + low income → different needs entirely
The combination tells you more than either feature alone.
Text Features
Turn text into numbers that models can use.
Bag of Words
Count how many times each word appears.
from sklearn.feature_extraction.text import CountVectorizer
documents = [
"I love this product",
"I hate this product",
"This product is amazing"
]
vectorizer = CountVectorizer()
bow = vectorizer.fit_transform(documents)
print(vectorizer.get_feature_names_out())
# ['amazing', 'hate', 'love', 'product', 'this', 'is', ...]
print(bow.toarray())
# [[0 0 1 1 1 0 ...] # "I love this product"
# [0 1 0 1 1 0 ...] # "I hate this product"
# [1 0 0 1 1 1 ...]] # "This product is amazing"
TF-IDF
Like bag of words, but weights common words less.
Why? The word "the" appears everywhere. Words like "product" are more meaningful.
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
tfidf = vectorizer.fit_transform(documents)
print(tfidf.toarray())
# More meaningful: "amazing", "hate", "love" get higher weights
# Less meaningful: "the", "is", "this" get lower weights
Character N-grams
Break text into chunks of N characters.
Example: "hello" becomes:
- Bigrams (2 chars): "he", "el", "ll", "lo"
- Trigrams (3 chars): "hel", "ell", "llo"
Useful for detecting typos, misspellings, or language without understanding grammar.
Feature Stores: What They Are and Why They Matter
A feature store is a database of pre-computed features that your models can use at serving time.
The Problem Without a Feature Store
User clicks on a product at 9:00:01 AM.
Your serving system needs to predict click-through rate:
1. Compute: "clicks in last 7 days"
2. Compute: "average rating of products user clicked"
3. Compute: "time since last purchase"
4. ... 50 more features ...
This takes 500ms! User sees blank page. User leaves.
The Solution With a Feature Store
Every night, precompute all features for all users:
user_1234:
- clicks_7d: 42
- avg_product_rating: 4.2
- days_since_purchase: 3
- ... 50 more features ...
At serving time:
1. Look up user_1234 in feature store
2. Get all features instantly
3. Run ranking model (10ms)
4. Return results
Total latency: 15ms. Page loads instantly. User clicks. Everyone happy.
Why Feature Stores Matter in Production
- Speed: Pre-computed features serve in milliseconds
- Consistency: Same features for training and serving (prevents train-serving skew)
- Reusability: One feature store serves all teams and models
- Monitoring: Easy to track when features change (data drift detection)
🎯 Interviewer's Perspective: Feature stores are increasingly important. Be ready to discuss:
- Why train-serving skew happens without feature stores
- Examples: Airbnb's Zipline, Uber's Michelangelo, Feast (open source)
- Trade-offs: build vs buy, real-time vs batch updates
Why Features Matter More Than Models
There's a popular saying in ML:
"80% of model performance comes from good features, 20% from model choice."
This is actually understating it. Features often matter more.
The Empirical Reality
Let's say you're building a click prediction model. You test two scenarios:
Scenario A: Great features + simple model
- Features: User history, item popularity, temporal patterns, cross-features
- Model: Logistic regression
- Accuracy: 87%
Scenario B: Weak features + fancy model
- Features: Raw item ID, basic bucketing
- Model: Gradient boosted forest with hyperparameter tuning
- Accuracy: 72%
Result: Feature engineering wins by 15 percentage points.
Why Features Matter So Much
Model job: Find patterns in features
Features job: Represent the problem effectively
If features don't capture relevant information,
even the best model can't help.
Example: Predicting house prices
Without feature: [3, 2, 1995] (bedrooms, bathrooms, year)
Better features: [3, 2, 1995, 0.8, 5.2]
(bedrooms, bathrooms, year, distance_to_school, safety_score)
Model can't invent that information from raw features!
The 80/20 Rule in Practice
ML project time allocation (typical):
Data cleaning/preparation: 40%
Feature engineering: 30%
Model selection/tuning: 20%
Deployment/monitoring: 10%
Most time goes to data, not models!
🎯 Interviewer's Perspective: When asked "What's your first step in an ML project?", a great answer is "Exploratory Data Analysis and feature engineering", not "What model should I use?". It shows you understand where ML value comes from.
Feature Interactions
What Are Feature Interactions?
An interaction is when two features together tell you more than each one alone.
Simple example: Age + Income
Age = 25, Income = $30k → Model predicts "price-sensitive"
Age = 65, Income = $30k → Model predicts "value-conscious retiree"
Age = 25, Income = $500k → Model predicts "premium buyer"
Age = 65, Income = $500k → Model predicts "wealthy"
The combination (age_income_pair) matters more than individual features!
Why Interactions Capture Non-linear Relationships
Most real-world relationships are non-linear. Interactions let simpler models capture them.
Example: E-commerce pricing
Without interaction:
price_sensitivity = 0.5 * age_normalized + 0.3 * income_normalized
(linear: each feature contributes equally)
With interaction:
price_sensitivity = 0.5 * age + 0.3 * income + 0.2 * (age * income)
(captures non-linear: youth AND low-income = extra price sensitivity)
How to Create Effective Interactions
Method 1: Domain knowledge (manual)
# Retail example
data['seasonal_popularity'] = data['time_of_year'] * data['product_category']
data['user_item_affinity'] = data['user_preference'] * data['item_quality']
data['recency_engagement'] = data['days_since_interaction'] * data['engagement_rate']
# These capture: popular items CHANGE based on season
Method 2: Polynomial features
from sklearn.preprocessing import PolynomialFeatures
# Create all 2-way interactions
poly = PolynomialFeatures(degree=2, include_bias=False)
original_features = [[5, 10], [3, 8], [7, 12]] # age, income
interactions = poly.fit_transform(original_features)
# Output: [[5, 10, 25, 50, 100], # x1, x2, x1^2, x1*x2, x2^2
# [3, 8, 9, 24, 64],
# [7, 12, 49, 84, 144]]
# Now has: age, income, age^2, age*income, income^2
Watch out: With 100 features, degree-2 interactions create 5,000 new features (curse of dimensionality).
Method 3: Tree-based models learn interactions automatically
# Random Forests and XGBoost learn interactions implicitly
# Example tree split:
if age > 30:
if income > 100k:
prediction = premium_buyer
else:
prediction = mid_segment
else:
if income > 100k:
prediction = young_high_earner
else:
prediction = price_sensitive
# This tree learned: age AND income matter together!
💡 Tip: If using logistic regression, create interactions manually. If using trees, they learn automatically. This is one reason trees often outperform linear models on real data.
Feature Selection Theory
With thousands of possible features, which ones should you use?
Mutual Information
Mutual information (MI) measures how much knowing feature X helps predict target Y.
Formula (simplified):
MI(X, Y) = Sum over all x,y: P(X=x, Y=y) * log(P(X=x, Y=y) / (P(X=x)*P(Y=y)))
Interpretation:
- MI = 0: Feature tells us nothing about target
- MI = high: Feature strongly predicts target
Numerical example:
Target: Click (Yes/No)
Feature: "User previously clicked"
- 80% of clickers had clicked before
- 10% of non-clickers had clicked before
→ Mutual information = 0.45 (high, useful feature)
Feature: "Random number 1-100"
- 50% of clickers see any number
- 50% of non-clickers see any number
→ Mutual information = 0.0 (useless feature)
Feature: "User country"
- 40% of clickers from US
- 35% of non-clickers from US
→ Mutual information = 0.02 (weak signal)
Code example:
from sklearn.feature_selection import mutual_info_classif
import pandas as pd
X = df[['feature_1', 'feature_2', 'feature_3', ...]]
y = df['target']
# Calculate MI for each feature
mi_scores = mutual_info_classif(X, y, random_state=42)
# Create ranking
feature_importance = pd.DataFrame({
'feature': X.columns,
'mi_score': mi_scores
}).sort_values('mi_score', ascending=False)
print(feature_importance)
# feature mi_score
# 0 user_purchase_history 0.48
# 1 user_engagement 0.42
# 2 item_category 0.15
# 3 time_of_day 0.08
# 4 random_id 0.00
Correlation Analysis
For numeric features, correlation is simpler but less sophisticated.
import pandas as pd
correlation_with_target = df.corr()['target'].abs().sort_values(ascending=False)
# Shows: which features move together with target
# -1.0: Perfect negative correlation (as X increases, Y decreases)
# 0.0: No correlation
# 1.0: Perfect positive correlation
# Example output:
# user_spend_last_30d 0.65
# user_engagement_rate 0.58
# product_reviews_count 0.42
# product_category 0.08
# user_id_hash 0.01
Limitation: Correlation assumes linear relationship.
Feature X: [1, 2, 3, 4, 5]
Target Y: [1, 4, 9, 16, 25] (Y = X²)
Correlation = 0.95 (looks good!)
But relationship is non-linear (quadratic)
MI would capture this better
Feature Importance from Tree Models
Tree-based models provide built-in feature importance.
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train)
# Get feature importance
importance = pd.DataFrame({
'feature': X_train.columns,
'importance': model.feature_importances_
}).sort_values('importance', ascending=False)
print(importance)
# feature importance
# 0 user_purchase_history 0.35
# 1 user_engagement 0.28
# 2 item_category 0.18
# 3 time_of_day 0.12
# 4 random_id 0.07
# Drop low-importance features
keep_features = importance[importance['importance'] > 0.05]['feature'].tolist()
X_train_selected = X_train[keep_features]
How it works: More important features are used in earlier tree splits (split at top of tree = more impact).
Feature Scaling: When and Why
The Problem Without Scaling
Imagine: age (0-100) and income ($0-
M)data = {
'age': [25, 45, 65],
'income': [50000, 120000, 200000]
}
# Raw model input might compute:
distance = sqrt((25-45)² + (50000-120000)²)
= sqrt(400 + 4900000000)
≈ 70000 (income dominates completely!)
# The model is influenced 99.9% by income, almost ignoring age
# This is bad: age might be just as important!
Solution 1: Standardization (Z-score normalization)
Transform to mean=0, std=1:
X_standardized = (X - mean(X)) / std(X)
Example:
age: mean=45, std=20
age[25] → (25-45)/20 = -1.0
age[65] → (65-45)/20 = +1.0
income: mean=123333, std=75000
income[50k] → (50000-123333)/75000 = -0.978
income[200k] → (200000-123333)/75000 = +1.022
Now both are on same scale!
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print(X_scaled)
# age income
# 0 -1.0 -0.978
# 1 0.0 -0.031
# 2 1.0 +1.022
Solution 2: Normalization (Min-Max)
Transform to 0-1 range:
X_normalized = (X - min(X)) / (max(X) - min(X))
Example:
age: min=25, max=65, range=40
age[25] → (25-25)/40 = 0.0
age[65] → (65-25)/40 = 1.0
income: min=50k, max=200k, range=150k
income[50k] → (50000-50000)/150000 = 0.0
income[200k] → (200000-50000)/150000 = 1.0
Same scale again!
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
X_normalized = scaler.fit_transform(X)
print(X_normalized)
# age income
# 0 0.0 0.0
# 1 0.5 0.467
# 2 1.0 1.0
When Each Model Needs Scaling
| Model | Needs Scaling? | Why |
|---|---|---|
| Logistic Regression | Yes | Distance-based weights |
| Linear Regression | Yes | Feature scale affects weights |
| Neural Networks | Yes | Gradient descent converges faster with normalized inputs |
| SVM | Yes | Distance-based algorithm |
| K-Means | Yes | Distance-based clustering |
| KNN | Yes | Depends on distance metric |
| Decision Trees | No | Splits don't depend on scale |
| Random Forest | No | Each tree is scale-invariant |
| Gradient Boosting (XGBoost) | No | Tree-based, scale-invariant |
| Naive Bayes | No | Probability-based |
Rule of thumb: If distance matters (NN, SVM, KNN), scale. If splits matter (trees), don't scale.
💡 Tip: Always scale your training and test data. Fit scaler on training data, then apply to test data (never fit on test data - that's data leakage).
Feature Stores: Architecture Deep Dive
Online vs Offline Features
A feature store is a database that serves precomputed features to ML models.
Offline Features (Batch)
Precomputed daily/hourly, stored in data warehouse.
Example:
User 1234:
- total_purchases_30d: 5
- avg_order_value: 25
- days_since_last_purchase: 2
- preferred_category: electronics
- churn_risk_score: 0.15
Computed every night, stored in DynamoDB or similar
Pros:
- Very fast at serving time (just lookup)
- Can use expensive computations
- Batch consistency (all models use same values)
Cons:
- Stale data (yesterday's value, not real-time)
- Not suitable for highly dynamic features
Online Features (Real-time)
Computed on-demand when model needs it.
Example:
Real-time feature request:
Input: user_id=1234, item_id=5678
System computes:
- user_recency_score (computed now, based on latest clicks)
- item_trending_score (computed now, based on latest views)
- user_item_interaction_history (computed now)
Returns features immediately for model serving
Pros:
- Real-time, always fresh
- Captures latest user behavior
Cons:
- Slower (adds latency)
- Must be cheap to compute (limited time budget)
- Harder to debug (values change constantly)
Point-in-Time Correctness
The biggest challenge: features must represent the state at training time, not current state.
Example of data leakage without point-in-time correctness:
Training data:
User clicked on "Laptop" on 2024-03-15 at 10:00 AM
At 10:01 AM, compute features for this user:
- recent_clicks: 1 (just saw 1 item)
- avg_price_of_viewed: 200
Later, at 2024-03-20, compute same features:
- recent_clicks: 10 (saw 10 more items)
- avg_price_of_viewed: $800
Now training and serving data are different!
Model learned on fresh features, but serves on aged features.
Solution: Time travel
Feature store must track timestamp for each feature value:
User 1234, Feature: "total_purchases_7d"
timestamp=2024-03-20 08:00: value=5
timestamp=2024-03-20 09:00: value=6 (another purchase)
timestamp=2024-03-20 10:00: value=6
timestamp=2024-03-21 08:00: value=7
Training: If label is from 2024-03-20 08:30, use value=5
Serving: Use current value=7 (or closest to request time)
Feature Store Architecture
┌─────────────────────────────────────────────────────┐
│ Feature Store (e.g., Feast, Tecton, Offline) │
├─────────────────────────────────────────────────────┤
│ │
│ Offline Store (Data Warehouse) Online Store │
│ ┌─────────────────────────┐ (Redis, etc) │
│ │ Historical features │ ┌───────────┐ │
│ │ - user_purchase_history │ │ Fresh │ │
│ │ - user_engagement_stats │ →→→ │ values │ │
│ │ - item_popularity │ │ (cached) │ │
│ └─────────────────────────┘ └───────────┘ │
│ ↑ ↑ │
│ Nightly batch Real-time updates from │
│ computation source systems │
│ │
└─────────────────────────────────────────────────────┘
↓ ↓
Training pipeline Model serving
(historical data) (live requests)
Why Feature Stores Matter
- Consistency: Training and serving use same features
- Speed: Pre-computed features are instant lookup
- Reuse: One feature store serves all teams/models
- Monitoring: Easy to detect feature data drift
- Governance: Clear definition of what each feature is
Real-world examples:
- Uber Michelangelo: Feature store for ride prediction
- Airbnb Zipline: Feature store for search ranking
- Feast (Open Source): Lightweight feature store
🎯 Interviewer's Perspective: Feature stores are increasingly important at scale. Be ready to discuss train-serving skew, point-in-time correctness, and why they're non-trivial to build correctly.
Key Takeaways
Feature engineering determines ML success more than model choice. Common patterns:
| Pattern | Use Case | Example |
|---|---|---|
| Bucketing | Compress continuous values | Age 0-100 → age buckets |
| One-hot encoding | Convert categories to binary | Color → is_red, is_blue, is_green |
| Normalization | Scale to common range | Age and income to 0-1 scale |
| Log transform | Compress skewed ranges | Spending 0-00,000 → log scale |
| Time features | Extract day, hour, etc. | Timestamp → hour, day_of_week |
| Aggregation | Summarize multiple rows | Click count, average rating |
| Cross features | Combine for interaction | Age + income → behavior signal |
| Text features | Turn words into numbers | TF-IDF, embeddings |