Free lesson · ML System Design Interviews
Model Development
Two-Stage Architecture
Like a funnel, ranking systems typically use two stages:
- Candidate Generation (Retrieval): 500M posts → 500 candidates
- Ranking (Scoring): 500 candidates → 50 final posts
Why? Ranking is expensive; we can't score all 500M posts per user per request.
500M Posts
|
|-- Apply filters, index lookups (fast) -----|
| |
v v
[Candidate Generation] [Feature Store]
(10-100ms) (cached features)
|
v
~500 Candidates
|
|-- Full feature set, complex model -----|
| |
v v
[Ranking Model] [Neural Network]
(10-100ms) (expensive scoring)
|
v
~50 Ranked Posts + Scores
|
v
[Diversity & Filtering]
(deduplicate, balance)
|
v
[Feed to User]
Stage 1: Candidate Generation
Strategy: Multi-Source Retrieval
Pull posts from multiple sources, deduplicate:
┌─────────────────────────────────────────┐
│ CANDIDATE GENERATION (Retrieval) │
├─────────────────────────────────────────┤
│ │
│ [1] Posts from followed users │
│ → Quick lookup in user's follow │
│ graph. 50-100 posts. │
│ │
│ [2] Trending posts (today, this hour) │
│ → From pre-computed trending list. │
│ 50-100 posts. │
│ │
│ [3] Content-based (interests) │
│ → Match user interests to post │
│ topics via index (Elasticsearch). │
│ 100-200 posts. │
│ │
│ [4] Collaborative filtering │
│ → "Users like you liked this." │
│ 50-100 posts. │
│ │
│ [Deduplication] │
│ Remove overlaps → ~500 candidates │
│ │
└─────────────────────────────────────────┘
Why This Works
- Diversity: Multiple sources mean varied content
- Speed: Each source uses pre-computed indices or caches
- Coverage: New posts come from trending, follow-based posts; niche content from interest-based
Pseudocode
def get_candidates(user_id, limit=500):
"""Retrieve ~500 candidate posts for ranking."""
candidates = set()
# Source 1: User's network
followed_users = get_user_follows(user_id)
network_posts = fetch_recent_posts(
author_ids=followed_users,
hours_back=24,
limit=100
)
candidates.update(network_posts)
# Source 2: Trending
trending_posts = fetch_trending_posts(
hours_back=3,
limit=100
)
candidates.update(trending_posts)
# Source 3: Interest-based (Elasticsearch)
user_interests = get_user_interests(user_id)
interest_posts = search_posts_by_topic(
topics=user_interests,
hours_back=24,
limit=200
)
candidates.update(interest_posts)
# Source 4: Collaborative filtering
similar_users = find_similar_users(user_id, k=10)
collab_posts = fetch_recent_posts(
author_ids=similar_users,
hours_back=24,
limit=100
)
candidates.update(collab_posts)
# Deduplicate and limit
final_candidates = list(candidates)[:limit]
return final_candidates
# Usage
candidates = get_candidates(user_id='user_123')
print(f"Retrieved {len(candidates)} candidates") # ~500
Stage 2: Ranking
Rank the 500 candidates by engagement probability. Here's a progression from simple to complex.
Model 1: Logistic Regression (Baseline)
What it is: Linear model mapping features to probability.
Pros:
- Interpretable (see feature weights)
- Fast (< 1ms per prediction)
- Good baseline
Cons:
- Can't capture feature interactions
- Limited expressiveness
Formula:
P(engagement) = sigmoid(w₀ + Σ wᵢ * featureᵢ)
sigmoid(x) = 1 / (1 + e^(-x))
Code:
from sklearn.linear_model import LogisticRegression
import numpy as np
# Train
X_train = np.array(training_features) # shape: (N_samples, N_features)
y_train = np.array(training_labels) # shape: (N_samples,), values: 0 or 1
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
# Predict
X_test = np.array(test_features)
engagement_probs = model.predict_proba(X_test)[:, 1] # P(engagement)
# Ranking
ranked_posts = sorted(zip(post_ids, engagement_probs), key=lambda x: x[1], reverse=True)
Baseline AUC: ~0.65 (not great, but a starting point)
Model 2: Gradient Boosted Trees (XGBoost)
What it is: Ensemble of decision trees, each correcting previous errors.
Pros:
- Captures non-linear patterns and feature interactions
- Fast inference (~10ms per batch)
- Robust to outliers
- Feature importance (which features matter?)
Cons:
- Less interpretable than linear model
- Slower to train
Why It Matters: XGBoost can learn interactions like:
- "Young users interested in tech engage more with AI posts"
- "Posts with images from verified authors get higher engagement"
These require the model to combine multiple features, which XGBoost does naturally.
Code:
import xgboost as xgb
# Train
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)
params = {
'objective': 'binary:logistic',
'max_depth': 7,
'learning_rate': 0.1,
'n_estimators': 100,
'eval_metric': 'auc'
}
model = xgb.train(params, dtrain, num_boost_round=100)
# Predict
engagement_probs = model.predict(dtest)
# Feature importance
importance = model.get_score(importance_type='weight')
print(f"Top features: {sorted(importance.items(), key=lambda x: x[1], reverse=True)[:5]}")
# Output: [('topic_overlap', 1200), ('user_engagement_rate', 950), ...]
Improved AUC: ~0.72
Model 3: Deep Learning (Neural Network)
What it is: Multiple layers of neurons learning complex patterns.
Pros:
- Handles raw embeddings (text, user embeddings) naturally
- Captures very complex interactions
- State-of-the-art performance
Cons:
- Slower training
- Slower inference (~50-100ms per batch)
- Requires careful tuning
- Less interpretable
Why It Matters: Post embeddings (768-dim vectors from a language model) are high-dimensional. A neural net can use them directly; XGBoost struggles with very high dimensions.
Architecture:
Input Layer (50 features + 768-dim embeddings)
↓
Dense(256, ReLU)
↓
Dropout(0.2)
↓
Dense(128, ReLU)
↓
Dropout(0.2)
↓
Dense(64, ReLU)
↓
Output(1, Sigmoid) → P(engagement)
Code:
import tensorflow as tf
from tensorflow import keras
# Build model
model = keras.Sequential([
keras.layers.Dense(256, activation='relu', input_shape=(X_train.shape[1],)),
keras.layers.Dropout(0.2),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dropout(0.2),
keras.layers.Dense(64, activation='relu'),
keras.layers.Dense(1, activation='sigmoid')
])
# Compile
model.compile(
optimizer='adam',
loss='binary_crossentropy',
metrics=['auc']
)
# Train
history = model.fit(
X_train, y_train,
epochs=10,
batch_size=32,
validation_split=0.2
)
# Predict
engagement_probs = model.predict(X_test)[:, 0]
Improved AUC: ~0.75
Theory: Multi-Task Learning
What Is Multi-Task Learning?
In practice, engagement isn't one-dimensional. A user might click a post (lightweight engagement), like it (medium), or comment/share (heavy engagement). These signals are correlated but distinct.
Multi-task learning trains a single neural network to predict multiple engagement types simultaneously, with a shared feature extraction layer and task-specific output heads.
Why It Beats Individual Models
Naive approach: Train four separate models (click, like, comment, share).
Problem: Redundant computation, less data per model, no knowledge transfer.
Better approach: One backbone learns shared patterns (e.g., "posts with images are engaging"), then task-specific heads learn distinctions (e.g., "images boost clicks more; text boosts comments more").
# Shared-bottom multi-task architecture
class MultiTaskModel(nn.Module):
def __init__(self, feature_dim=50, shared_dim=128):
super().__init__()
# Shared base: processes features once
self.shared_layers = nn.Sequential(
nn.Linear(feature_dim, shared_dim),
nn.ReLU(),
nn.Dropout(0.2),
nn.Linear(shared_dim, 64),
nn.ReLU()
)
# Task-specific heads
self.click_head = nn.Linear(64, 1)
self.like_head = nn.Linear(64, 1)
self.comment_head = nn.Linear(64, 1)
self.share_head = nn.Linear(64, 1)
def forward(self, features):
# Shared representation
shared = self.shared_layers(features)
# Task-specific predictions
click_logits = self.click_head(shared)
like_logits = self.like_head(shared)
comment_logits = self.comment_head(shared)
share_logits = self.share_head(shared)
return {
'click': torch.sigmoid(click_logits),
'like': torch.sigmoid(like_logits),
'comment': torch.sigmoid(comment_logits),
'share': torch.sigmoid(share_logits)
}
# Training
model = MultiTaskModel()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for features, labels in train_loader:
outputs = model(features)
# Multi-task loss: weighted sum
loss = (
1.0 * F.binary_cross_entropy(outputs['click'], labels['click']) +
1.5 * F.binary_cross_entropy(outputs['like'], labels['like']) +
3.0 * F.binary_cross_entropy(outputs['comment'], labels['comment']) +
5.0 * F.binary_cross_entropy(outputs['share'], labels['share'])
)
loss.backward()
optimizer.step()
Why This Works Better
- Shared representation: Early layers learn universal patterns (colors, text length, recency) once, benefiting all tasks
- Regularization: Predicting multiple targets prevents overfitting to noise in single target
- Data efficiency: With limited engagement data, sharing helps each task learn from related signals
- Business alignment: Output captures nuance—ranking can prioritize different signals for different users
Key insight: If you predict only clicks, you're ignoring that comments (showing deeper interest) are actually valuable. Multi-task learning captures this trade-off.
Theory: Position Bias in Ranking
The Problem: Visibility ≠ Quality
Posts ranked higher in a feed get more impressions. More impressions naturally lead to more clicks. But higher ranking ≠ higher relevance.
Real scenario:
- Post A shown at position 1: 100 impressions, 8 clicks → 8% CTR
- Post B shown at position 50: 10 impressions, 1 click → 10% CTR
Your model sees Post A as better (more clicks). But Post B actually has higher intrinsic engagement—it's just buried.
Consequence: Trained model learns to rank highly-visible posts higher, perpetuating the bias.
Inverse Propensity Scoring (IPS)
The solution: weight training examples inversely to their visibility.
def compute_position_weights(positions):
"""
Posts shown at top positions are over-represented.
Downweight them during training.
Formula: weight = 1 / P(position)
where P(position) = probability of being shown at that position
"""
# Empirically, position 1 shown 80% of time, position 10 shown 20%
position_propensity = {
1: 0.80,
2: 0.60,
3: 0.40,
4: 0.25,
5: 0.15,
10: 0.05
}
weights = []
for pos in positions:
propensity = position_propensity.get(pos, 0.01)
weight = 1.0 / propensity # Inverse weighting
weights.append(weight)
return np.array(weights)
# Example
positions = [1, 1, 1, 10, 10] # Three posts at top, two at bottom
weights = compute_position_weights(positions)
# [1.25, 1.25, 1.25, 20.0, 20.0]
# During training:
weighted_loss = weights * cross_entropy_loss
# Bottom-position posts heavily influence training, correcting bias
Why IPS Works
By upweighting rare observations (bottom positions) and downweighting common ones (top positions), the model learns from the true distribution of user preferences, not the visible distribution.
Practical impact: After IPS, a less-visible but better post gets ranked higher, improving feed quality.
Callout: Position bias affects all ranking systems (search, e-commerce, recommendations). Always measure and correct it.
Feature Interactions: Why They Matter
Example 1: Content Type × Device
A video post on mobile gets more engagement than on desktop (smaller screen, more captive). But a code snippet post gets more engagement on desktop.
Logistic regression treats these separately:
P = sigmoid(w_video * is_video + w_mobile * is_mobile + ...)
This says: is_video contributes the same regardless of device.
XGBoost & Neural Networks can learn:
if is_video and is_mobile: +bonus
if is_code and is_desktop: +bonus
Example 2: User Interests × Author
A user interested in AI will engage with AI posts more, but especially from AI researchers vs. random accounts. This is an interaction between topic_overlap and author_authority.
How to capture:
- Explicitly create cross-features (XGBoost):
ai_interest_and_ai_author_feature = user_interested_in_ai * author_has_ai_posts - Or let deep learning learn implicitly via hidden layers
Multi-Objective Optimization
The Problem
We want to optimize for multiple engagement types:
- Clicks (lightweight)
- Likes (light engagement)
- Comments (heavy engagement)
- Shares (very heavy engagement)
- Time spent (implicit)
They trade off. A sensational post gets clicks but might not get comments. A thoughtful post might get fewer clicks but more comments.
Solution 1: Composite Objective
Combine into a single score:
engagement_score = 1.0 * clicks + 1.5 * likes + 3.0 * comments + 5.0 * shares
Then optimize P(engagement_score > 0).
Pro: Simple. Con: Weights are arbitrary.
Solution 2: Multi-Task Learning
Train separate models for each signal, then combine at serving:
Post Features
├─→ [Click Model] → P(click)
├─→ [Like Model] → P(like)
├─→ [Comment Model] → P(comment)
└─→ [Share Model] → P(share)
Final score = 1.0*P(click) + 1.5*P(like) + 3*P(comment) + 5*P(share)
Pro: Each model specializes. Con: More training, slower.
Solution 3: Meta-Learning
Train a model to predict (P_click, P_like, P_comment, P_share) with a shared base layer and task-specific heads.
Code sketch (PyTorch):
class MultiTaskRankingModel(nn.Module):
def __init__(self):
super().__init__()
# Shared feature processing
self.shared_layer = nn.Linear(50, 128)
self.relu = nn.ReLU()
# Task-specific heads
self.click_head = nn.Linear(128, 1)
self.like_head = nn.Linear(128, 1)
self.comment_head = nn.Linear(128, 1)
self.share_head = nn.Linear(128, 1)
def forward(self, features):
# Shared representation
shared = self.relu(self.shared_layer(features))
# Task-specific predictions
p_click = torch.sigmoid(self.click_head(shared))
p_like = torch.sigmoid(self.like_head(shared))
p_comment = torch.sigmoid(self.comment_head(shared))
p_share = torch.sigmoid(self.share_head(shared))
return p_click, p_like, p_comment, p_share
# Training
model = MultiTaskRankingModel()
optimizer = torch.optim.Adam(model.parameters())
for epoch in range(10):
for features, labels in train_loader:
p_click, p_like, p_comment, p_share = model(features)
# Loss = weighted sum of task-specific losses
loss = (
bce_loss(p_click, labels[:, 0]) +
1.5 * bce_loss(p_like, labels[:, 1]) +
3.0 * bce_loss(p_comment, labels[:, 2]) +
5.0 * bce_loss(p_share, labels[:, 3])
)
loss.backward()
optimizer.step()
Training Pipeline
End-to-End Flow
Day 1: Impressions logged
↓
Day 2: Labeling (24-hour window closed)
↓
Day 3: Feature engineering
↓
Day 4: Training
↓
Day 5: Evaluation & Shadow testing
↓
Day 6: Deploy to 5% of users (A/B test)
↓
Day 7: Full rollout (if good results)
Code: Training Loop
import logging
logger = logging.getLogger(__name__)
def train_ranking_model(
training_data_path,
model_type='xgboost',
hyperparams=None,
output_model_path='./model.pkl'
):
"""Train engagement ranking model."""
# Load data
logger.info(f"Loading training data from {training_data_path}")
data = pd.read_csv(training_data_path)
X = data.drop('engagement_label', axis=1)
y = data['engagement_label']
logger.info(f"Data shape: {X.shape}, Label distribution: {y.value_counts()}")
# Train/val split (temporal)
cutoff_idx = int(len(X) * 0.8)
X_train, X_val = X[:cutoff_idx], X[cutoff_idx:]
y_train, y_val = y[:cutoff_idx], y[cutoff_idx:]
# Initialize model
if model_type == 'xgboost':
model = xgb.XGBClassifier(
max_depth=hyperparams.get('max_depth', 7),
learning_rate=hyperparams.get('lr', 0.1),
n_estimators=hyperparams.get('n_trees', 100),
eval_metric='auc',
random_state=42
)
elif model_type == 'neural_net':
model = build_neural_net(input_dim=X_train.shape[1])
# Train
logger.info("Training...")
if model_type == 'xgboost':
model.fit(
X_train, y_train,
eval_set=[(X_val, y_val)],
verbose=False
)
else:
model.fit(X_train, y_train, epochs=10, validation_data=(X_val, y_val), verbose=0)
# Evaluate
train_auc = roc_auc_score(y_train, model.predict_proba(X_train)[:, 1])
val_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
logger.info(f"Train AUC: {train_auc:.4f}, Val AUC: {val_auc:.4f}")
# Save
import pickle
with open(output_model_path, 'wb') as f:
pickle.dump(model, f)
logger.info(f"Model saved to {output_model_path}")
return model, train_auc, val_auc
# Usage
model, train_auc, val_auc = train_ranking_model(
'data/impressions_2024_03.csv',
model_type='xgboost',
hyperparams={'max_depth': 7, 'lr': 0.1, 'n_trees': 100},
output_model_path='./models/ranking_v2.pkl'
)
Handling Feedback Loops & Cold Start
Cold Start: New Posts
Problem: New posts have no engagement signals yet. Model can't score them fairly.
Solution: Use authority boosting.
def score_post(post_id, user_id, model):
"""Score a post for a user."""
features = get_features(post_id, user_id)
base_score = model.predict_proba(features)[1]
# Boost if post is new (encourage exploration)
if post_age_hours < 1:
boost = 0.1 # Add 10% for very new posts
return base_score + boost
elif post_age_hours < 6:
boost = 0.05
return base_score + boost
else:
return base_score
Cold Start: New Users
Problem: New users have no interaction history.
Solution: Serve trending content + ask for interests upfront.
def get_initial_feed(user_id):
"""For brand-new users, serve trending content."""
if get_user_interaction_count(user_id) < 5:
# New user: serve trending posts
posts = fetch_trending_posts(limit=20)
return posts
else:
# Regular user: serve personalized feed
candidates = get_candidates(user_id)
return rank_and_return(candidates, user_id)
Feedback Loop: Popularity Bias
Problem: Popular posts get promoted, get more engagement, get promoted more.
Solution: Randomized exploration during serving.
def rank_with_exploration(candidates, user_id, model, exploration_rate=0.05):
"""Rank with some randomness to test less-obvious posts."""
scores = [model.predict(get_features(post_id, user_id)) for post_id in candidates]
# Sort by score
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
# Randomly shuffle bottom 10% (exploration)
split_idx = int(len(ranked) * (1 - exploration_rate))
top_posts = ranked[:split_idx]
explore_posts = ranked[split_idx:]
random.shuffle(explore_posts)
return [p for p, _ in top_posts + explore_posts]
Summary Callouts
💡 Tip: Start simple (logistic regression), measure baseline, then improve. Don't jump to neural networks without understanding the problem.
📝 Note: XGBoost is the sweet spot for production ranking: fast, interpretable feature importance, captures interactions naturally.
⚠️ Watch out: Inference latency grows with model complexity. A 100ms model might be too slow for 100k QPS. Profile before deploying.
🎯 Interviewer's Perspective: Show you understand the trade-offs between simplicity and expressiveness. Mention feature interactions—that's a sophisticated observation.
✅ Decisions made:
- Two-stage architecture: candidate generation + ranking
- Primary model: XGBoost (good balance)
- Multi-objective: weighted combination (click, like, comment, share)
- Cold start: authority boosting + trending content
- Feedback loop: exploration rate during serving