Free lesson · ML System Design Interviews

ML Framing & Data

Medium10 min readmachine-learningml-system-designinterview-prep

From Business Goal to ML Problem

Business goal: "Increase user engagement by showing relevant posts"

ML problem: "Predict the probability that a user will engage with a post"

This translation is crucial. We're turning a vague business goal into a concrete ML task: binary or multi-class classification.

ML Objective

Definition

For each (user, post) pair, predict a score representing engagement likelihood:

engagement_score = P(user engages with post)

Engagement = any of: click, like, comment, share, time spent > X seconds.

Why Probability?

A probability score is flexible:

• Ranking: Sort by score (highest first)
• Thresholding: Show only if P > 0.5
• Allocation: Use score to decide post placement
• A/B testing: Compare scores between models

Theory: Implicit vs Explicit Feedback

The Feedback Problem

Building ranking systems requires knowing what users prefer. There are two sources:

1. Explicit feedback: User actively rates (thumbs up, stars, "I like this")
2. Implicit feedback: User's behavior reveals preference (click, time spent, share)

Why Implicit Is Better (But Noisier)

Explicit advantages:

• Clear signal (5-star rating means something definite)
• No interpretation needed

Explicit disadvantages:

• Rare: Only ~0.1% of users actually rate content
• Sparse data: Not enough signal to train good models
• Biased: People who rate tend to have strong opinions (extreme ratings)

Implicit advantages:

• Abundant: Every impression generates a signal (click or no-click)
• Natural: Users don't have to do extra work
• Realistic: Captures actual behavior

Implicit disadvantages:

• Noisy: Click doesn't always mean "good"—might be accidental, clickbait, or morbid curiosity
• Exposure bias: We only see clicks on posts we showed; posts we didn't show might have been even better

Example: Exposure Bias

Suppose your model shows Post A (predicted 70% engagement). User clicks it (100% observed engagement). Model updates: "Great! Post A is very good."

But what about Post B (predicted 30% engagement)? You never showed it, so you never learned its true engagement rate. It might actually be 95% but you'll never know because your model thought it was bad.

Solution: Negative Sampling

When a user doesn't click Post C, it's ambiguous:

• Post C was bad (negative feedback), OR
• We never showed Post C to user (no data)

To reduce this ambiguity, use negative sampling: for each clicked post, sample N non-clicked posts as negatives. This balances the dataset.

def create_training_pairs(clicked_posts, all_posts, neg_samples=4):
    """
    For each clicked post, create training examples.
    """
    training_data = []

    for clicked_post in clicked_posts:
        # Positive example
        training_data.append({
            'post_id': clicked_post,
            'user_id': user_id,
            'engagement': 1
        })

        # Negative examples (random posts the user didn't engage with)
        negative_posts = random.sample(all_posts, k=neg_samples)
        for neg_post in negative_posts:
            training_data.append({
                'post_id': neg_post,
                'user_id': user_id,
                'engagement': 0
            })

    return training_data

# Example
user_clicked = [post_1, post_5, post_17]  # 3 posts user engaged with
all_posts = list(range(1, 1001))  # 1000 posts total

# Create training data: 3 positives + 3*4=12 negatives = 15 examples per user
training = create_training_pairs(user_clicked, all_posts, neg_samples=4)

Hard Negative Sampling

Even better: don't sample negatives uniformly. Sample hard negatives—posts your model thinks are good but user ignored.

def hard_negative_sampling(user_id, clicked_posts, model, all_posts, k=4):
    """
    Sample negatives that the model thinks the user will like,
    but user actually ignored (these are informative).
    """
    # Get model's score for all posts
    scores = model.predict(user_id, all_posts)

    # Exclude posts user already clicked
    candidate_negatives = [
        (post_id, score) for post_id, score in zip(all_posts, scores)
        if post_id not in clicked_posts
    ]

    # Sort by score (descending) and pick top-k
    # These are posts the model thinks user will like, but user didn't engage
    hard_negatives = sorted(candidate_negatives, key=lambda x: x[1], reverse=True)[:k]

    return [post_id for post_id, _ in hard_negatives]

# Example
user_id = 123
clicked = [1, 5]
negatives = hard_negative_sampling(user_id, clicked, model, all_posts=list(range(1,1001)))
# Returns posts the model predicted highly (e.g., [42, 103, 215, 67])
# but user actually ignored—these teach the model its mistakes

Key principle: Implicit feedback is noisier but abundant. Use negative sampling and prioritize hard negatives to learn effectively from imperfect signals.

Engagement Signal (Target Variable)

Types of Engagement

Combining Signals

Option 1: Single composite label

engagement = (clicks >= 1) OR (likes >= 1) OR (comments >= 1) OR ...

Pro: Simple. Con: Loses nuance.

Option 2: Weighted combination

engagement_score = 1*clicks + 1.5*likes + 3*comments + 5*shares
binary_label = engagement_score > 0.5

Pro: Captures importance. Con: Arbitrary weights.

Option 3: Multi-task learning (later) Predict each signal separately, combine at serving time.

For MVP: Use Option 1 (any engagement within 24 hours). Simple, effective.

Input Features

User Features (Static + Dynamic)

Derivation: From user profile + interaction logs.

Post Features (Static)

Derivation: From post metadata + NLP models.

Context Features (At Request Time)

Why context matters: A user might engage with memes at 10pm on mobile but news articles at 9am on desktop.

Interaction Features (Relationship)

Data Sources

1. User Interaction Logs

• Timestamp, user ID, post ID, action (click, like, comment, share)
• Collected when user interacts with feed
• ~1B events/day

2. User Profile Data

• Demographics, interests, follow lists, account creation date
• Updated when user edits profile
• ~10M user records

3. Post Metadata

• Author, timestamp, content, hashtags, media info
• Updated when post is created/edited
• ~500M posts/day

4. Impression Logs

• User ID, post ID, timestamp, position in feed, did user engage
• Critical for training; shows what was presented vs. what was engaged
• ~10B impressions/day (not all lead to engagement)

5. User Embeddings

• Dense vector representation of user interests
• From clustering user interests or neural networks
• Pre-computed in batch

6. Post Embeddings

• Dense vector of post content (from NLP model)
• Pre-computed in batch
• Updated for new posts

Label Definition

Positive Label

User engaged with post within 24 hours of impression.

def is_engagement(impression_log):
    """
    impression: (user_id, post_id, timestamp_impression)
    interaction: (user_id, post_id, action, timestamp_action)
    """
    matched_interactions = query_interactions(
        user_id=impression.user_id,
        post_id=impression.post_id,
        time_range=(impression.timestamp, impression.timestamp + 24*3600)
    )
    return len(matched_interactions) > 0

Negative Label

User did not engage within 24 hours.

Data Leakage Risk

Only use post/user features from before impression. Don't use future engagement patterns.

Data Challenges & Solutions

Challenge 1: Position Bias

Posts ranked higher are clicked more, but not necessarily because they're better—just because they're visible.

Example:

• Post A shown at position 1: 10% CTR
• Post B shown at position 5: 5% CTR
• But if swapped, Post B at position 1 might have 8% CTR

Solution: IPS (Inverse Propensity Scoring) Weight samples by position.

# Higher positions get lower weight to correct bias
weight = 1 / (1 + position_in_feed)

# Use in loss: weighted_loss = weight * cross_entropy_loss

Or use randomized position tests where you occasionally show posts out-of-order to measure true preference.

Challenge 2: Popularity Bias

Popular posts (from big accounts, trending topics) get more engagement regardless of personalization.

Example: A post by Elon Musk gets high engagement for everyone. But is it because everyone should see it, or because it's popular?

Solution:

• Use content-based features (not just popularity)
• Boost undiscovered content during serving
• Track both absolute engagement and engagement lift (engagement above baseline)

Challenge 3: Feedback Loop

Promoted posts get more engagement → trained model learns to promote them more → they're promoted more → more engagement. Creates a runaway loop.

Example: Once your model learns to show posts from verified accounts, verified accounts dominate the feed, new creators can't break in.

Solution:

• Explore-exploit: Occasionally show lower-ranked posts to test them
• Fairness constraints: Reserve feed space for new creators, diverse topics
• Monitor diversity metrics alongside engagement

Challenge 4: Data Delay

Engagement (label) isn't immediate. A post engagement might occur hours later.

Example: A post is shown at 9am. User engages at 3pm same day. You only label it positive after 3pm.

Solution:

• Train only on old data (older than 24-48 hours)
• Define labeling windows: collect labels after N days
• Accept that your training data is always somewhat stale

Challenge 5: Class Imbalance

Engagement rate is low: maybe 5-10% of impressions result in engagement.

Solution:

• Downsampling: Sample more negatives
• Upweighting: Weight positive examples higher
• Threshold tuning: Lower decision threshold to increase recall

Feature Engineering: Code Example

import pandas as pd
import numpy as np
from datetime import datetime, timedelta

def engineer_features(user_df, post_df, interaction_history):
    """
    Combine user, post, and context features.

    Args:
        user_df: DataFrame with user info
        post_df: DataFrame with post info
        interaction_history: user's recent interactions

    Returns:
        feature_vector: dict of feature_name -> value
    """

    features = {}

    # === User Features ===
    user_info = user_df.loc[user_id]
    features['user_age'] = user_info['age']
    features['user_join_days_ago'] = (datetime.now() - user_info['join_date']).days
    features['user_follower_count'] = user_info['follower_count']
    features['user_engagement_rate'] = user_info['engagement_count'] / user_info['post_count']

    # Encode interests as count
    features['user_interest_count'] = len(user_info['interests'])

    # === Post Features ===
    post_info = post_df.loc[post_id]
    features['post_hours_old'] = (datetime.now() - post_info['timestamp']).total_seconds() / 3600
    features['post_text_length'] = len(post_info['text'])
    features['post_has_media'] = 1 if post_info['media_url'] else 0
    features['post_hashtag_count'] = len(post_info['hashtags'])

    author_info = user_df.loc[post_info['author_id']]
    features['author_follower_count'] = author_info['follower_count']
    features['author_is_verified'] = author_info['is_verified']

    # === Context Features ===
    now = datetime.now()
    hour = now.hour
    features['hour_of_day'] = hour
    features['is_evening'] = 1 if (hour >= 18 or hour < 6) else 0
    features['is_weekend'] = 1 if now.weekday() >= 5 else 0

    # === Interaction Features ===
    user_interactions = interaction_history[interaction_history['user_id'] == user_id]
    features['user_has_followed_author'] = 1 if post_info['author_id'] in user_interactions['followed_users'].values else 0
    features['user_interactions_with_author'] = len(
        user_interactions[user_interactions['author_id'] == post_info['author_id']]
    )

    # Topic overlap: how many post topics match user interests?
    post_topics = set(post_info['hashtags'])
    user_interests = set(user_info['interests'])
    features['topic_overlap'] = len(post_topics & user_interests)

    return features

# Example usage
sample_features = engineer_features(
    user_df=users,
    post_df=posts,
    interaction_history=interactions
)

print(sample_features)
# Output:
# {
#     'user_age': 28,
#     'user_engagement_rate': 0.08,
#     'post_hours_old': 2.5,
#     'post_has_media': 1,
#     'author_follower_count': 100000,
#     'hour_of_day': 14,
#     'user_has_followed_author': 1,
#     'topic_overlap': 2,
#     ...
# }

Training Data Collection

Data Pipeline

Raw Impression Logs
    ↓
[24-hour Wait] ← label collection window
    ↓
Impression + Label (engagement yes/no)
    ↓
Feature Engineering
    ↓
Training Dataset
    ↓
Train/Val/Test Split
    ↓
Model Training

Example Data Snapshot

Train/Val/Test Split

Critical: Split by time, not randomly.

# WRONG: Random split (leakage risk)
train_data, test_data = train_test_split(data, test_size=0.2, random_state=42)

# CORRECT: Temporal split
cutoff_date = datetime(2024, 3, 1)
train_data = data[data['timestamp'] < cutoff_date]
test_data = data[data['timestamp'] >= cutoff_date]

Why? Your future model will predict on future data. Training on future data introduces look-ahead bias.

Summary Callouts

💡 Tip: Engagement is not binary. A user who spends 10 minutes reading is different from someone who clicks and leaves in 2 seconds. Track multiple signals.

📝 Note: Position bias is real and subtle. If you ignore it, your model learns to optimize for visibility, not relevance.

⚠️ Watch out: Data delays mean your training labels are always old. If you train on data from yesterday, you're learning patterns from 48+ hours ago.

🎯 Interviewer's Perspective: Data scientists who don't think about data quality and bias usually build mediocre models. Show you understand these challenges.

✅ Decisions made:

• ML task: Binary classification (engagement yes/no)
• Target window: Engagement within 24 hours
• Features: User, post, context, interaction (30-50 features)
• Data source: Impression logs + user profiles
• Challenge mitigation: IPS for position bias, temporal split for train/test

Back to ML System Design Interviews