Free lesson · ML System Design Interviews

Evaluation & Serving

Medium8 min readmachine-learningml-system-designinterview-prep

Offline Metrics (Evaluation on Test Data)

Offline metrics measure model performance on historical data. They're fast and cheap to compute.

Metric 1: AUC-ROC (Area Under Curve)

What it is: Probability that model ranks a random positive engagement higher than random negative.

Why it matters: Tells if model can distinguish engaged from non-engaged posts.

Interpretation:

• 0.50 = random (useless)
• 0.65 = decent
• 0.70 = good
• 0.75+ = very good

Code:

from sklearn.metrics import roc_auc_score

y_true = test_labels
y_pred_probs = model.predict_proba(X_test)[:, 1]

auc = roc_auc_score(y_true, y_pred_probs)
print(f"AUC-ROC: {auc:.4f}")

Pros: Threshold-agnostic, doesn't care about class balance. Cons: Doesn't care about ranking order (just binary classification).

Metric 2: Logloss (Binary Cross-Entropy)

What it is: Average penalty for wrong predictions.

Formula:

logloss = -1/N * Σ [y_i * log(p_i) + (1-y_i) * log(1-p_i)]

Where:
- y_i = true label (0 or 1)
- p_i = predicted probability

Why it matters: Rewards confident correct predictions, penalizes confident wrong ones.

Interpretation:

• 0.10 = very good (high confidence, usually right)
• 0.20 = good
• 0.30+ = poor

Code:

from sklearn.metrics import log_loss

logloss = log_loss(y_test, y_pred_probs)
print(f"Logloss: {logloss:.4f}")

Metric 3: NDCG (Normalized Discounted Cumulative Gain)

What it is: How well do we rank relevant posts higher?

Why it matters: Ranking order matters. A feed with the best post at position 5 is worse than position 1.

Formula (simplified):

DCG = Σ (2^relevance_i - 1) / log2(position_i + 1)
NDCG = DCG / IDCG  (where IDCG is optimal ranking)

Interpretation:

• 0.0 = terrible (best posts ranked last)
• 0.5 = mediocre
• 0.8 = good
• 1.0 = perfect ranking

Example:

Your ranking:   [good post, bad post, good post] → NDCG: 0.75
Ideal ranking:  [good post, good post, bad post] → NDCG: 1.0

Logloss would treat them similarly (both have 2 goods & 1 bad).
NDCG penalizes putting good posts low.

Code:

from sklearn.metrics import ndcg_score

# y_true[i] = relevance of post i (0 = not engaged, 1 = engaged)
# y_scores[i] = model's predicted score

ndcg = ndcg_score([y_test], [y_pred_probs])
print(f"NDCG: {ndcg:.4f}")

Metric Comparison Table

For a feed system: Track all three. Optimize NDCG or AUC; use Logloss to catch overfitting.

Online Metrics (A/B Tests)

Online metrics measure real user behavior. They take time but are the ground truth.

Metric 1: Click-Through Rate (CTR)

Definition: Clicks / Impressions

Example: 10,000 feed loads, 850 clicks = 8.5% CTR

Why it matters: Direct engagement signal.

Caveats:

• Can be gamed (sensational posts get clicks but don't satisfy)
• Position bias (posts shown higher have higher CTR)

Code:

def compute_ctr(impressions_df):
    """
    impressions_df columns: user_id, post_id, timestamp, engagement
    """
    total_impressions = len(impressions_df)
    total_clicks = impressions_df['engagement'].sum()
    ctr = total_clicks / total_impressions
    return ctr

# Example
ctr = compute_ctr(impressions)
print(f"CTR: {ctr:.4f} ({ctr*100:.2f}%)")

Metric 2: Time Spent on Feed

Definition: Average minutes per session

Why it matters: Indicates content quality and satisfaction.

Measurement:

def compute_time_spent(session_logs):
    """
    session_logs: user_id, session_start, session_end
    """
    time_spent_seconds = (session_logs['session_end'] - session_logs['session_start']).dt.total_seconds()
    avg_minutes = time_spent_seconds.mean() / 60
    return avg_minutes

# Example
avg_time = compute_time_spent(sessions)
print(f"Avg time on feed: {avg_time:.1f} minutes")

Metric 3: Like/Comment/Share Rates

Definition: (Likes / Impressions), (Comments / Impressions), etc.

Why it matters: Different engagement types signal different satisfaction levels.

Code:

def compute_engagement_rates(impressions_df):
    """Compute multi-type engagement rates."""
    total = len(impressions_df)
    return {
        'click_rate': impressions_df['clicked'].sum() / total,
        'like_rate': impressions_df['liked'].sum() / total,
        'comment_rate': impressions_df['commented'].sum() / total,
        'share_rate': impressions_df['shared'].sum() / total,
    }

rates = compute_engagement_rates(impressions)
print(f"Click: {rates['click_rate']:.3f}, Like: {rates['like_rate']:.3f}, ...")

Metric 4: Daily Active Users (DAU)

Definition: Unique users who opened the feed yesterday

Why it matters: Overall platform health.

Code:

def compute_dau(events_df, date):
    """Compute DAU for a specific date."""
    day_events = events_df[events_df['date'] == date]
    dau = len(day_events['user_id'].unique())
    return dau

dau = compute_dau(events, '2024-03-15')
print(f"DAU: {dau:,.0f}")

A/B Testing Strategy

Setup

Control Group (50% of users)
├─ Serve with OLD ranking model
└─ Measure baseline metrics

Treatment Group (50% of users)
├─ Serve with NEW ranking model
└─ Compare metrics

Duration & Sample Size

Rule of thumb: Run for 1-2 weeks to smooth daily variance.

Sample size: Depends on effect size you want to detect.

from scipy import stats

# Suppose baseline CTR is 8%, you want to detect 1% absolute lift (8% → 9%)
baseline_ctr = 0.08
target_ctr = 0.09
alpha = 0.05  # Type I error
beta = 0.20   # Type II error (power = 80%)

# Power analysis
effect_size = target_ctr - baseline_ctr
n_samples = (
    (stats.norm.ppf(1 - alpha/2) + stats.norm.ppf(1 - beta))**2 *
    (baseline_ctr * (1 - baseline_ctr) + target_ctr * (1 - target_ctr))
) / effect_size**2

print(f"Need {n_samples:,.0f} impressions per group to detect {effect_size:.1%} lift")
# Output: Need ~230,000 impressions per group

Statistical Test

After 1-2 weeks, test if treatment is significantly better:

from scipy.stats import chi2_contingency

# Build contingency table
contingency = [
    [control_clicks, control_impressions - control_clicks],
    [treatment_clicks, treatment_impressions - treatment_clicks]
]

# Chi-square test
chi2, p_value, dof, expected = chi2_contingency(contingency)

if p_value < 0.05:
    print(f"Significant difference! p={p_value:.4f}")
    print(f"Treatment is better. Roll out to 100%.")
else:
    print(f"No significant difference. p={p_value:.4f}")
    print(f"Keep iterating.")

Key Principles

1. Randomly assign users to control/treatment (not by time, not by device)
2. Run full weeks (avoid day-of-week effects)
3. Track guardrail metrics (e.g., uninstall rate, don't let it increase)
4. Use intent-to-treat (once assigned, don't switch users)

Serving: Real-Time Ranking

System Architecture

User Opens App
    ↓
[Serving API]
├─ Get user_id
├─ Call candidate generation
├─ Fetch features (feature store)
├─ Score posts (ranking model)
├─ Apply diversity filters
├─ Return feed
    ↓
[JSON Response] (50 ranked posts)

Latency Budget

For < 200ms end-to-end, break down:

Code: Serving Endpoint

from flask import Flask, request, jsonify
import pickle
import time

app = Flask(__name__)

# Load model
with open('models/ranking_v2.pkl', 'rb') as f:
    model = pickle.load(f)

# Load cached feature store
feature_cache = load_feature_store()

@app.route('/feed', methods=['POST'])
def get_feed():
    """Return personalized feed for user."""
    start_time = time.time()

    data = request.json
    user_id = data['user_id']
    limit = data.get('limit', 50)

    try:
        # Stage 1: Candidate Generation (~20ms)
        t0 = time.time()
        candidates = get_candidates(user_id, limit=500)
        logger.info(f"Candidate generation: {(time.time() - t0)*1000:.1f}ms")

        # Stage 2: Feature Engineering (~30ms)
        t0 = time.time()
        feature_vectors = []
        for post_id in candidates:
            features = get_features(user_id, post_id, feature_cache)
            feature_vectors.append(features)
        logger.info(f"Feature engineering: {(time.time() - t0)*1000:.1f}ms")

        # Stage 3: Scoring (~50ms)
        t0 = time.time()
        scores = model.predict_proba(feature_vectors)[:, 1]
        logger.info(f"Scoring: {(time.time() - t0)*1000:.1f}ms")

        # Stage 4: Ranking & Filtering (~30ms)
        t0 = time.time()
        ranked = sorted(
            zip(candidates, scores),
            key=lambda x: x[1],
            reverse=True
        )
        final_posts = diversify(ranked[:limit])
        logger.info(f"Filtering: {(time.time() - t0)*1000:.1f}ms")

        # Return
        response = {
            'posts': [
                {
                    'post_id': post_id,
                    'score': float(score),
                    'rank': i
                }
                for i, (post_id, score) in enumerate(final_posts[:limit])
            ]
        }

        total_time = (time.time() - start_time) * 1000
        logger.info(f"Total latency: {total_time:.1f}ms")

        return jsonify(response)

    except Exception as e:
        logger.error(f"Error serving feed: {e}")
        # Fallback to simple trending
        return jsonify({
            'posts': get_trending_posts(limit)
        })

def diversify(ranked_posts, limit=50):
    """Remove near-duplicates and balance topics."""
    result = []
    seen_authors = set()
    seen_topics = {}

    for post_id, score in ranked_posts:
        post = fetch_post(post_id)

        # Skip if already showed post from this author
        if post['author_id'] in seen_authors:
            continue

        # Balance topics (max 20% from one topic)
        topic = post.get('topic', 'other')
        if seen_topics.get(topic, 0) > limit * 0.2:
            continue

        result.append((post_id, score))
        seen_authors.add(post['author_id'])
        seen_topics[topic] = seen_topics.get(topic, 0) + 1

        if len(result) == limit:
            break

    return result

Feature Store: Pre-Computing Features

Computing features on-the-fly is slow. Pre-compute and cache them.

What Gets Cached?

+──────────────────────────────────────+
│      Feature Store (Offline)         │
├──────────────────────────────────────┤
│                                      │
│ [1] User features (static, daily)    │
│     - age, interests, follower count │
│     - engagement_rate, activity_level│
│     - Updated: once per day          │
│                                      │
│ [2] Post features (static)           │
│     - author, timestamp, length      │
│     - topic, sentiment               │
│     - Updated: when post is created  │
│                                      │
│ [3] Interaction features (cached)    │
│     - has_user_followed_author       │
│     - user_topic_interest            │
│     - Updated: daily                 │
│                                      │
└──────────────────────────────────────┘

Storage

Options:

• Redis (in-memory): Ultra-fast, for hot features
• Memcached: Similar to Redis
• HBase/DynamoDB (distributed): Scalable, slightly slower
• Local cache (on serving servers): Fastest but stale

Typical setup: Hot user/post features in Redis (100MB), everything else in HBase.

Code: Feature Store Client

import redis

class FeatureStore:
    def __init__(self, redis_host='localhost'):
        self.redis = redis.Redis(host=redis_host, port=6379)

    def get_user_features(self, user_id):
        """Fetch cached user features."""
        key = f"user:{user_id}"
        cached = self.redis.get(key)

        if cached:
            return json.loads(cached)
        else:
            # Cache miss; compute and store
            features = compute_user_features(user_id)
            self.redis.setex(key, 3600, json.dumps(features))  # 1-hour TTL
            return features

    def get_post_features(self, post_id):
        """Fetch cached post features."""
        key = f"post:{post_id}"
        cached = self.redis.get(key)

        if cached:
            return json.loads(cached)
        else:
            features = compute_post_features(post_id)
            self.redis.set(key, json.dumps(features))  # No expiry (posts are permanent)
            return features

# Usage
fs = FeatureStore()
user_features = fs.get_user_features('user_123')
post_features = fs.get_post_features('post_456')

Monitoring & Alerting

Key Metrics to Monitor

Code: Monitoring Dashboard

import logging
from prometheus_client import Counter, Histogram

# Prometheus metrics
request_count = Counter('feed_requests_total', 'Total feed requests')
latency_histogram = Histogram('feed_latency_ms', 'Feed latency in ms')
ctr_gauge = Gauge('feed_ctr', 'Click-through rate')
model_auc_gauge = Gauge('model_auc', 'Model AUC')

@app.route('/feed', methods=['POST'])
def get_feed():
    start_time = time.time()

    try:
        response = serve_feed(request.json)
        request_count.inc()
        latency_histogram.observe((time.time() - start_time) * 1000)
        return jsonify(response)

    except Exception as e:
        logger.error(f"Feed error: {e}")
        request_count.labels(status='error').inc()
        return jsonify({'error': 'Service unavailable'}), 500

# Batch job: Update metrics hourly
def update_metrics_hourly():
    while True:
        ctr = compute_ctr_last_hour()
        model_auc = evaluate_model_sample()

        ctr_gauge.set(ctr)
        model_auc_gauge.set(model_auc)

        time.sleep(3600)

Feature Drift Detection

Monitor if feature distributions change over time:

def detect_feature_drift(current_features, baseline_features, threshold=0.1):
    """
    Use Kolmogorov-Smirnov test to detect distribution shift.
    """
    from scipy.stats import ks_2samp

    for feature_name in current_features.columns:
        current_dist = current_features[feature_name]
        baseline_dist = baseline_features[feature_name]

        ks_stat, p_value = ks_2samp(current_dist, baseline_dist)

        if ks_stat > threshold:
            logger.warning(
                f"Feature drift detected: {feature_name} "
                f"(KS stat: {ks_stat:.3f}, p-value: {p_value:.4f})"
            )

            # Trigger retraining
            trigger_retraining()

End-to-End Pipeline

Daily Batch Job (Midnight UTC)
    ↓
[Collect Data]
└─ Get impressions from last 24 hours
└─ Deduplicate, clean, validate

    ↓
[Feature Engineering]
└─ Compute user features
└─ Compute post features
└─ Join with interaction history

    ↓
[Training]
└─ Load latest data (>24 hours old, labeled)
└─ Train new model
└─ Evaluate on test set

    ↓
[Shadow Testing] (6-12 hours)
└─ Serve new model offline
└─ Compare metrics to production model
└─ No user-facing impact

    ↓
[Online Testing] (1-2 weeks)
└─ Serve new model to 5% of users
└─ Monitor CTR, time spent, DAU
└─ Compare to control group

    ↓
[Decision]
├─ If better: Roll out to 100%
└─ If worse: Roll back

Summary Callouts

💡 Tip: Optimize for NDCG, not AUC. AUC measures binary classification; NDCG measures ranking quality, which matters for feeds.

📝 Note: Latency is critical. 100ms difference in page load time can drop DAU by 5%. Profile every component.

⚠️ Watch out: Feature drift happens silently. Monitor your feature distributions, not just model metrics.

🎯 Interviewer's Perspective: Asking about monitoring and drift detection shows you think about production systems, not just model accuracy.

✅ Decisions made:

• Offline metrics: AUC, Logloss, NDCG
• Online metrics: CTR, time on feed, DAU
• A/B test duration: 1-2 weeks
• Serving latency target: <200ms (p99)
• Feature store: Redis for hot features
• Monitoring: Prometheus + Grafana

Back to ML System Design Interviews