Free lesson · ML System Design Interviews

Harmful Content Detection System Design

Medium8 min readmachine-learningml-system-designinterview-prep

Module 4: Evaluation, Serving & Monitoring

Offline Metrics: How Good Is the Model?

Before deploying, we test the model on a held-out test set. Here are the key metrics:

Metric 1: Precision and Recall

from sklearn.metrics import precision_score, recall_score, f1_score

# Assume we have true labels and model predictions
y_true = [1, 0, 1, 1, 0, 0, 1, 0]  # True: harmful or not
y_pred = [1, 0, 0, 1, 0, 1, 1, 0]  # Model predicts

TP = sum((y_true[i] == 1 and y_pred[i] == 1) for i in range(len(y_true)))  # 2
FP = sum((y_true[i] == 0 and y_pred[i] == 1) for i in range(len(y_true)))  # 1
FN = sum((y_true[i] == 1 and y_pred[i] == 0) for i in range(len(y_true)))  # 1
TN = sum((y_true[i] == 0 and y_pred[i] == 0) for i in range(len(y_true)))  # 4

precision = TP / (TP + FP)  # 2 / 3 = 0.67
recall = TP / (TP + FN)      # 2 / 3 = 0.67
f1 = 2 * (precision * recall) / (precision + recall)  # 0.67

print(f"Precision: {precision:.2f}")  # 0.67
print(f"Recall: {recall:.2f}")        # 0.67
print(f"F1: {f1:.2f}")                # 0.67

What Do These Mean?

Metric	Formula	Meaning	Why It Matters
Precision	TP / (TP + FP)	Of the things we flagged, how many were actually harmful?	False positives feel like censorship. Users hate being wrongly flagged.
Recall	TP / (TP + FN)	Of the harmful content, how much did we catch?	Missing bad stuff is worse than over-flagging. If you miss violence, users get hurt.
F1 Score	2 × (P × R) / (P + R)	Harmonic mean of precision and recall	Balances both. High F1 = good at both catching and not over-flagging.

For Content Moderation, Prioritize Recall

Scenario 1: High Precision, Low Recall (Threshold = 0.95)
  → Catch 70% of harmful content, 5% false positives
  → Miss 30% of harmful content (some users hurt, videos spread)
  → Safer users, but platform has a real problem

Scenario 2: Low Precision, High Recall (Threshold = 0.5)
  → Catch 95% of harmful content, 20% false positives
  → Some false alarms, but platform is much safer
  → Users annoyed by false positives, but most harmful content is gone

Better to choose Scenario 2 (high recall).

Per-Category Metrics

Report precision/recall separately for each category:

Violence:
  Precision: 0.91  ✓ High
  Recall: 0.88     ✓ High (catching 88% of violence)

Hate Speech:
  Precision: 0.78  ⚠️ Lower (more false positives)
  Recall: 0.85     ✓ Good

Misinformation:
  Precision: 0.62  ✗ Low (hard to detect)
  Recall: 0.71     ⚠️ Only catching 71%

Nudity:
  Precision: 0.94  ✓ High
  Recall: 0.92     ✓ High

Interpretation: You're confident about violence and nudity. Misinformation is harder (lower precision). Hate speech has medium performance.

Confusion Matrix (For Binary Classification)

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

y_true = [1, 0, 1, 1, 0, 0, 1, 0]
y_pred = [1, 0, 0, 1, 0, 1, 1, 0]

cm = confusion_matrix(y_true, y_pred)
#     Predicted
#     0  1
# 0 [ 4, 1]  (TN, FP)
# 1 [ 1, 2]  (FN, TP)

ConfusionMatrixDisplay(cm).plot()
plt.show()

Online Metrics: Real-World Performance

Offline metrics are good, but they don't tell you everything. Once deployed, track these:

Metric 1: User Appeals Rate

Users can appeal when they think you wrongly removed their content.

Total content removed: 50,000
User appeals: 2,000
Appeal rate: 4%

If appeal rate is high (>5%), your precision is probably too low.
If appeal rate is low (<1%), you might be under-flagging (precision too high, recall too low).

Metric 2: Appeal Approval Rate

Of the appeals, how many do you uphold (agree the user was right)?

User appeals: 2,000
Appeals upheld: 300
Appeal approval rate: 15%

High approval rate = your model is making mistakes
Low approval rate = your model is trustworthy
Healthy range: 5-10%

Metric 3: Time to Removal

How long does it take to remove harmful content?

Average time from post to removal:
- Violence: 2 hours (quick, human reviewers prioritize)
- Hate speech: 4 hours (takes longer to verify)
- Spam: 6 hours (lower priority)

SLA (Service Level Agreement): Remove 90% of harmful content within 24 hours.

Metric 4: False Positive Rate

Estimate false positive rate from appeals:

If 100,000 posts are removed and 10,000 appeal:
  → 10% appeal rate
If 1,000 of those appeals are upheld (user was right):
  → False positive rate ≈ 1,000 / 100,000 = 1%

But real false positive rate might be higher (not everyone appeals).
Correct estimate: appeals_upheld / total_removed
(assuming appeal rate is constant across correct and incorrect decisions)

Real-Time Serving Architecture

When a user uploads content, it goes through this pipeline:

User uploads content (text, image, video)
  ↓
1. Pre-processing (resize image, extract frames from video)
  ↓
2. Feature extraction (text embeddings, CNN features, metadata)
  ↓
3. Model inference (text model, image model, metadata model in parallel)
  ↓
4. Ensemble (combine predictions)
  ↓
5. Apply thresholds
  ↓
Decision:
  - Score > 0.95: AUTO_REMOVE (confident it's bad)
  - Score 0.5-0.95: QUEUE_FOR_REVIEW (send to human)
  - Score < 0.5: PUBLISH (allow post to go live)

Latency Requirements

Goal: <500ms end-to-end

Breakdown (100ms budget per step):
  - Pre-processing: 50ms
  - Feature extraction: 100ms
  - Text model inference: 50ms
  - Image model inference: 100ms (heaviest, runs on GPU)
  - Metadata model: 10ms
  - Ensemble & thresholding: 20ms
  ─────────────────────────────
  Total: ~330ms ✓

If any step times out, fallback to publishing with delayed moderation check (batch process later).

Human-in-the-Loop Pipeline

Your system doesn't decide everything alone. Humans handle ambiguous cases.

Content uploaded
  ↓
Model predicts score
  ├─ If score > 0.95 → Auto-remove (log decision for audit)
  ├─ If score 0.5-0.95 → Queue for human review
  │   ↓
  │   Human reviewer sees: [thumbnail, text, context, model confidence]
  │   Human decides: Remove, Restore, or Escalate
  │   ↓
  │   Log decision → Use to retrain model
  │
  └─ If score < 0.5 → Publish

Why Humans-in-the-Loop?

Mistakes are costly: If you auto-remove a legitimate post, user feels censored
Context matters: Models miss nuance (sarcasm, dark humor, legitimate criticism)
Cultural differences: What's normal in Brazil might be offensive in Japan
Feedback loop: Human decisions improve the model

Scaling Humans

As your platform grows, you need more reviewers. But hiring is slow. Solutions:

Tiered review: Easy decisions go to junior reviewers, hard ones to experts
Queuing: Route certain categories to experts (hate speech experts, violence experts)
Outsourcing: Contract with moderation services (but maintain quality control)

Monitoring & Feedback Loop

What to Monitor in Production

# Dashboard showing real-time metrics

dashboard_metrics = {
    "throughput": {
        "posts_per_minute": 5000,
        "images_per_minute": 2000,
        "videos_per_minute": 500
    },
    "latency": {
        "p50": 180,  # 50th percentile
        "p95": 450,  # 95th percentile
        "p99": 890   # 99th percentile (milliseconds)
    },
    "model_performance": {
        "violence_precision": 0.91,
        "violence_recall": 0.88,
        "hate_speech_precision": 0.78,
        "hate_speech_recall": 0.85,
        # ... etc
    },
    "user_feedback": {
        "appeal_rate": 0.04,        # 4% of removals appeal
        "appeal_approval_rate": 0.08  # 8% of appeals are upheld
    },
    "errors": {
        "model_timeouts": 12,       # Failed to finish in 500ms
        "inference_errors": 5,      # Model crashed
        "total_errors": 17          # Total error rate: 17 / (5M posts) = 0.0003%
    }
}

Detecting New Types of Harmful Content

Over time, new harmful patterns emerge. Your monitoring should detect this:

# Track the distribution of content flagged by category

flagged_distribution = {
    "violence": 0.3,
    "hate_speech": 0.25,
    "nudity": 0.2,
    "misinformation": 0.15,
    "spam": 0.1
}

# If you suddenly see a spike in a new slur (unknown word):
if new_slur_detected:
    alert = "New harmful pattern detected. Escal to trust & safety team."
    # Add slur to keyword list
    # Retrain model with examples of new slur
    # Deploy updated model within 24 hours

Feedback Loop: Learning from Humans

# Log every human review decision

human_decision_log = {
    "content_id": "xyz",
    "model_prediction": 0.82,  # Model said 82% likely harmful
    "human_decision": "REMOVE",  # But human removed it
    "agreement": True,  # Model and human agree ✓
    "timestamp": "2024-03-15T14:32:10Z",
    "reviewer_id": "reviewer_42"
}

# Collect disagreements
disagreement_log = [log for log in all_logs if log["agreement"] == False]

# When disagreements are high (>5%), retrain the model:
if len(disagreement_log) / len(all_logs) > 0.05:
    print("Model accuracy declining. Retraining...")
    # Add disagreement examples to training data
    # Retrain model
    # A/B test new model on 10% of traffic
    # Roll out if performance improves

A/B Testing: Validate Changes Before Full Rollout

Experiment: Test a new model version

Setup:
  - Control group (10% traffic): Old model
  - Treatment group (10% traffic): New model
  - Duration: 1 week

Metrics to track:
  - Appeal rate (should be lower with new model)
  - Appeal approval rate (should be lower = fewer false positives)
  - Latency (should not increase)
  - User satisfaction (survey: do you feel your content is treated fairly?)

Results (after 1 week):
  Old model: 4.2% appeal rate, 8% approval rate
  New model: 3.8% appeal rate, 7% approval rate

Conclusion: New model is better. Roll out to 100% traffic.

Deployment & Rollback Strategy

Step 1: Deploy new model to shadow traffic
  → Run new model on 0% of real decisions, 100% of logged decisions
  → Compare predictions with old model
  → If disagreement rate < 5%, proceed to step 2

Step 2: Canary deployment
  → Run new model on 1% of real traffic
  → Monitor error rates and latency
  → If metrics are good, proceed to step 3

Step 3: Gradual rollout
  → 1% → 5% → 10% → 50% → 100% over 1 week
  → Monitor at each step

Step 4: Full rollout
  → New model is now the standard
  → Keep old model as instant rollback option
  → Delete old model after 1 month

Rollback: If something breaks:

New model deployed 1 minute ago:
  - Error rate suddenly jumps to 5% (was 0.01%)
  - Latency increases to 2000ms (was 300ms)
  → ROLLBACK IMMEDIATELY (1 click, takes 30 seconds)
  → Investigate the bug
  → Fix and redeploy

Operational Challenges

Challenge 1: Model Staleness

Models degrade over time. Why?

New types of harmful content emerge
Language evolves (new slang)
User behavior changes

Solution: Retrain every month
  Week 1: Collect labeled data from new posts + human reviews
  Week 2: Retrain model
  Week 3: Test A/B
  Week 4: Deploy

Challenge 2: Adversarial Attacks

Users try to evade the model:

"k-i-l-l" instead of "kill"
"naZi" instead of "nazi"
Harmul Symbols hidden in images

Solution: Adversarial training
  1. Find evasion attempts in user-generated examples
  2. Add them to training data
  3. Retrain model to catch these variants
  4. Repeat monthly

Challenge 3: Language & Cultural Diversity

What's harmful in English might not be in Swahili. Different cultures have different norms.

Solution: Localized models
  - Train separate models for top languages
  - Or use multilingual models with language-specific fine-tuning
  - Partner with local moderators for validation

Cost Analysis

Annual cost to moderate 100M posts per day:

Item                        Cost
─────────────────────────────────
Hardware (GPUs for inference): $2M
Human reviewers (500 people): 5M
Data labeling (10k labels/mo): M
Engineering (10 people): $2M
Operations & monitoring: M
─────────────────────────────────
Total: ~$21M

Cost per post: $21M / (100M * 365) = $0.0006 per post

That's cheap because the model handles 99.9% of decisions automatically. Humans only review 0.1%.

Summary: From Model to Production

Offline Evaluation: Precision, recall, F1 on test set
Online Metrics: Appeals, approval rates, time to removal
Real-Time Serving: <500ms latency, auto-remove at >0.95, review queue for 0.5-0.95
Human-in-the-Loop: Humans decide ambiguous cases, improve model
Monitoring: Track performance, detect new harmful patterns, log everything
Feedback Loop: Use human decisions to retrain
A/B Testing: Validate changes before rollout
Deployment: Shadow → Canary → Gradual rollout, with instant rollback

Next, we'll put it all together and discuss advanced topics.

Practice Exercise

Suppose your model has:

Precision: 0.85
Recall: 0.92

What does this mean in plain English?
If you process 1M posts and 500 are actually harmful, how many will your model catch? How many false positives?
Would you increase or decrease the threshold? Why?

Key Takeaways

Recall > Precision for content moderation (missing harmful content is worse than false alarms)
Humans-in-the-Loop for ambiguous cases
Monitor everything: Latency, appeals, accuracy, errors
Feedback loop: Human decisions improve the model
A/B test before full rollout
Design for failure: Rollback strategy, fallbacks

Back to ML System Design Interviews