Free lesson · ML System Design Interviews

Harmful Content Detection System Design

Medium7 min readmachine-learningml-system-designinterview-prep

Module 1: Problem Definition & Discovery

What Are We Building?

Imagine you run a social platform with millions of users posting content every day. Some of that content is great—photos, stories, helpful advice. But some of it is dangerous or harmful: violence, hate speech, nudity, misinformation, spam.

Your job: Build a system that automatically detects and removes this harmful content before it spreads.

This is like running an airport. You want to let good passengers through fast, but catch anything dangerous before it gets on the plane.

Types of Harmful Content

Let's be specific about what we're trying to catch:

Each type has different severity. Violence and child exploitation are emergencies. Spam is annoying but less urgent.

Why This Matters

User Safety: Harmful content traumatizes users and can lead to real-world violence or suicide.

Legal Compliance: Many countries now have laws requiring platforms to remove illegal content (hate speech, child exploitation, etc.). Fines are in the billions.

Brand Reputation: If your platform becomes known as a hate speech haven, advertisers leave and trust erodes.

Scale: You can't hire enough humans to manually review millions of posts per day.

The Airport Analogy

Think of your content moderation system like airport security:

• Goal: Catch dangerous items (weapons, explosives) without stopping every passenger.
• False positives: Stop someone with nail clippers (they're not dangerous but look suspicious). Annoying but acceptable.
• False negatives: Let a weapon through. Catastrophic.
• Speed: You have 5 seconds per passenger. If screening is too slow, the airport backs up.
• Layers: Metal detector, X-ray, pat-down. Multiple signals together catch more than one alone.

Your ML system needs the same balance: catch most harmful content, tolerate some false alarms, run fast, and use multiple signals.

Mock Interview: Discovery Phase

Exchange 1: Scope & Content Types

Interviewer: "What types of content are we dealing with?"

Candidate: "I'd say text posts, images, videos, and maybe comments. Text is straightforward, but images and video are harder because the model needs to understand visual content, not just words."

Interviewer: "Right. How would you distinguish between a news article about violence and an actual violent video someone posted?"

Candidate: "That's tricky. A news article might say 'A building was bombed today' with a headline. A violent video just shows the explosion directly. I'd probably need to combine signals—context, metadata (is it from a news source?), and the image/video itself. Context really matters here."

Exchange 2: What Counts as Harmful?

Interviewer: "Define 'harmful.' Is a blurry photo of violence the same as high-definition gore? Is a meme joking about violence the same as someone actually threatening to hurt people?"

Candidate: "Good point. Severity matters. I'd probably create categories:

• Critical (remove immediately): Direct threats, actual violence videos, child exploitation
• High (remove, maybe review): Hate speech, graphic violence, explicit sexual content
• Medium (flag for review): Borderline misinformation, edgy jokes
• Low (maybe do nothing): Spam, low-confidence matches"

Interviewer: "And who decides those thresholds?"

Candidate: "That's a policy decision, not a pure ML thing. The company's legal, trust & safety, and policy teams would decide. I'd just build the system flexible enough to tune those thresholds."

Exchange 3: Human Review Pipeline

Interviewer: "How many pieces of content do you expect per day?"

Candidate: "Let's say 100 million posts per day. We definitely can't manually review all of them. But maybe 0.1% are flagged by the model as potentially harmful. That's 100,000 to review. Still huge."

Interviewer: "So how do you use humans?"

Candidate: "I'd have the model score everything from 0 to 1. Then:

• Score > 0.95: Auto-remove (very confident it's bad)
• Score 0.5-0.95: Send to human reviewers (model is uncertain)
• Score < 0.5: Let it through (probably safe)

This way humans focus on the hard cases where the model is unsure. For the easy cases—obvious violence or hate speech—we auto-remove and save human time."

Interviewer: "What if a human reviewer makes a mistake?"

Candidate: "We'd log their decisions, maybe require a second opinion on critical cases, and feedback their decisions back to the ML model to retrain it."

Exchange 4: Real-Time vs Batch Detection

Interviewer: "Does content need to be checked the moment it's uploaded, or can we check it later?"

Candidate: "Ideally in real-time. If someone posts a violent video, we want to catch it before 10,000 people see it. That's maybe a few hundred milliseconds of latency tolerance."

Interviewer: "That's fast. What if real-time processing fails?"

Candidate: "Good fallback: if the real-time path times out, we could either (a) delay publishing the content a bit and re-try, or (b) publish it optimistically and run a batch check later, then remove it if needed. Option (a) is safer but slower. Option (b) is faster but content might briefly go live."

Exchange 5: False Positive Tolerance

Interviewer: "When your model incorrectly flags something as harmful, what happens?"

Candidate: "The user can appeal. They say 'Hey, this is a meme, not actual hate speech.' We review the appeal, and if they're right, we restore the content and use that signal to improve the model."

Interviewer: "How often can you afford to be wrong?"

Candidate: "It depends on the category. For hate speech, maybe 5-10% false positives is okay if we catch 90% of the real stuff. For violence, we probably want higher precision too—false positives feel like censorship. But we definitely can't have high false negatives (missing actual harmful content). Missing is worse than over-flagging."

Exchange 6: Scale & Global Platform

Interviewer: "Your platform is global. How does scale affect the design?"

Candidate: "Huge impact. 100 million posts per day means we need a fast pipeline. Can't use slow models. Also, we might need regional customization—what counts as hate speech in India might be different from Brazil. And we need low latency in all regions, so maybe we cache models close to users (edge deployment)."

Exchange 7: Multilingual Support

Interviewer: "What about languages? Most harmful content is text-based, but you probably have users in 50+ countries."

Candidate: "That's a big challenge. Toxicity classifiers are usually trained on English. We'd need to either:

1. Collect labeled data in many languages (expensive)
2. Translate everything to English first (adds latency, loses nuance)
3. Use multilingual models like mBERT that work across languages
4. Partner with local moderators who speak the language

Probably a mix of all four."

Exchange 8: Moderation Speed & Latency

Interviewer: "What's your target latency for the moderation check?"

Candidate: "For real-time detection at upload, I'd target <500ms end-to-end. That's fast enough that users don't notice. For images, we might need GPU inference. For text, CPUs are fine. For video, we'd probably sample frames rather than process the whole thing."

Functional Requirements

FR1: Detect and flag harmful content across text, images, and videos.

FR2: Provide a score (0-1) representing confidence that content is harmful.

FR3: Support at least 5 content categories (violence, hate speech, nudity, misinformation, spam).

FR4: Enable human review workflows for borderline cases.

FR5: Allow users to appeal content removal decisions.

FR6: Log all moderation actions for auditing and legal compliance.

FR7: Support 50+ languages.

Non-Functional Requirements

NFR1: Latency <500ms for real-time content checks.

NFR2: Throughput: process 100M+ posts per day.

NFR3: Recall >90% (catch at least 90% of harmful content).

NFR4: Precision >85% (at most 15% false positives).

NFR5: 99.9% availability (system rarely goes down).

NFR6: Scalable to 10x traffic growth.

NFR7: Support A/B testing of model changes.

Key Challenges

Challenge 1: Context Matters

A headline saying "200 killed in building collapse" is not harmful, even though it mentions death. A video of violence is. The same word—"kill"—can be fine in context ("This new movie will kill at the box office") or harmful ("I'm going to kill you").

Implication: You need more than keyword matching. You need real language understanding.

Challenge 2: Subjectivity

What counts as hate speech? Is criticizing a government "hate speech"? Is edgy humor "misinformation"? Different humans (and cultures) disagree.

Implication: Your model won't be perfect. Build systems to handle disagreement and ambiguity.

Challenge 3: Evolution & Adversarial Content

Users are clever. If you block the word "n-word", people spell it "n1gg3r" or use dog whistles ("13" for the KKK, "88" for white power). Harmful content evolves.

Implication: Your system must retrain frequently and adapt to new harmful patterns.

Challenge 4: False Positives Feel Like Censorship

If your system removes a legitimate post, the user feels unfairly silenced. This hurts trust more than missing some harmful content.

Implication: Humans-in-the-loop matter. Transparency matters. Appeals matter.

Summary: The Big Picture

You're building a system to automatically detect harmful content on a platform with millions of users. It needs to be fast (<500ms), accurate (90%+ recall, 85%+ precision), and support human review for uncertain cases. The hard parts aren't the ML—they're handling context, subjectivity, evolution, and scale.

Next, we'll look at how to frame this as an ML problem and prepare the data.

Discussion Questions

1. If you had to choose between false positives and false negatives, which would you prioritize? Why?
2. How would you handle misinformation that's technically true but misleading? (E.g., "Vaccines cause autism" is false, but "Some people reported side effects after vaccines" is true.)
3. What's your biggest worry about deploying an automated moderation system? Bias? Speed? Costs?

Back to ML System Design Interviews