System design interview question
Design a Distributed Anomaly Detection and Alert Routing System (like Datadog/PagerDuty at Scale)
Design a system that ingests millions of time-series metrics from distributed sources, detects anomalies in real-time using dynamic thresholds and ML models, and routes alerts to on-call teams with deduplication and intelligent escalation. The system must handle 500K metrics/second ingestion, support 10M+ monitored entities across 6 continents, provide sub-second anomaly detection latency, ensure alerts reach the correct team within 5 seconds 99.9% of the time, and gracefully handle model training jobs that run asynchronously without blocking the critical path. Consider handling bursty traffic, alert fatigue from correlated anomalies, and the complexity of routing alerts through multiple notification channels (Slack, PagerDuty, SMS) with team-specific escalation policies.
Practice this question step by step with an AI interviewer that reviews your requirements, your architecture and your trade-offs, and scores the design.