System design interview question
Design a Distributed Real-Time Anomaly Detection and Alerting System like Datadog's Monitoring Platform at Scale
Design a system that ingests, processes, and monitors high-volume metrics from millions of distributed sources (servers, containers, applications) in real-time, detects anomalies using statistical models, and delivers alerts with sub-second latency to on-call teams. The system must handle 500M+ metrics per minute from 2M+ monitored entities across 50+ global regions, support custom alert rules (threshold-based, ML-based, composite), maintain 99.99% detection accuracy, and provide a queryable time-series database for historical analysis. Consider multi-tenancy, cost optimization for storage and compute, handling late-arriving data, managing alert fatigue, and supporting both rule-based and ML-driven detection patterns.
Practice this question step by step with an AI interviewer that reviews your requirements, your architecture and your trade-offs, and scores the design.