System design interview question
Design a Distributed Job Scheduling and Execution Platform for a Global Analytics Workflow Engine (Like Airflow/Dagster at 100M Task Runs/Day)
You are designing the backend for a distributed job scheduling platform that handles 100M task executions per day across 50+ data centers globally. The system must support complex DAG-based workflows with dependencies, dynamic task retries, multi-tenant isolation, and real-time job status tracking. Workflows range from millisecond-latency real-time transforms to 24-hour batch jobs. You need to design a system that can atomically persist job state, coordinate task execution across geographically distributed workers, handle task failures and cascading retries, provide exactly-once execution guarantees for critical workflows, and expose queryable job history and metrics. The platform must support 10K concurrent workflow submissions, 50K active task executions at any moment, and maintain sub-second latency for job submission APIs while handling network partitions and worker failures gracefully.
Practice this question step by step with an AI interviewer that reviews your requirements, your architecture and your trade-offs, and scores the design.