System design interview question
Design a Multi-Model LLM Inference Router with Dynamic Fallback & Cost Optimization
Design a production-grade inference routing system that intelligently dispatches LLM requests across multiple heterogeneous model providers (OpenAI, Anthropic, local vLLM, etc.) based on latency SLAs, cost budgets, model capabilities, and real-time provider health. The system must support prompt caching, request deduplication, graceful degradation with fallback chains, and adaptive routing that learns from historical performance. Key entities include RequestRouter, ModelProvider (with health tracking), RoutingPolicy, CostTracker, and CacheManager. Complex edge cases: handling partial provider outages, managing in-flight request retries without duplication, enforcing cost limits mid-request, and reconciling billing across async callbacks. Senior candidates should demonstrate understanding of Strategy/Chain of Responsibility patterns for routing policies, Observer pattern for health monitoring, Decorator pattern for request enrichment (caching, retry logic), and concurrency primitives for safe state management across distributed callbacks.
Practice this question step by step with an AI interviewer that reviews your requirements, your architecture and your trade-offs, and scores the design.