System design interview question
Design a Real-Time Event Streaming Platform with Dynamic Topic Routing and Backpressure Handling
Design a distributed event streaming system similar to Kafka or Pulsar where producers publish events to topics, consumers subscribe to topics with dynamic routing rules, and the system must handle backpressure, consumer groups, partition rebalancing, and delivery guarantees (at-least-once, exactly-once). Key actors include Producers, Consumers, ConsumerGroups, Topics, Partitions, and a BrokerCluster. The system must support: (1) dynamic subscription with regex-based topic matching and filter predicates, (2) consumer group coordination with automatic rebalancing when consumers join/leave, (3) offset management and commit strategies, (4) backpressure propagation when consumers lag, (5) pluggable serialization/deserialization, and (6) delivery semantics negotiation. Handle edge cases such as: consumer crashes during rebalancing, network partitions causing duplicate processing, out-of-order offset commits, and scaling to millions of events/sec across thousands of partitions.
Practice this question step by step with an AI interviewer that reviews your requirements, your architecture and your trade-offs, and scores the design.