AlgoInsight

System Design

Designing a Data Pipeline That Ingests Billions of Events a Day

A pipeline handling a thousand events a day and one handling billions are not the same system at a different scale - they're different systems. Here's how to actually architect ingestion at the scale where every shortcut breaks.

By Admin2026-06-1817 min readdata-pipelinestreamingkafkaflinkscalabilitydata-engineering

There's a comforting myth that a system handling a billion events a day is just the small version scaled up - same design, bigger machines. It isn't true. At that volume, the shortcuts that work fine for a thousand events a day actively break: you can't write each event straight to a database, you can't process them one at a time synchronously, you can't tolerate any component that doesn't scale horizontally, and you certainly can't lose data when a machine inevitably fails. Designing for billions of events a day is a different discipline, and it's the kind of system I work on. Here's how to architect it - the components, why each exists, and the problems that only appear at scale.

The shape of the pipeline

Every large-scale event pipeline, whatever the specifics, has the same fundamental shape - a series of stages, each decoupled from the next:

  1. Ingestion - events arrive from many sources and are accepted into the system.
  2. Buffering / messaging - events land in a durable, scalable buffer (a log like Kafka) that decouples the fast, spiky arrival of events from the slower, steadier processing.
  3. Stream processing - events are processed, transformed, enriched, aggregated (a processor like Flink), in real time.
  4. Storage - processed data lands in stores suited to how it'll be used (a data lake, a warehouse, serving databases).
  5. Serving - the data is served to consumers: dashboards, analytics, applications, other systems.

The decoupling between stages is the foundation. Each stage scales independently and absorbs the variability of the others. Let me walk through each and the scale problems it solves.

graph LR
    SRC[Event sources<br/>apps, services, devices] --> ING[Ingestion layer]
    ING --> BUF[(Durable buffer<br/>Kafka)]
    BUF --> SP[Stream processing<br/>Flink]
    SP --> LAKE[(Data lake /<br/>warehouse)]
    SP --> SERVE[(Serving stores)]
    LAKE --> CONS[Analytics / ML]
    SERVE --> APP[Applications / dashboards]

This diagram shows a large-scale event pipeline. Event sources - applications, services, and devices - send events to an ingestion layer. From there events flow into a durable buffer such as Kafka, which decouples arrival from processing. A stream processor such as Flink reads from the buffer and processes events in real time, writing results to a data lake or warehouse for analytics and machine learning, and to serving stores for applications and dashboards. Each stage is decoupled and scales independently.

Stage 1 & 2: ingestion and the durable buffer

The first scale lesson: never write incoming events directly to your final storage. At a thousand events a day you could. At billions a day, the arrival rate is enormous and spiky - bursts that would overwhelm a database trying to absorb them synchronously. So the front of the pipeline is an ingestion layer that accepts events and immediately hands them to a durable, scalable buffer - a distributed log like Kafka.

This buffer is the keystone of the whole design, and it does several jobs at once:

Getting this layer right is most of the battle. At billions of events a day, the durable, partitioned buffer is what makes everything downstream tractable - it turns a firehose into something processable, durably and at scale.

Stage 3: stream processing at scale

From the buffer, a stream processor (like Flink) reads events and does the real work: filtering, transforming, enriching, aggregating, computing real-time metrics. At this scale, the processor must itself be a distributed system, processing partitions in parallel across many machines.

The hard problems here are the ones stream processing always faces, now at volume:

The processor turns the raw event stream into useful, correct results - in real time. This is where the pipeline earns its value: not just moving events, but computing meaning from them as they flow.

Stage 4 & 5: storage and serving

Processed data needs to land somewhere, and at scale you don't use one store - you use the right store for each access pattern (polyglot persistence):

The principle: the pipeline fans processed data out to stores matched to how it'll be consumed. Bulk analytical data goes to the lake and warehouse; fast-access results go to serving stores. Forcing everything into one store would serve some needs poorly - at this scale, you match each consumer to the store built for it.

A worked example: one event's journey

To make the architecture concrete, follow a single event through the pipeline. A user watches a video, and the app emits a "video watched" event - a small record with the user, the video, a timestamp, and some context.

The event hits the ingestion layer, which accepts it and writes it to the durable buffer (Kafka), into a partition (say, chosen by user ID so all of one user's events stay ordered together). The event is now durable - even if everything downstream fails, it's safe in the log. The app, meanwhile, got an immediate acknowledgment and moved on; it didn't wait for any processing.

A stream processor (Flink) reads the event from Kafka. It might enrich it (join it with video metadata), update real-time aggregates (increment "views for this video this minute," in event time, as fault-tolerant state), and detect patterns. The processed results flow onward: the raw event is written to the data lake for later batch analytics and machine learning; aggregated metrics go to a warehouse for analytical queries and to serving stores for low-latency access by dashboards. Moments later, a real-time dashboard reflects the view, an analyst can query it historically, and a machine-learning job can use it for recommendations.

One small event, and it fanned out durably to real-time metrics, bulk analytics, and serving - each stage decoupled, each scaling on its own. Multiply that by billions of events a day, flowing continuously, and you have the system. Tracing one event shows why the architecture is shaped as it is: the buffer makes the event durable and decouples the fast emit from the slower processing, the processor computes meaning in real time, and the fan-out gets the result to every consumer in the form each needs.

The ingestion layer and partitioning strategy

A closer look at the front of the pipeline, because choices there ripple through everything. The key decision is how events are partitioned in the durable buffer, because partitioning determines both scale and ordering. Events are spread across many partitions for parallel throughput, but events that must stay ordered relative to each other need to share a partition - which you arrange by giving them the same partition key.

The classic choice is to partition by an entity ID (user, device, account), so all events for one entity land in the same partition, in order, processable together with their ordering intact - while different entities spread across partitions for scale. Get this right and you have both massive parallel throughput and the ordering guarantees your processing needs. Get it wrong - partition by something that creates imbalance - and you get hot partitions (one overloaded while others idle), the same hot-spot problem that bad shard keys cause in databases. At a billion events a day, a poor partitioning choice is the difference between a pipeline that scales smoothly and one that bottlenecks on a single overloaded partition. This is why I treat the partitioning key as one of the most important early decisions in a pipeline's design, not a detail to settle later.

The recurring theme: decoupling buys resilience

Step back and the same principle appears at every stage: decoupling. Ingestion is decoupled from processing by the buffer. Processing is decoupled from storage by writing results out asynchronously. Different storage consumers are decoupled from each other, each reading what it needs. This isn't incidental - it's the core design philosophy that makes scale survivable. Because the stages are decoupled, each can scale independently, fail independently, and absorb the variability of the others. A spike in ingestion is absorbed by the buffer rather than overwhelming processing. A slow storage write doesn't block the processor. A new consumer can be added without disturbing the rest. At a billion events a day, with machines constantly failing somewhere and load constantly shifting, this decoupling is what keeps the whole system from being only as strong as its weakest, busiest component. Tight coupling at this scale is fragility; decoupling is resilience - and every architectural choice in the pipeline serves it.

The problems that only appear at scale

Beyond the happy path, designing for billions of events a day means designing for the problems that only show up at volume. These are what separate a real large-scale pipeline from a toy:

These aren't edge cases at a billion events a day - they're daily realities. A pipeline that handles the happy path but ignores backpressure, lag monitoring, schema evolution, and bad data will fall over in production. The discipline of large-scale data engineering is largely the discipline of handling these well.

Lambda, Kappa, and keeping it simple

A brief architectural note: historically, teams ran two paths - a real-time streaming path for fast-but-approximate results and a separate batch path for slow-but-accurate results (the "Lambda architecture"). It worked but meant maintaining two systems with duplicated logic. The trend, enabled by stream processors that are both fast and correct (exactly-once, event-time), is toward a single streaming path that serves both needs (the "Kappa architecture") - one pipeline, less duplication. The lesson I'd emphasize: prefer the simpler single-path design when your stream processor is trustworthy enough to be your source of truth, and add complexity only when a specific need forces it. Simplicity at scale is a feature, because every additional moving part is another thing that can fail at a billion events a day.

Interview: "Design a system that ingests a huge volume of events" (analytics pipelines, metrics systems, activity tracking) is a common system design prompt, and the expected architecture is exactly this: ingestion into a durable partitioned buffer (Kafka), stream processing (Flink) for real-time transformation and aggregation, and fan-out to storage matched to access patterns (data lake, warehouse, serving stores). The senior signals are naming why the buffer exists (absorb spikes, decouple, durability, scale), addressing the scale-only problems (backpressure, consumer lag, schema evolution, exactly-once, bad data), and preferring a simple single streaming path. That shows you've thought about ingestion at real scale, not just drawn the boxes.

The bottom line

A pipeline ingesting billions of events a day is a different system from a small one, not a scaled-up version. Its shape is universal: ingestion into a durable, partitioned buffer (Kafka) that absorbs spikes, decouples producers from consumers, and provides durability; stream processing (Flink) that transforms and aggregates in real time with event-time correctness and exactly-once guarantees; and fan-out to storage matched to use (data lake for bulk, warehouse for analytics, serving stores for fast access). What makes it real is handling the scale-only problems - backpressure, consumer lag monitoring, schema evolution, end-to-end data integrity, bad data, and cost.

The art is decoupling each stage so it scales independently, choosing the right tool for each, and respecting that at a billion events a day every shortcut breaks and every component must scale horizontally and survive failure. Get the durable buffer right, process correctly, store by access pattern, and handle the hard scale problems - that's how the firehose becomes something you can trust. We cover large-scale data pipelines and streaming architecture across System Design Fundamentals.

Go deeper

More articles