AlgoInsight

System Design

Batch vs Stream Processing

I run a pipeline at Netflix that takes in more than 8 billion events a day. The single decision that shapes a data system like that is whether you process in batches or as a stream. Here's how I think about it - and how I learned the difference the expensive way.

By Admin2026-06-1814 min readbatch-processingstream-processingdata-engineeringkafkaflinksystem-design

There is a number on a dashboard I watch most days: events per second flowing into the viewing pipeline I help run at Netflix. On a normal evening, when a few hundred million people settle in to watch something, that number climbs into the millions per second, and over a day it adds up to more than 8 billion events. Every play, pause, seek, and stop. Every quality switch. Every "are you still watching?"

Early in my career I would have built a system to handle that data exactly the wrong way - and I did, more than once, before I understood the distinction that governs everything in data processing: do you process data in batches, or as a continuous stream? It sounds like an implementation detail. It is actually the decision that determines what your system can and cannot do. This is how I think about it now, after years of getting it wrong and then getting it right at a scale that doesn't forgive mistakes.

The two ways to process data

There are two fundamental approaches to processing data, and almost every data system is built on one or a combination of both.

Batch processing collects data over a period of time and then processes it all together, in one large chunk, on a schedule. You accumulate a day's worth (or an hour's, or a week's) of data, then run a job that processes the whole pile at once. Think of it like doing laundry: you let it pile up, then wash a full load.

Stream processing processes data continuously, one event at a time (or in tiny micro-batches), as it arrives. There's no waiting to accumulate a pile - each event is handled the moment it shows up. Think of it like washing each dish the instant you're done using it, instead of letting them stack in the sink.

graph TD
    subgraph BATCH[Batch Processing]
        B1[Data accumulates over hours] --> B2[Scheduled job runs] --> B3[Process the whole pile at once] --> B4[Results, later]
    end
    subgraph STREAM[Stream Processing]
        S1[Event arrives] --> S2[Process immediately] --> S3[Result, now]
        S1b[Next event arrives] --> S2
    end

This diagram contrasts the two approaches. In batch processing, data accumulates over hours, then a scheduled job runs and processes the whole pile at once, producing results later. In stream processing, each event is processed immediately as it arrives, producing a result right away, continuously, event after event.

The difference isn't just timing - it's a different shape of system, with different strengths, costs, and failure modes. Let me take each one honestly.

Batch processing: powerful, patient, and cheaper

Batch is the older approach, and for a long time it was the only approach. You schedule a job - say, every night at 2 a.m. - and it churns through everything that accumulated since the last run. The nightly job that computes "how many hours of content were watched yesterday, by region" is a batch job. It doesn't need yesterday's number until this morning, so there's no reason to compute it continuously.

What batch is genuinely good at:

The price you pay for all that is latency. With batch, results are always stale by design - as old as the gap since the last run. If your job runs nightly, your answers can be up to a day old. For "yesterday's total watch hours," that's completely fine. For "is this account being used fraudulently right now," it's useless. A whole day of fraud would happen before the nightly job even noticed.

Important: Batch processing's defining trade-off: high throughput and low cost, but high latency. Results are only as fresh as your last scheduled run. This is perfect when you don't need up-to-the-second answers - reporting, analytics, billing, training data prep - and a poor fit when freshness is the whole point.

Stream processing: immediate, alive, and harder

Stream processing flips the model. Instead of waiting, you process each event as it arrives, so results are continuously up to date. The system is always on, always consuming the incoming flow.

The early mistake I made - more than once - was assuming that because batch was simpler and cheaper, I should default to it and only reach for streaming when forced. That's backwards in a world that increasingly expects things to happen now. When a viewer hits play and the system needs to update "continue watching" instantly, when a security system needs to flag a suspicious login as it happens, when a dashboard needs to show what's happening this second - batch simply cannot do it. The latency is structural; no amount of tuning a nightly job makes it real-time.

What stream processing gives you:

And what it costs you:

The hardest lesson I learned at scale is that streaming's difficulty isn't in processing one event - that's easy. It's in processing billions of them correctly when some arrive late, some arrive twice, and the machine doing the processing occasionally dies mid-stream. Getting "exactly-once" correctness in a stream is one of the real challenges of data engineering, and it's why streaming systems like the ones built on Apache Kafka and Apache Flink exist - they solve those hard problems so you don't have to rebuild them.

How I actually choose

After all the mistakes, here's the decision that matters: how fresh does the answer need to be?

If the answer can be hours or a day old without anyone caring, use batch. It's simpler, cheaper, and higher-throughput, and you should not pay the complexity tax of streaming for data nobody needs in real time. Reporting, analytics, billing runs, generating training datasets, periodic aggregations - batch all the way.

If staleness defeats the purpose - if a delayed answer is a wrong answer - use streaming. Fraud detection, real-time dashboards, live recommendations, monitoring and alerting, anything a user experiences as "instant." The complexity is worth it because there's no alternative that meets the requirement.

Batch Stream
Processes Accumulated data, on a schedule Each event, as it arrives
Latency High (hours/days) Low (ms/seconds)
Throughput Very high High, but per-event overhead
Complexity Lower Higher
Cost Lower (runs and stops) Higher (always on)
Best for Reporting, analytics, billing Real-time: fraud, dashboards, alerts

A concrete scenario: the same data, two jobs

Let me make this concrete with the kind of decision I actually face. Take a single stream of viewing events. Two different questions need answering from it:

Question one: "Update this user's 'Continue Watching' row the moment they stop a show." This has to be instant - a viewer who stops mid-episode and opens the app on another device expects to see it right there. A delay of even a few minutes feels broken. This is a freshness requirement, so it goes to the streaming path: the stop event is processed the moment it arrives, and the user's state updates within seconds.

Question two: "How many total hours of this title were watched last week, by country?" This needs to be complete and accurate, but nobody needs it in real time - it feeds business reporting reviewed periodically. This is a completeness requirement, so it goes to the batch path: a scheduled job aggregates the full week's events at once, efficiently, and produces precise totals.

Same events, two completely different processing decisions, driven entirely by what each answer needs to be. This is the daily reality of data engineering at scale, and it's why the freshness-versus-completeness question is the one I keep coming back to.

Micro-batching: the middle ground

There's a useful middle option worth knowing: micro-batching. Instead of processing one event at a time (pure streaming) or a huge daily pile (pure batch), micro-batching processes small batches very frequently - say, every few seconds. It's a hybrid that gets most of streaming's freshness with some of batch's efficiency, by grouping a few seconds of events and processing them as a tiny batch.

This blurs the line between batch and stream usefully. Some popular processing frameworks are built around micro-batching precisely because it's a practical compromise: near-real-time latency (a few seconds, not milliseconds) with the throughput benefits of processing in groups. When you don't need millisecond freshness but a daily batch is far too stale, micro-batching often hits the sweet spot. It's a reminder that batch and stream aren't a strict binary - there's a spectrum, and micro-batching lives in the middle.

A mistake I see constantly

The most common mistake I see - and made myself - is reaching for streaming because it sounds more advanced, when the requirement doesn't actually need it. Streaming is harder to build, harder to operate, and more expensive to run. If the answer can be a few hours old without anyone caring, building a complex always-on streaming system for it is wasted effort and ongoing operational burden. The discipline is to start from the freshness requirement and earn your way into streaming, not to default to it because it's the impressive choice. Equally, the opposite mistake - forcing batch onto a genuinely real-time need and then bolting on awkward workarounds - is just as costly.

Both mistakes come from the same root: choosing the processing model by its reputation rather than by the requirement. Anchor every decision to the question's actual freshness need, and you'll reach for the right tool far more often.

The honest answer: you usually need both

Here's what nobody told me early on, and what I'll tell you: most serious systems don't choose. They use both, for different jobs.

At the scale I work at, the same viewing events feed two paths. A streaming path processes them in real time - updating "continue watching," powering live operational dashboards, feeding real-time personalization - because those need to be immediate. And a batch path processes the same events in nightly aggregations for analytics, business reporting, and training the models that drive recommendations - because those need completeness and efficiency, not immediacy. The streaming layer answers "what's happening now," the batch layer answers "what happened, completely and precisely." (This combined pattern is well known in data engineering, and there are named architectures for blending the two - but the core idea is simply: use streaming where freshness matters, batch where completeness and cost matter, over the same underlying event data.)

So the real skill isn't picking a side. It's recognizing, for each question your system needs to answer, whether that question needs an immediate answer or a complete one - and routing it to the right kind of processing. Get that mapping right and a system handling billions of events a day stays both fast where it must be and economical where it can be.

Interview: In a data-heavy system design, showing you understand this distinction is a strong signal. Frame it by freshness: "The fraud check needs to be real-time, so that's stream processing; the daily revenue report can be batch, since it just needs to be complete and is fine being a few hours old." And the senior move is recognizing you'll often want both over the same data - a streaming path for immediate needs and a batch path for complete, efficient aggregation. That's how real large-scale data systems are actually built.

A word on named architectures

The pattern of running streaming and batch paths over the same data has names you'll encounter. The Lambda architecture runs both a batch layer (for complete, accurate results) and a speed/streaming layer (for fresh, approximate results), then merges them - giving you accuracy and freshness at the cost of maintaining two code paths. The Kappa architecture simplifies this by treating everything as a stream and reprocessing the stream when you need to recompute, avoiding the duplicate batch code. You don't need to memorize these, but knowing they exist signals you understand the core tension they address: how to get both the freshness of streaming and the completeness of batch without maintaining two entirely separate systems. The practical reality, whatever you call it, is the one I keep returning to - route each answer to the processing it needs, and find the least painful way to do both where you need both.

What I'd tell my earlier self

The version of me building data pipelines in Lagos and Amsterdam reached for batch by default and bolted on real-time features awkwardly when forced, because streaming felt intimidating. The version of me at Netflix knows the question is simpler and sharper than that: does this answer need to be fresh, or complete? Fresh means stream. Complete-and-cheap means batch. Most real systems need both, for different answers, over the same data.

That reframe - from "which technology" to "what does this answer need to be" - is the whole thing. Get it right and the architecture follows. We go deep on data processing and the systems that power it across System Design Fundamentals.

Go deeper

More articles