AlgoInsight

System Design

Apache Flink: How Real-Time Processing Actually Works

Everyone says 'real-time stream processing' like it's one thing. It isn't — it's a set of genuinely hard problems about time, state, and correctness. Here's how Flink actually solves them, from someone who runs it in production.

By Admin2026-06-1816 min readflinkstream-processingreal-timedata-engineeringstreamingsystem-design

"Real-time stream processing" gets thrown around like it's a single, simple thing — data comes in, you process it as it arrives, done. The reality, once you run it at scale, is that stream processing is a set of genuinely hard problems wearing a friendly name. What does "now" even mean when events arrive out of order? How do you compute a running total over an infinite stream that never ends? How do you guarantee you didn't double-count when a machine crashes mid-stream? Apache Flink is one of the leading systems for answering these questions, and I work with it on real, high-volume streams. This is how Flink actually makes real-time processing work — the hard parts and how it solves them.

Batch vs. stream: the fundamental shift

First, the mental shift. Batch processing operates on a finite, bounded dataset: you have all the data, you run a job over it, you get a result. Think of processing yesterday's logs — the data is complete and sitting still. Stream processing operates on an unbounded, never-ending flow of data: events arrive continuously, forever, and you process them as they come, producing results continuously rather than once at the end.

This difference is deeper than "faster." With batch, you have the whole dataset, so questions like "what's the total?" have a clear answer — you've seen everything. With a stream, the data never ends, so "what's the total?" needs reframing: the total over what? The last minute? The last hour? You're computing over a moving, infinite flow, which forces you to confront time and boundaries in ways batch never does. Flink is built natively for this unbounded, continuous world (and can treat batch as a special, bounded case of streaming).

graph LR
    SRC[Source: event stream] --> OP1[Transform / filter]
    OP1 --> OP2[Window / aggregate<br/>stateful]
    OP2 --> OP3[Process]
    OP3 --> SINK[Sink: storage / output]

This diagram shows a Flink dataflow. Events enter from a source stream, pass through a transform or filter operation, then through a stateful windowing or aggregation step, then a processing step, and finally out to a sink such as storage or another system. Data flows continuously through this pipeline of operators, processed as it arrives rather than in a single batch at the end.

The hardest problem: event time vs. processing time

Here's the problem that separates people who understand stream processing from people who don't. In a stream, there are two notions of time, and confusing them causes subtle, maddening bugs.

In a perfect world these would be close. In reality they diverge, because events arrive late and out of order. A mobile app loses connectivity and uploads events ten minutes later; network paths differ; a partition lags. So an event that happened at 12:00 might arrive at 12:10, after events that happened at 12:05. If you compute "transactions in the 12:00–12:01 minute" by processing time, you'll miss the late-arriving event that truly belongs in that minute. The answer is usually wrong in ways that matter.

Flink's power is that it handles event time properly — it lets you compute results based on when events actually happened, not when they happened to arrive. This is essential for correctness. Computing "revenue per minute" or "events per user per hour" only makes sense in event time; doing it in processing time gives answers that shift based on network conditions, which is nonsense. The fact that Flink treats event time as a first-class concept is a big part of why it's trusted for correctness-sensitive stream processing.

Watermarks: deciding when "enough" late data has arrived

But event time raises a question: if events can arrive late, how do you ever know you've seen all the events for, say, the 12:00–12:01 minute? You can't wait forever — the stream never ends. Flink's answer is watermarks.

A watermark is Flink's way of tracking progress in event time — essentially a marker that says "I believe I've now seen all events up to event-time T." When a watermark for 12:01 passes, Flink concludes that the 12:00–12:01 window is complete enough to emit its result, even though a straggler might still arrive. Watermarks let Flink balance correctness against latency: wait a bit for late events (more correct), but not forever (you need results eventually). You configure how long to wait based on how late your data realistically arrives.

Watermarks are the elegant mechanism that makes event-time processing practical: they're how Flink decides "now I can compute the result for this time window," handling out-of-order data gracefully without waiting indefinitely. Understanding watermarks is understanding how real stream processing reconciles "events arrive late and unordered" with "I need to produce timely results."

Windowing: computing over an infinite stream

This connects to the next core concept. Since a stream is infinite, you can't compute "the total" — you compute over windows, bounded slices of the stream. Windowing is how you ask finite questions of an infinite flow:

Windows, computed in event time and triggered by watermarks, are how Flink turns an endless stream into a sequence of finite, meaningful results. "Revenue per hour," "active users per 5-minute window," "events per user session" — all of these are windowed aggregations over the stream. Windowing is the workhorse of stream processing, and getting the window type right is how you ask the question you actually mean.

Stateful processing: the stream remembers

Many useful stream computations need state — memory of what's come before. To compute a running count per user, Flink must remember each user's count so far. To detect a pattern across events, it must remember the events leading up to it. Flink is a stateful stream processor: operators can maintain state, updated as each event flows through.

This sounds simple but is genuinely hard at scale, because the state must be: large (potentially huge — state for millions of keys), fast (accessed on every event), and — critically — fault-tolerant. If a machine holding state crashes, you can't just lose the running counts. Which leads to the problem that, to me, is Flink's most impressive feat.

Exactly-once: correctness through crashes

Here's the question that keeps data engineers up at night: in a stream processing system spread across many machines, processing millions of events, what happens when a machine crashes mid-stream? Naively, you'd either lose events (the ones in flight) or double-process them (re-processing after recovery), and both corrupt your results. If you're counting revenue and a crash causes you to double-count a window, your numbers are wrong.

Flink provides exactly-once processing semantics: even through failures, each event affects the results exactly once — not lost, not double-counted. It achieves this through checkpointing. Periodically, Flink takes a consistent snapshot of the entire streaming computation — all the operator state, plus exactly where it was in the input streams — and saves it durably. If a machine crashes, Flink restores from the last checkpoint: it resets the state and rewinds the input to exactly that point, then resumes. Because the state and the input position are restored together consistently, no event is lost or counted twice. The computation continues as if the crash hadn't happened.

This is a remarkable guarantee. Exactly-once-through-failures over an infinite stream of millions of events is the kind of correctness that makes Flink trustworthy for things where the numbers must be right. The checkpointing mechanism — consistent snapshots of state plus stream position — is the engine that delivers it. (It rhymes with how databases use checkpoints and logs for durability; the deep ideas of distributed data systems recur everywhere.)

stateDiagram-v2
    [*] --> Processing
    Processing --> Checkpoint: periodic snapshot
    Checkpoint --> Processing: state + position saved
    Processing --> Crash: machine fails
    Crash --> Restore: load last checkpoint
    Restore --> Processing: resume exactly where snapshot was

This state diagram shows Flink's exactly-once recovery. During normal processing, Flink periodically takes a checkpoint — a consistent snapshot of all operator state plus the exact position in the input streams — and saves it durably, then continues processing. If a machine crashes, Flink restores from the last checkpoint, loading the saved state and rewinding the input to exactly that position, then resumes. Because state and input position are restored together, no event is lost or double-counted.

A concrete example: counting events per minute, correctly

Let me ground all of this in one computation: "count events per user per minute." Sounds trivial; doing it correctly over a real stream uses everything above.

You define a tumbling window of one minute, keyed by user — so Flink maintains a separate count for each user, for each minute. As events flow in, Flink routes each to the right user's state and increments the count for the minute the event actually happened in (event time, from the event's timestamp — not when it arrived). Because events arrive late and out of order, Flink doesn't close the 12:00–12:01 window the instant wall-clock time hits 12:01. It waits until the watermark for 12:01 passes — its judgment that it's seen essentially all events up to that event-time — and only then emits each user's count for that minute. Meanwhile, all those per-user counts are state, checkpointed periodically so a crash doesn't lose them, and the whole thing runs with exactly-once semantics so a failure doesn't double-count.

So a "trivial" per-minute count, done right, is windowing plus event time plus watermarks plus fault-tolerant state plus exactly-once — every concept working together. That's why stream processing is harder than it sounds, and why a system like Flink that handles all of it correctly is worth using rather than rolling your own. The naive version (count by arrival time, keep counts in memory, hope nothing crashes) gives wrong answers the moment data arrives late or a machine fails — which, at scale, is constantly.

Where Flink keeps state: the state backend

A practical detail worth knowing, since state is so central: Flink stores its state in a state backend, and the choice matters at scale. For modest state, it can be kept in memory (fast, but bounded by RAM). For large state — millions of keys, more than fits comfortably in memory — Flink can use a backend that spills state to local disk while keeping it fast, so state can grow large without exhausting memory. Either way, the state is checkpointed to durable storage periodically, so it survives failures regardless of where it lives during processing.

This matters because real stream processing often involves enormous state — imagine maintaining a running aggregate for every user, every device, every session across a massive user base. The ability to hold large state efficiently, spill it when it exceeds memory, and checkpoint it durably is part of what makes Flink usable for serious workloads rather than just small ones. When people are surprised that a stream processor can maintain state for hundreds of millions of keys reliably, the state backend (plus checkpointing) is the answer.

A note on micro-batch vs. true streaming

Worth a brief mention because it comes up: some systems approximate stream processing by running many tiny batch jobs in rapid succession ("micro-batching") — processing small chunks of the stream every few seconds. This works for many cases but introduces a latency floor (you wait for the next micro-batch) and can complicate true event-time handling. Flink processes events as a genuine continuous stream (record by record), which gives lower latency and cleaner event-time semantics. The distinction isn't always decisive — micro-batch is fine for plenty of use cases — but for the lowest-latency, most correct event-time processing, true streaming is the stronger model, and it's part of why Flink is favored for demanding real-time workloads.

Why this matters in production

Put it together and you see why Flink is trusted for serious real-time work. It processes unbounded streams continuously; it handles event time correctly so results reflect when things actually happened, not when they arrived; it uses watermarks to produce timely results despite late, out-of-order data; it computes over windows to ask finite questions of an infinite stream; it maintains large, fast state; and it guarantees exactly-once correctness through failures via checkpointing. Each of those is a hard problem, and Flink solves them together.

In real-time analytics, fraud detection, live dashboards, monitoring, and event-driven processing at scale — anywhere you need to compute correct results over a continuous stream as it happens — this combination is what you need. The "real-time" part is easy to say; the correctness-over-an-infinite-stream-through-failures part is what Flink actually delivers, and it's why it earns its place in demanding pipelines.

Interview: Stream processing comes up in system design whenever a problem needs real-time computation over a continuous flow of events — live analytics, fraud detection, real-time metrics. Demonstrating that you understand the genuinely hard parts — event time versus processing time (handling late, out-of-order events), windowing (finite questions over infinite streams), stateful processing, and exactly-once guarantees through failures — signals real depth. Saying "I'd use a stream processor like Flink, computing windowed aggregations in event time with exactly-once semantics" shows you grasp that real-time processing is about correctness under hard conditions, not just speed.

The bottom line

Apache Flink makes real-time processing work by solving the hard problems underneath the friendly phrase. It processes unbounded streams continuously rather than finite batches. It distinguishes event time (when something happened) from processing time (when it's processed), computing in event time so results are correct despite late, out-of-order data — and uses watermarks to decide when a time window is complete enough to emit. It computes over windows (tumbling, sliding, session) to ask finite questions of an infinite stream. It maintains state so the stream can remember. And it guarantees exactly-once correctness through machine failures via checkpointing — consistent snapshots of state plus stream position.

"Real-time" is the easy part to say; correct, stateful, exactly-once computation over an endless stream is the hard part Flink delivers. Understand event time, watermarks, windows, state, and checkpointing, and you understand how real-time processing actually works. We cover stream processing and real-time systems across System Design Fundamentals.

Go deeper

More articles