AlgoInsight

System Design

The Kafka Outage That Taught Me Distributed Messaging

I thought I understood message queues until the day our pipeline went down and I watched what happens when millions of events have nowhere to go. That outage taught me more about distributed messaging than any book had. Here's what I learned, the hard way.

By Admin2026-06-1816 min readkafkadistributed-messagingmessage-queuestreamingsystem-designdata-engineering

I thought I understood message queues. I'd read the docs, drawn the boxes-and-arrows diagrams, shipped systems that used them. Producers send messages, a broker holds them, consumers process them - what's complicated? Then, during my years working on high-volume data systems, I lived through the kind of incident that rearranges your understanding: our messaging backbone faltered, and I watched, more or less helplessly at first, what actually happens when millions of events have nowhere to go and nowhere to be processed. That day taught me more about distributed messaging than any amount of reading had, because it showed me the why behind every design decision I'd been taking for granted.

This is what I learned, organized around the things the outage made impossible to ignore. If you're preparing for system design interviews, this is the intuition behind every "I'll put a message queue here" you'll ever say.

What a message broker actually does (and why you need one)

Start with the thing the outage clarified first: why you put a message broker between systems at all.

Imagine a service that produces events - user actions, transactions, whatever - and several other systems that need those events. The naive design has the producer call each consumer directly: when something happens, the producer notifies the analytics system, the notification system, the search-index updater, and so on. This works until it doesn't. If one consumer is slow, the producer waits. If a consumer is down, the producer fails or loses the event. If you add a new consumer, you change the producer. The producer is tightly coupled to every consumer, and the whole thing is as fragile as its weakest part.

A message broker (like Kafka) breaks this coupling. The producer sends events to the broker and is done - it doesn't know or care who consumes them. Consumers read from the broker independently, at their own pace. The broker sits in the middle as a durable buffer, decoupling producers from consumers in time and in knowledge. The producer can keep producing even if a consumer is down; consumers can be added without touching producers; a slow consumer doesn't slow the producer. This decoupling is the entire point, and it's why message brokers are foundational to large systems.

I knew all that. What I hadn't felt, until the outage, was how much the entire system silently depends on that broker being there.

graph LR
    P[Producer] --> B[(Message Broker)]
    B --> C1[Consumer: Analytics]
    B --> C2[Consumer: Notifications]
    B --> C3[Consumer: Search index]
    B --> C4[Consumer: Warehouse]

This diagram shows a message broker decoupling a producer from consumers. The producer sends events to the broker and is done - it doesn't contact consumers directly. Multiple consumers - analytics, notifications, a search index, a data warehouse - each read from the broker independently at their own pace. The broker in the middle decouples the producer from the consumers, so a slow or failed consumer doesn't affect the producer, and new consumers can be added without changing the producer.

Lesson 1: the broker is durable for a reason

When our messaging backbone faltered, the first thing I was grateful for - and the first thing I truly understood - was durability. Kafka, and brokers like it, don't just pass messages through and forget them. They persist messages to disk, keeping them for a configured retention period. Messages aren't lost the moment they're delivered; they sit durably in the broker.

This is why, when things went wrong, we hadn't simply lost everything in flight. The events producers had already written were safe on the broker's disk. The durability I'd treated as a checkbox was the reason a bad incident wasn't a catastrophic, unrecoverable data-loss event. A message broker that holds messages durably means a downstream problem is a delay, not a loss - and that distinction, under pressure, is everything. I stopped thinking of the broker as a pipe and started thinking of it as a durable log that happens to let you read from it.

Lesson 2: replication is what keeps the broker itself alive

The deeper realization: the broker decouples everything and holds the durable truth, which makes the broker itself the thing you absolutely cannot afford to lose. So how does the broker avoid being a single point of failure?

Replication. A distributed broker like Kafka doesn't keep your data on one machine - it replicates each piece of data across multiple machines (brokers in a cluster). Each partition of data has a leader and followers, and the followers keep copies. If the machine holding the leader fails, a follower that has an up-to-date copy takes over, and the system keeps going without losing data. This is what makes the broker durable not just against a disk, but against an entire machine dying.

Living through an incident makes you appreciate replication viscerally. The broker is the heart of the system; replication is what gives the heart redundancy. Understanding that a distributed message broker is itself a replicated, fault-tolerant distributed system - not a single magic box - was a turning point in how I designed everything afterward. (See the replication and single-point-of-failure guides.)

Lesson 3: partitions are how it scales

Here's the thing that the volume of the incident drove home. We weren't dealing with thousands of messages - we were dealing with enormous, relentless streams. A single machine couldn't possibly handle that throughput. So how does a broker handle millions of messages per second?

Partitioning. Kafka splits a stream of events (a "topic") into multiple partitions, spread across machines. Each partition is an independent, ordered log, and different partitions live on different brokers. Producers write across partitions, and - crucially - consumers can read partitions in parallel, with different consumers in a group each handling different partitions. This is how the throughput scales: instead of one log bottlenecked on one machine, you have many partitions across many machines, processed in parallel.

The trade-off I learned to respect: ordering is guaranteed within a partition, but not across partitions. So you choose how events map to partitions (often by a key) based on what ordering you actually need. Events that must stay ordered relative to each other (all events for one user, say) go to the same partition by sharing a key. This is exactly the sharding idea applied to a message log, and getting the partitioning right is as consequential here as choosing a shard key is for a database.

graph TD
    T[Topic: events] --> P0[Partition 0<br/>ordered log]
    T --> P1[Partition 1<br/>ordered log]
    T --> P2[Partition 2<br/>ordered log]
    P0 --> CG0[Consumer A]
    P1 --> CG1[Consumer B]
    P2 --> CG2[Consumer C]

This diagram shows Kafka partitioning. A topic of events is split into multiple partitions - partition zero, one, and two - each an independent ordered log living on different machines. Consumers in a group read different partitions in parallel: consumer A reads partition zero, B reads partition one, C reads partition two. This parallelism is how the broker scales to millions of messages per second. Ordering is guaranteed within a partition but not across partitions.

Lesson 4: consumer lag is the metric that matters

Now the part of the outage that actually hurt: consumer lag. Because the broker durably holds messages and consumers read at their own pace, there's a gap between the latest message produced and the latest message a consumer has processed. That gap is lag. In normal operation it's tiny. When a consumer slows down or stops - which is what happened - lag grows. Messages keep arriving and piling up in the broker faster than the consumer can work through them.

Watching lag climb during the incident was watching the problem in a single number. The broker was fine - durably holding everything. The producers were fine - still writing. But a consumer had fallen behind, and the lag told us exactly how far behind, and whether it was catching up or falling further. I learned that monitoring consumer lag is the single most important thing you watch in a messaging system, because it's the early warning that something downstream is struggling, long before data is actually lost (which only happens if lag grows so large that messages age out of the retention window before being read).

Lag also reframed the recovery: because the broker had everything durably, recovery was about getting the consumer healthy and letting it catch up through the backlog - processing the piled-up messages until lag returned to normal. The durable broker turned a potential data-loss disaster into a catch-up problem. That's the whole value proposition of the design, experienced under fire.

Lesson 5: backpressure and the limits of buffering

The last lesson was about the limits. A durable broker buffering a growing backlog is wonderful, but it's not infinite - disk fills, retention windows pass. The broker buys you time, not immunity. If a consumer stays down long enough, the backlog can grow past what you can hold or catch up from.

This is where backpressure matters - the system's ability to signal "slow down" when downstream can't keep up, rather than letting the backlog grow unbounded until something breaks. The incident taught me to design for the consumer falling behind: enough retention and capacity to absorb realistic outages, monitoring to catch lag early, the ability to scale consumers up to chew through a backlog faster, and an understanding of how long the system can tolerate a consumer being down. The broker's buffering is a shock absorber with a real limit, and respecting that limit is part of running messaging at scale.

Offsets: how consumers remember where they are

One mechanism I came to appreciate during recovery, because it's what made catching up possible: offsets. Each partition is an ordered log, and every message in it has a position - an offset. A consumer tracks which offset it has processed up to, and that's how it remembers its place in the stream. When a consumer reads, it picks up from its last committed offset and moves forward.

This simple idea has profound consequences. Because the broker durably retains messages and consumers track their own offset, a consumer that falls behind hasn't lost anything - it just has a lower offset than the latest message, and it catches up by processing forward from where it is. A consumer that crashes and restarts resumes from its last committed offset, not from scratch. And - this is the part that surprised me how much it mattered - you can rewind: reset a consumer's offset to an earlier point to reprocess messages, because they're still there in the durable log. During the incident, offsets were how we reasoned about exactly where each consumer stood and how far it had to go. Understanding that consumers are just cursors moving through a durable, retained log - tracked by offset - is understanding the mechanism beneath all the behavior I described. It's also what makes Kafka replayable, which turns out to be one of its most powerful properties well beyond recovery.

What we changed afterward

The most valuable output of any incident is what you change because of it. After ours, the lessons turned into concrete practice. We monitored consumer lag aggressively, with alerts that fired when lag started climbing abnormally - long before it became dangerous. We made sure retention was generous enough to absorb a realistic outage window, so a consumer being down for a while was survivable rather than catastrophic. We built the ability to scale consumers up quickly to chew through a backlog when lag did grow. And we got disciplined about treating the broker as the durable source of truth it is - designing downstream systems to be able to fall behind and catch up safely, rather than assuming everything always keeps pace.

None of that was new knowledge in the abstract; the outage just converted it from things I'd read into things we did, reflexively, from then on. That conversion - from knowing to doing - is what an incident gives you, and it's why the engineers I trust most with distributed systems are usually the ones who've watched one strain under load and had to reason their way back to health while the lag number climbed.

The intuition the outage left me with

What changed that day wasn't facts - I knew the facts. It was intuition. I came out understanding viscerally that:

Every one of those is something I'd "known" before and only truly understood after watching the system under stress. That's why I tell engineers: you can read about distributed messaging, but the understanding that sticks comes from grasping why each piece exists - what failure it's protecting against. The outage just taught me those whys all at once.

Interview: When a system design problem involves multiple systems reacting to events - and many do - reaching for a message broker (Kafka) and explaining why is a strong move. The reasons to articulate: it decouples producers from consumers (so a slow or failed consumer doesn't break producers), it durably buffers events (so downstream failures are delays, not losses), it partitions for scale (parallel consumption, ordering within a partition), and it's replicated (the broker isn't a single point of failure). Mentioning that you'd monitor consumer lag signals real operational experience. Understanding messaging as decoupling-plus-durable-buffering, rather than just "a queue," is exactly the depth interviewers look for.

The bottom line

A message broker like Kafka decouples producers from consumers through a durable, replicated, partitioned log. Producers write and move on; the broker persists events durably across replicated machines; consumers read independently at their own pace; partitioning spreads the load for massive throughput with ordering guaranteed within each partition. The metric that matters is consumer lag - the gap between produced and processed - because durability turns downstream failures into recoverable catch-up problems rather than data loss, as long as you respect that the broker's buffering buys time, not infinite immunity.

I learned all of that properly the hard way, watching the design protect us under stress. You don't have to live through an outage to absorb it - but understanding why each piece exists, against which failure, is what turns "put a queue here" into real engineering judgment. We cover distributed messaging and event-driven design across System Design Fundamentals.

Go deeper

More articles