AlgoInsight

System Design

Change Data Capture (CDC), Explained by Someone Who Runs It

Your database changes constantly, and half a dozen other systems need to know - the search index, the cache, the data warehouse. CDC is how you stream those changes reliably instead of the fragile hacks most teams reach for first. This is how it actually works in production.

By Admin2026-06-1815 min readchange-data-capturecdcstreamingdata-engineeringkafkasystem-design

Here is a problem that shows up in every system of any size, and that most teams initially solve badly. Your primary database is the source of truth - users, orders, whatever your core data is. But that data needs to be reflected in other places too: a search index so people can search it, a cache so reads are fast, a data warehouse so analysts can query it, maybe another service that needs to react when something changes. So how do all those other systems stay in sync with the database as it constantly changes?

The instinct most teams reach for is some combination of polling the database for changes, or writing to the database and the other systems at the same time ("dual writes"). Both are fragile, and I've watched both cause subtle, maddening data inconsistencies. The robust answer is Change Data Capture (CDC) - capturing every change to the database as a stream of events that other systems consume. I run CDC pipelines feeding downstream systems from operational databases, and this is how it actually works, why the naive alternatives fail, and where it shines.

The problem CDC solves

Let me make the problem concrete. Your database is changing all the time - rows inserted, updated, deleted. Several other systems need to reflect those changes:

So the question is: how do those systems learn about every change to the database, reliably, without missing any?

The two naive approaches, and why they fail:

CDC solves this cleanly by going to the source of truth that the database already maintains.

What CDC is: read the database's own change log

Here's the elegant core of CDC. Databases already keep an internal, ordered log of every change they make - the transaction log (also called the write-ahead log, or binlog, depending on the database). This log exists for the database's own purposes (crash recovery, replication), and it is a complete, ordered, authoritative record of every insert, update, and delete.

Change Data Capture reads that change log and turns each change into an event that other systems can consume. Instead of polling or dual-writing, you tap the database's own definitive record of what changed, and stream those changes out. Every insert, update, and delete becomes an event - "user 123 was updated, here are the new values," "order 456 was deleted" - published as a stream.

Because you're reading the database's authoritative log, you capture every change, in order, exactly as it happened - including deletes, which polling struggles with. And you do it without adding query load to the database (reading the log is cheap and is what replication does anyway). This is why CDC is robust where polling and dual writes are fragile: it's sourced from the database's own complete record of truth.

graph LR
    APP[Application] --> DB[(Primary Database)]
    DB --> LOG[Transaction log<br/>every insert/update/delete]
    LOG --> CDC[CDC reads the log]
    CDC --> STREAM[Stream of change events]
    STREAM --> SEARCH[Search index]
    STREAM --> CACHE[Cache]
    STREAM --> WH[Data warehouse]
    STREAM --> SVC[Other services]

This diagram shows how Change Data Capture works. The application writes to the primary database as normal. The database maintains a transaction log of every insert, update, and delete. CDC reads that log and turns each change into a stream of change events. Downstream consumers - a search index, a cache, a data warehouse, and other services - each consume the stream and update themselves accordingly, staying in sync with the database without polling or dual writes.

How it flows in production

In a real pipeline, the change events from CDC are usually published to a streaming platform - in my world, that's typically Apache Kafka - which acts as the backbone. The database's changes flow into the CDC system, which emits them as events onto a stream, and each downstream consumer subscribes to that stream and updates itself. The search index consumes the changes and reindexes; the cache consumes them and invalidates stale entries; the warehouse consumes them and loads the updates; a service consumes them and reacts.

This is a beautiful pattern because it's decoupled and reliable. The database doesn't know or care who's consuming its changes - it just writes to its log as always. New consumers can be added without touching the database or the application: want a new system to stay in sync? Subscribe it to the change stream. And because the stream is durable (the events persist), a consumer that falls behind or restarts can catch up from where it left off, without losing changes. This connects directly to stream processing and to the pub-sub and message-queue patterns - CDC turns a database's changes into exactly the kind of event stream those patterns thrive on.

Important: The conceptual leap that makes CDC click: a database isn't just a place data sits - it's a continuous stream of changes. Every insert, update, and delete is an event, and the transaction log is that event stream, already recorded. CDC simply exposes it. Once you see your database as a source of change events rather than a static store, a whole class of synchronization problems - keeping caches, indexes, warehouses, and services in sync - turns into "consume the change stream," which is far more robust than polling or dual writes.

A worked example: a change flows through the pipeline

Let me trace a single change end to end, because seeing it concretely makes CDC click. Suppose a user updates their email address in your app.

The application does one simple thing: it updates the user's row in the primary database. That's it - the application doesn't notify the search index, doesn't touch the cache, doesn't write to the warehouse. It just does the one database write it always did.

Behind the scenes, the database records that update in its transaction log, as it does for every change. The CDC system, continuously reading that log, sees the update and emits a change event: "user 123 updated, email changed to the new value." This event is published to the streaming platform.

Now the downstream consumers, each subscribed to the stream, react independently. The search-index consumer sees the change and re-indexes user 123 so searches reflect the new email. The cache consumer sees it and invalidates (or updates) the cached copy of user 123 so reads don't serve the old email. The data-warehouse consumer loads the change so analytics are current. A notification service might see it and send a "your email was changed" confirmation.

One application write, and every dependent system updated itself reliably and in order - with no dual writes, no polling, and no coupling between the application and those downstream systems. The application didn't even know they exist. That is the elegance of CDC: the database write is the only thing the application does, and the change stream fans that single fact out to everything that needs it.

Log-based vs query-based CDC

Not all CDC is equal, and the distinction matters. There are two broad approaches:

Knowing that log-based CDC is the robust approach - and why (complete, ordered, low-overhead, captures deletes) - versus query-based as the simpler-but-flawed fallback is part of understanding CDC properly. Whenever you can read the log, you should.

CDC and the outbox pattern

A related pattern worth knowing, because it solves the dual-write problem from a different angle: the transactional outbox. When a service needs to both update its database and publish an event reliably (the dual-write problem), instead of writing to two places, it writes the event into an "outbox" table in the same database transaction as its data change. Because both writes are in one transaction, they succeed or fail together - no inconsistency. Then CDC (or a poller) reads the outbox table and publishes those events to the stream. The outbox pattern uses CDC-style thinking to make event publishing atomic with the data change, neatly sidestepping the dual-write trap. It's a common companion to CDC in event-driven systems, and recognizing it signals you understand how to publish events reliably alongside database changes.

A note from running it at scale

Running CDC across many operational databases feeding many downstream systems, the thing I've come to appreciate most is the decoupling. When a new team needs the data - a new analytics use case, a new service that reacts to changes - they don't ask the owning team to add code to publish events, and they don't build a fragile polling job against someone else's database. They subscribe to the existing change stream. The change stream becomes a shared, reliable backbone that any number of consumers can tap without burdening the source database or its owners. At scale, that decoupling is worth as much as the reliability - it lets data flow wherever it's needed without every new need turning into a coordination project. CDC, done well, turns "keep these systems in sync" from a recurring source of bugs and meetings into a solved piece of infrastructure.

Where CDC shines

The use cases where I reach for CDC:

The unifying theme: anywhere you need other systems to stay in sync with a database's changes, CDC is the robust, decoupled way to do it.

The challenges, honestly

CDC is powerful, but I'll be honest about what it takes to run well:

None of these are reasons to avoid CDC - they're the realities of running it, and mature tooling (the CDC platforms and streaming systems built for this) handles much of it. But they're why CDC is a deliberate piece of infrastructure, not something you bolt on casually.

Interview: CDC is a strong concept to bring up when a system design involves keeping multiple data stores in sync - a database plus a search index plus a cache, for instance. Proposing CDC over polling or dual writes signals real data-engineering maturity: "To keep the search index and cache in sync with the database, I'd use change data capture - stream the database's changes as events so the downstream stores update reliably, rather than polling or risking dual-write inconsistency." Knowing why dual writes are fragile (non-atomic, silent drift) and that CDC reads the database's own change log is exactly the depth that distinguishes someone who's run real data systems.

The bottom line

Change Data Capture solves the universal problem of keeping other systems - search indexes, caches, data warehouses, downstream services - in sync with a database that's constantly changing. The naive approaches (polling, dual writes) are inefficient and fragile, prone to silent data drift. CDC instead reads the database's own transaction log - its complete, ordered, authoritative record of every change - and streams each insert, update, and delete as an event that downstream systems consume. It's decoupled (the database doesn't know its consumers), reliable (sourced from the database's own truth, durable in the stream), and extensible (add consumers by subscribing).

The mental shift is seeing a database not as a static store but as a continuous stream of change events - and CDC as the way to expose that stream. Run it well, and a whole class of synchronization problems becomes "consume the change stream." We cover data synchronization, streaming, and the patterns behind them across System Design Fundamentals.

Go deeper

More articles