Free lesson · System Design Fundamentals

The Problem with Distributed Transactions

Medium10 min readsystem-designhlddistributed-systems

Lesson 22.1 - The Problem with Distributed Transactions

The first lesson of Module 22.

A customer clicks "Place Order" on an e-commerce platform. Behind the scenes, this single button click must trigger changes across multiple independent systems: the Inventory Service must decrement the product count from 5 to 4, the Payment Service must charge $99.99 to the customer's credit card, the Order Service must create an order record with status "confirmed," and the Notification Service must send a confirmation email. All four operations must succeed together. If the payment fails, the inventory must not be decremented. If the inventory is already at 0, the payment must not be charged. If the order record is created but the payment fails, the order must be rolled back.

On a single database, this is trivial - a single transaction wraps all four operations:

BEGIN;
  UPDATE inventory SET quantity = quantity - 1 WHERE product_id = 42 AND quantity > 0;
  INSERT INTO payments (customer_id, amount) VALUES (7, 99.99);
  INSERT INTO orders (customer_id, product_id, status) VALUES (7, 42, 'confirmed');
  -- Notification is triggered by the application after COMMIT
COMMIT;

If any statement fails, the entire transaction rolls back. ACID guarantees (M12.3) handle everything - atomicity ensures all-or-nothing, isolation prevents other transactions from seeing intermediate state, durability ensures the committed result survives crashes.

But in a microservices architecture, each service has its own database. The Inventory Service has a PostgreSQL instance for inventory. The Payment Service has a separate PostgreSQL instance (or uses Stripe's API - an external system entirely). The Order Service has its own database. There is no single BEGIN...COMMIT that spans all four databases. Each database's transaction is independent - committing on the Payment Service's database doesn't atomically commit on the Inventory Service's database.

This is the distributed transaction problem: how do you make multiple independent systems either ALL commit or ALL roll back, when they can't share a single transaction?

Why It's Fundamentally Hard

The Two Generals Problem

The distributed transaction problem is, at its core, an instance of the Two Generals Problem - a fundamental impossibility result in distributed computing (proven by E.A. Akkoyunlu, 1975).

Two armies (General A and General B) are positioned on opposite sides of an enemy city. They must coordinate a simultaneous attack - if only one army attacks, it will be destroyed. They can only communicate by sending messengers through the enemy-held valley. Messengers can be captured (messages can be lost).

flowchart LR
    subgraph GA["General A"]
        A["'I'll attack at dawn.\nDo you agree?'"]
    end
    
    subgraph Valley["Enemy Valley"]
        M1["Messenger 1 →\n(might be captured!)"]
        M2["← Messenger 2\n(might be captured!)"]
        M3["Messenger 3 →\n(might be captured!)"]
    end
    
    subgraph GB["General B"]
        B["'I agree. But did A\nget my agreement?\nShould I send\nconfirmation of\nmy agreement?'"]
    end
    
    GA -->|"sends"| M1 -->|"delivers?"| GB
    GB -->|"sends"| M2 -->|"delivers?"| GA
    GA -->|"sends"| M3 -->|"delivers?"| GB

    style Valley fill:#fb7185,stroke:#f43f5e,color:#000

A diagram illustrating the Two Generals Problem. General A wants to coordinate an attack with General B. General A sends a messenger through the enemy valley with the message "I will attack at dawn - do you agree?" But the messenger might be captured - General A doesn't know if General B received the message. General B receives the message and sends a messenger back: "I agree." But this messenger might also be captured - General B doesn't know if General A received the agreement. So General A sends a third messenger: "I got your agreement - attack is confirmed." But General B doesn't know if this confirmation was received. The chain never ends: each side needs confirmation that their last message was received, but the confirmation itself can be lost. No finite number of messages can guarantee that both generals know the other will attack. This is proven to be impossible - no protocol, no matter how many messages are exchanged, can guarantee coordination over an unreliable communication channel. The relevance to distributed transactions is direct: General A is the coordinator saying "commit this transaction." General B is a participant saying "I can commit." The message between them (the network) can be lost. The coordinator can never be 100% certain that the participant received the "commit" message and actually committed. And the participant can never be 100% certain that the coordinator decided to commit (what if the coordinator crashed after sending "commit" to participant 1 but before sending it to participant 2?).

The implication for distributed transactions: there is no protocol that guarantees ALL participants will reach the same decision (commit or abort) when messages can be lost and machines can crash. Every distributed transaction protocol (2PC, 3PC, Saga) is a practical compromise - providing useful guarantees under specific assumptions while acknowledging that some failure scenarios can leave the system in an uncertain state.

The Three Failure Scenarios That Break Everything

Failure 1: Participant crashes after voting YES but before receiving the commit decision. The participant has voted "yes, I can commit" and has prepared (locked resources, written to WAL). But it crashes before hearing the coordinator's final decision. When it recovers, it doesn't know: did the coordinator decide COMMIT (in which case the participant should commit) or ABORT (in which case the participant should roll back)? The participant is stuck - it can't commit (maybe the coordinator aborted) and it can't abort (maybe the coordinator committed and other participants have committed). It must wait for the coordinator to come back online and tell it the decision. This is the uncertainty period - and during this period, the participant holds locks on the data, blocking other transactions.

Failure 2: Coordinator crashes after sending PREPARE but before sending COMMIT/ABORT. All participants have voted YES. They're waiting for the coordinator's decision. The coordinator crashes. Nobody knows the decision. The participants can't commit (maybe the coordinator was about to abort). They can't abort (maybe the coordinator was about to commit and another participant already committed). They're all stuck, holding locks, blocking other transactions, waiting for a coordinator that might not come back for hours. This is the blocking problem of 2PC - the most severe practical limitation.

Failure 3: Network partition between coordinator and some participants. The coordinator decides COMMIT and sends the decision to Participants A and B. But the message to Participant C is lost (network partition). Participants A and B commit. Participant C doesn't receive the decision - it's in the uncertainty period, holding locks. If the partition lasts an hour, Participant C holds locks for an hour. If Participant C's administrator decides to manually abort (to release the locks), the system is inconsistent: A and B committed, C aborted. The atomic all-or-nothing guarantee is violated.

Why "Just Use a Single Database" Is Often the Right Answer

Before diving into distributed transaction protocols, consider: do you actually NEED distributed transactions?

Many microservice architectures create distributed transaction problems unnecessarily. If the Inventory, Payment, and Order tables were in the SAME database (a monolithic database), the entire "Place Order" operation would be a single local transaction - fast, simple, ACID-compliant, no distributed coordination needed.

The honest assessment:

The microservice premium: splitting into separate databases buys you independent scaling, independent deployment, technology diversity (PostgreSQL for orders, Cassandra for analytics), and fault isolation (one service's database failure doesn't crash others). But it costs you distributed transaction complexity. Many teams pay this cost prematurely - splitting into microservices with separate databases before they have the scale, team size, or operational maturity to handle the complexity. The result: the team spends more time on distributed transaction management than on product features.

The rule: keep data that participates in the same transaction in the same database. Split databases only when you have a concrete operational reason (different scaling needs, different consistency models, team autonomy boundaries) - not because "microservices are the right architecture."

The Four Approaches to Distributed Transactions

When you genuinely need to coordinate across multiple databases or services, four approaches exist - each with different trade-offs:

Approach 1: Two-Phase Commit (2PC) - Lesson 22.2

A coordinator orchestrates the transaction across all participants. Phase 1: PREPARE (ask all participants if they can commit). Phase 2: COMMIT or ABORT (tell all participants the decision). Strong guarantee: atomic commit across all participants. Severe limitation: the coordinator is a single point of failure - if it crashes, participants block indefinitely.

Approach 2: Three-Phase Commit (3PC) - Lesson 22.3

An improvement over 2PC that adds a PRE-COMMIT phase, attempting to eliminate the blocking problem. The participant can make a decision even if the coordinator crashes. But 3PC doesn't work correctly under network partitions - a partitioned participant might make the wrong decision. Rarely used in practice.

Approach 3: Saga Pattern - Lesson 22.4

Abandon atomicity at the infrastructure level. Instead, model the distributed transaction as a sequence of local transactions, each with a compensating action (an undo operation). If step 3 fails, execute compensating actions for steps 2 and 1 (reverse the changes). The system is eventually consistent - during execution, intermediate states are visible. Much simpler than 2PC, no blocking, works with any database. The dominant modern approach.

Approach 4: Outbox Pattern - Lesson 22.5

Solve a specific sub-problem: how to reliably publish an event (to Kafka, to SNS) as part of a database transaction. Write the event to an "outbox" table in the SAME database as the business data (in the same transaction). A separate process reads the outbox table and publishes events to the message broker. This guarantees that the event is published if and only if the transaction commits - no dual-write inconsistency.

flowchart TD
    subgraph Approaches["Four Approaches"]
        TwoPC["22.2 Two-Phase Commit\n\n✅ Atomic across participants\n❌ Coordinator SPOF\n❌ Blocking on failure\n❌ Performance cost"]
        
        ThreePC["22.3 Three-Phase Commit\n\n✅ Non-blocking (in theory)\n❌ Unsafe under partitions\n❌ Rarely used in practice"]
        
        Saga["22.4 Saga Pattern\n\n✅ No coordinator SPOF\n✅ No blocking\n✅ Works with any DB\n⚠️ Eventually consistent\n⚠️ Compensation complexity"]
        
        Outbox["22.5 Outbox Pattern\n\n✅ Reliable event publishing\n✅ Atomicity via local DB tx\n✅ Simple implementation\n⚠️ Requires CDC or poller"]
    end

    style TwoPC fill:#fb7185,stroke:#f43f5e,color:#000
    style ThreePC fill:#fbbf24,stroke:#f59e0b,color:#000
    style Saga fill:#34d399,stroke:#10b981,color:#000
    style Outbox fill:#34d399,stroke:#10b981,color:#000

A comparison of the four distributed transaction approaches. Two-Phase Commit provides true atomicity across participants but has a coordinator single point of failure - if the coordinator crashes, participants block indefinitely holding locks. Three-Phase Commit attempts to fix the blocking problem with an extra phase but is unsafe under network partitions - a partitioned participant might make the wrong commit/abort decision. Rarely used in practice. The Saga pattern abandons atomic commit across services. Instead, it uses a sequence of local transactions with compensating actions (undo operations). If a step fails, previous steps are reversed by executing their compensation actions. The system is eventually consistent - intermediate states are visible during execution - but there is no coordinator SPOF and no blocking. This is the dominant modern approach for microservice architectures. The Outbox pattern solves a specific sub-problem: how to reliably publish an event alongside a database transaction. The event is written to an outbox table in the same database (same local transaction). A separate process reads the outbox and publishes to the message broker. This guarantees at-least-once event delivery exactly when the transaction commits.

The Consistency Spectrum

Distributed transactions exist on a spectrum from strong to weak consistency:

The trend in the industry is toward eventual consistency (Saga + Outbox) and away from strong distributed transactions (2PC). The reasons: 2PC's blocking problem is unacceptable for high-availability systems (a crashed coordinator blocks all participants), and the locking overhead of 2PC degrades throughput in high-concurrency environments. Saga patterns with compensating actions provide "good enough" consistency for most business operations - and the business logic for handling compensations (refunding a payment, restoring inventory) is often needed anyway for non-technical reasons (customer cancellations, returns, disputes).

The exception: financial systems. Banks, payment processors, and financial exchanges often use 2PC (or Google Spanner's Paxos-based distributed transactions) because financial regulations require atomic consistency - a partial transfer (money deducted from one account but not credited to another) is a regulatory violation, not just a UX issue. For these systems, the performance and availability costs of 2PC are acceptable because the correctness requirements are non-negotiable.

Six Takeaways from This Lesson

1. Distributed transactions are needed when one operation must update multiple independent databases/services atomically. "Either all succeed or all roll back" - across system boundaries.

2. The Two Generals Problem proves that perfect coordination over unreliable channels is impossible. Every distributed transaction protocol is a practical compromise, not a perfect solution.

3. Three failure scenarios break distributed transactions: participant crash after voting yes (stuck in uncertainty), coordinator crash after PREPARE (all participants block), network partition (some participants miss the decision).

4. "Just use a single database" is often the right answer. Splitting into separate databases per service creates distributed transaction complexity. Don't split prematurely - split only when you have concrete scaling or operational reasons.

5. Four approaches, each with trade-offs: 2PC (atomic but blocking), 3PC (non-blocking in theory, unsafe under partitions), Saga (eventually consistent, no blocking, compensation complexity), Outbox (reliable event publishing via local transaction).

6. The industry trend is toward Saga + Outbox (eventual consistency) and away from 2PC (strong consistency). Exception: financial systems where regulatory requirements demand atomic consistency.

Try It

Exercise: Warmup (15 minutes). List 5 operations in a system you work on that touch multiple services or databases. For each: what happens if one service succeeds and another fails? Is the current system handling this correctly?

Exercise: Core (25 minutes). Design a "Place Order" flow for an e-commerce platform with 4 services (Inventory, Payment, Order, Notification). Each has its own database. Identify: which pairs of services need atomic coordination? Which can tolerate eventual consistency? Propose an approach (2PC, Saga, or Outbox) for each pair.

Exercise: Stretch (30 minutes). Read the Two Generals Problem proof (link below). Then explain: why can't you solve it by adding more messages? Why is the problem fundamentally unsolvable regardless of the protocol?

What's Next

Lesson 22.2 covers Two-Phase Commit (2PC) - the classical protocol for atomic distributed transactions, its message flow, and its critical failure modes.

Back to System Design Fundamentals