Free lesson · System Design Fundamentals

Sync vs Async Communication

Easy9 min readsystem-designhlddistributed-systems

Lesson 9.1 - Sync vs Async Communication

The first lesson of Module 9.

In a synchronous call, the caller sends a request and blocks - waiting idle until the response arrives. The caller can't do anything else during this time. If the downstream service takes 3 seconds to respond, the caller waits 3 seconds. If the downstream service is down, the caller gets an error. If 10 services are called in sequence, the total time is the sum of all 10 response times. Synchronous communication is simple, predictable, and works well for operations where the caller needs an immediate answer - "what is user 42's balance?" But it creates tight coupling: the caller's performance depends on the callee's performance, the caller's availability depends on the callee's availability, and scaling the caller requires scaling the callee proportionally.

Asynchronous communication breaks this coupling. The caller sends a message and moves on without waiting for processing to complete. The message is stored in an intermediary (a queue, a topic, a log) that absorbs the difference between production and consumption rates. The consumer processes the message on its own schedule. The producer and consumer are decoupled in three dimensions: time (the consumer doesn't need to process immediately), availability (the consumer can be down when the message is sent - the queue holds it), and scale (producer and consumer scale independently).

The Two Models

flowchart TD
    subgraph Sync["Synchronous (Request-Response)"]
        SA["Service A"]
        SB["Service B"]
        SA -->|"1. Request\n(A waits...)"| SB
        SB -->|"2. Response\n(A unblocks)"| SA
        Note1["A is BLOCKED until\nB responds.\nA's latency includes B's latency.\nIf B is down, A fails."]
    end
    
    subgraph Async["Asynchronous (Message-Based)"]
        AA["Service A\n(Producer)"]
        Q["Queue / Topic\n(Intermediary)"]
        AB["Service B\n(Consumer)"]
        AA -->|"1. Send message\n(A continues immediately)"| Q
        Q -->|"2. Deliver message\n(when B is ready)"| AB
        Note2["A is NOT blocked.\nA doesn't wait for B.\nIf B is down, queue holds\nthe message.\nA and B scale independently."]
    end

    style SA fill:#fb7185,stroke:#f43f5e,color:#000
    style AA fill:#34d399,stroke:#10b981,color:#000
    style Q fill:#fbbf24,stroke:#f59e0b,color:#000

A side-by-side comparison of synchronous and asynchronous communication. In the synchronous model, Service A sends a request to Service B and waits - it's blocked, doing nothing, until B responds. A's latency includes B's full processing time. If B is slow, A is slow. If B is down, A gets an error. They're tightly coupled: A depends on B's performance, availability, and capacity. In the asynchronous model, Service A sends a message to an intermediary - a queue or topic. A continues immediately without waiting. The queue stores the message. Service B picks it up when it's ready - maybe immediately, maybe later. If B is down, the message waits in the queue until B recovers. A never knows or cares about B's state. They're decoupled in time (B processes later), availability (B can be temporarily down), and scale (A can produce faster than B consumes - the queue buffers the difference). The intermediary - the queue - is the key architectural component. It absorbs the mismatch between production and consumption rates, provides durability (messages persist through failures), and enables independent scaling of producers and consumers.

When to Use Synchronous

Synchronous communication is the right choice when:

1. The caller needs the result immediately. GET /users/42/balance - the client needs the balance NOW to display it. There's nothing to process asynchronously; the data must be fetched and returned in the same request. Any read operation that populates a UI is inherently synchronous.

2. The operation must be transactional. "Transfer

3. The downstream service is fast and reliable. If Service B responds in 5ms with 99.99% availability, the overhead of asynchronous messaging (queue writes, message serialization, consumer processing) adds more complexity than the synchronous call's coupling risk justifies.

4. The call chain is short. A → B (2 services) with synchronous calls is simple. A → B → C → D → E (5 services) with synchronous calls compounds latency and failure probability - each link adds latency and reduces reliability (M2.2's series availability calculation). Long synchronous chains are where async starts winning.

When to Use Asynchronous

Asynchronous communication is the right choice when:

1. The work takes a long time. Sending an email, processing a video, generating a PDF report, resizing images, running a batch computation. The user doesn't want to stare at a spinner for 30 seconds. Accept the request synchronously ("your report is being generated") and process asynchronously. Notify when done (via SSE, WebSocket, or polling - M8).

2. The downstream service is unreliable. If Service B has outages, synchronous calls to B cause A to fail during B's outages. With a queue, A drops messages into the queue and B processes them when it's back. B's outage doesn't affect A. The queue absorbs the disruption.

3. You need to absorb traffic spikes. A flash sale generates 10x normal order volume. The order processing service can handle 1x. With synchronous calls, the 9x overflow fails or queues internally. With a message queue, all 10x of orders are accepted into the queue immediately (fast writes), and the order processing service works through them at its capacity. The queue is the buffer - it smooths the spike over time.

4. Multiple consumers need the same event. "Order placed" needs to trigger: email confirmation, inventory update, analytics event, loyalty points, fraud check. Synchronous: the order service calls 5 services sequentially (slow, fragile - any failure blocks the chain). Async pub/sub (9.3): the order service publishes one event; 5 consumers each receive it independently. Adding a 6th consumer (recommendation update) requires zero changes to the order service.

5. Producer and consumer need to scale independently. The API tier handles 10,000 requests per second. The email service sends 100 emails per second. Synchronous coupling means the API tier is limited by the email service's capacity. With a queue, the API tier writes 10,000 messages per second to the queue, and the email service consumes at its own rate - with 100 consumers, it handles 10,000/second. Producer and consumer scale independently based on their own bottlenecks.

The Trade-Offs

The common mistake: making everything async because it sounds "more scalable." Async adds significant complexity - message ordering, exactly-once delivery, dead letter queues, monitoring message lag, debugging distributed message flows. Use synchronous as the default. Switch to async when you have a specific reason - long processing, unreliable downstream, traffic spikes, fan-out, independent scaling.

The Async Building Blocks

The remaining lessons in this module cover the specific patterns and tools:

flowchart TD
    Producer["Producer\n(your service)"]
    
    subgraph Patterns["Async Patterns"]
        Queue["Message Queue (9.2)\nPoint-to-point\nOne producer → one consumer\nRabbitMQ, SQS"]
        PubSub["Pub/Sub (9.3)\nFan-out\nOne event → many consumers\nKafka, SNS, Redis Pub/Sub"]
        CDC["CDC (9.4)\nDatabase changes as events\nDebezium, Kafka Connect"]
    end
    
    subgraph Concerns["Cross-Cutting Concerns"]
        Delivery["Delivery Semantics (9.5)\nAt-most-once, at-least-once,\nexactly-once"]
        DLQ["Dead Letter Queue (9.6)\nHandling unprocessable\nmessages"]
    end
    
    Consumer["Consumer\n(downstream service)"]
    
    Producer --> Patterns --> Consumer
    Patterns --> Concerns

    style Queue fill:#34d399,stroke:#10b981,color:#000
    style PubSub fill:#fbbf24,stroke:#f59e0b,color:#000
    style CDC fill:#a78bfa,stroke:#8b5cf6,color:#000

A diagram showing the async communication landscape covered in this module. The producer - your service - sends messages through one of three patterns. Message queues for point-to-point delivery - one producer sends a message, one consumer processes it. Used for task dispatch, background jobs. Pub/sub for fan-out - one event goes to many consumers. Used for event-driven architectures where multiple services react to the same event. CDC for streaming database changes as events - every INSERT, UPDATE, DELETE becomes a message. Used for keeping secondary systems in sync. Two cross-cutting concerns apply to all patterns: delivery semantics - the guarantees about whether messages are delivered at-most-once, at-least-once, or exactly-once - and dead letter queues - the mechanism for handling messages that can't be processed after repeated attempts.

A Brandon Story: The Email That Took Down Checkout

A team I'd advised had a checkout flow: when a user placed an order, the checkout service (1) saved the order to the database, (2) called the email service synchronously to send a confirmation email, and (3) returned the order confirmation to the user. Steps 1 and 3 took 50ms combined. Step 2 - the synchronous email call - took 200-500ms (connecting to the SMTP provider, rendering the template, waiting for delivery confirmation).

User-visible checkout latency: 250-550ms. Acceptable, but not great. Then the SMTP provider had a 15-minute outage. The email service returned 500 errors. Because the call was synchronous, every checkout attempt failed during the outage. Not because orders couldn't be saved - the database was fine. Because the email call failed, the entire checkout flow threw an error. Users saw "checkout failed" and abandoned their carts. Revenue loss during the 15 minutes: approximately $47,000.

The fix: decouple email from checkout. The checkout service saves the order and publishes an order.placed event to a message queue (SQS). The email service consumes from the queue and sends the confirmation. The checkout service returns the order confirmation immediately - 50ms, no email dependency.

Results:

• Checkout latency: 550ms → 50ms (5x improvement)
• Checkout availability: decoupled from email service availability. During the next SMTP outage, orders continued successfully. Confirmation emails were delayed by 15 minutes (the queue held them until the email service recovered) - but orders were placed, revenue was captured, and users saw "order confirmed" instantly.
• The queue absorbed 15 minutes of backed-up emails and the email service processed them at its normal rate over the following 5 minutes after recovery. No manual intervention.

Lessons:

• Synchronous calls to non-critical downstream services (email, analytics, notifications) create fragile dependencies. The checkout's availability shouldn't depend on the email provider's availability.
• Async decoupling trades immediate delivery for resilience. The email is delayed by seconds (or minutes during outages) - but the checkout never fails because of email.
• The queue acts as a buffer, shock absorber, and reliability layer. It holds messages during downstream outages and delivers them after recovery.
• Ask for every synchronous call in the critical path: "does the caller NEED this result right now?" If not - make it async.
• $47,000 in lost revenue from a 15-minute email outage. The SQS queue costs $0.40/month. The ROI of decoupling is extreme.

Six Takeaways from This Lesson

1. Synchronous: caller sends request and waits. Simple, immediate, tightly coupled. Default for reads and transactional operations.

2. Asynchronous: caller sends message and continues. Complex, eventual, loosely coupled. Use for long work, unreliable downstreams, traffic spikes, fan-out, independent scaling.

3. The queue is the key component. It decouples producer from consumer in time (process later), availability (survive outages), and scale (produce and consume at different rates).

4. Don't make everything async. Async adds ordering, delivery, and debugging complexity. Use synchronous as the default; switch to async for specific reasons.

5. Non-critical synchronous calls in the critical path are the #1 coupling mistake. Email, analytics, notifications - if it's not needed for the response, make it async.

6. The trade-off: synchronous = simple + immediate + coupled. Asynchronous = complex + eventual + decoupled. Choose based on whether the caller needs the result now.

Try It

Exercise: Warmup (15 minutes). In a system you work with, list 5 synchronous calls in the critical path. For each, answer: does the caller need the result immediately? Could it be async? What would the user experience change be?

Exercise: Core (25 minutes). Design the order-placement flow for an e-commerce system. The flow must: save the order (sync - needs confirmation), charge the card (sync - needs success/failure), send confirmation email (async?), update inventory (sync or async?), notify the warehouse (async?). Justify each choice.

Exercise: Stretch (30 minutes). Draw an architecture diagram showing a checkout service that publishes order events to a queue, with three async consumers: email, analytics, and inventory. Label: what happens when each consumer is down for 10 minutes? How does the queue handle the backlog?

What's Next

Lesson 9.2 covers message queues - the point-to-point async pattern where one producer sends a message and one consumer processes it. The workhorse of background job processing.

Back to System Design Fundamentals