System Design

15 Strategies to Reduce Latency in System Design

Latency is what users feel. A system can be correct, scalable, and still feel slow - and slow loses users and money. Here are fifteen proven ways to make systems faster, and when to reach for each.

By Admin2026-06-1813 min readlatencyperformancesystem-designcachingcdnscalability

Latency is the time between a user asking for something and getting it back. It is, more than almost any other metric, what users actually feel. A system can be perfectly correct, infinitely scalable, and beautifully designed - and if it takes three seconds to respond, people leave. Studies across the industry have repeatedly found that every extra hundred milliseconds of delay measurably reduces engagement and revenue. Speed isn't a luxury; it's a feature.

This guide collects fifteen practical strategies for reducing latency, grouped by where in the system they apply. None of them is a silver bullet - each is a tool with a cost - but together they form the toolkit experienced engineers reach for, and they come up constantly in system design interviews.

First, one piece of vocabulary that trips people up.

Latency vs throughput

These are different and often confused. Latency is how long a single request takes (measured in milliseconds). Throughput is how many requests the system handles per second. They're related but distinct: a system can have high throughput and high latency at the same time - processing millions of requests per second while each individual one takes a while, because requests are batched or queued.

When you optimize for latency, you're making each request faster. Sometimes that helps throughput too; sometimes it trades against it. Keep the distinction clear, because an interviewer will.

It also helps to have a feel for the raw numbers. Reading from memory takes nanoseconds. Reading from an SSD takes microseconds. A network round-trip within a data center takes well under a millisecond, but a round-trip across the world takes well over a hundred milliseconds - physics, not engineering, sets that floor. Most latency optimization is really about avoiding the expensive operations: fewer disk reads, fewer network hops, and especially fewer trips across long distances.

graph LR
    A[Where latency hides] --> B[Network<br/>round-trips, distance]
    A --> C[Data<br/>slow queries, disk reads]
    A --> D[Compute<br/>heavy work per request]
    A --> E[Architecture<br/>synchronous waits, coupling]

This diagram shows the four places latency hides in a system: the network, through round-trips and physical distance; the data layer, through slow queries and disk reads; compute, through heavy work done per request; and the architecture, through synchronous waits and tight coupling. The fifteen strategies that follow each attack one of these areas.

Data-layer strategies

1. Caching. The single highest-leverage latency tool. A cache stores the result of an expensive operation so the next request can skip the work entirely. Instead of querying the database, you check a fast in-memory cache first; if the answer is there (a "cache hit"), you return it in microseconds rather than milliseconds. Caching can live at many layers - in the browser, at a CDN, in front of the database, inside the application. The trade-off is staleness: a cache holds a copy of the truth that can fall out of date, so cache invalidation becomes a problem you have to manage.

2. In-memory data stores. Closely related: move hot data out of disk-based databases and into in-memory stores like Redis. Memory is orders of magnitude faster than disk, so data that's read constantly - session info, leaderboards, counters, frequently-accessed records - belongs in memory. The cost is that memory is more expensive and (usually) less durable than disk, so you store only what benefits most.

3. Database indexing. An index is a data structure that lets the database find rows without scanning the entire table. Without an index, finding one record in a million-row table means checking up to a million rows. With the right index, it's a handful of lookups. Indexing is often the difference between a query that takes seconds and one that takes milliseconds. The cost: each index makes writes slightly slower (the index must be updated too) and consumes storage, so you index the columns you actually query on, not everything.

4. Read replicas. Many systems read far more than they write. A read replica is a copy of your database that handles read queries, taking load off the primary. Spreading reads across several replicas means no single database is the bottleneck, and queries return faster. The trade-off is consistency: replicas lag slightly behind the primary, so a read replica may briefly return stale data.

5. Denormalization and precomputation. Instead of computing an answer when it's requested, compute it ahead of time and store it ready to serve. A social media timeline, for example, can be precomputed and stored per-user so that loading it is a simple fast read rather than an expensive on-the-fly assembly. Denormalization (deliberately duplicating data to avoid joins) is the same idea applied to database design - you trade extra storage and write complexity for much faster reads.

Tip: Most latency problems in read-heavy systems are solved by some combination of strategies 1 through 5. Before reaching for exotic optimizations, ask: am I caching what's hot, indexing what I query, and precomputing what's expensive? That covers the majority of real-world cases.

Network-layer strategies

6. Content Delivery Networks (CDNs). A CDN is a network of servers distributed around the world that cache your content close to users. When someone in Tokyo requests an image, it's served from a nearby CDN edge in Tokyo rather than from an origin server in Virginia - turning a 150-millisecond cross-ocean trip into a few milliseconds. CDNs are essential for static content (images, video, scripts, stylesheets) and increasingly used for cacheable API responses too. The win comes directly from defeating physical distance.

7. Geographic data placement. The same principle applied to your whole system: put your servers and data near your users. If most of your users are in Europe, running your infrastructure in a European region cuts the baseline latency for every request. Large global systems run in multiple regions and route each user to the nearest one. The cost is the complexity of keeping data synchronized across regions.

8. Edge computing. A step beyond CDNs: instead of just caching static content at the edge, run actual computation there. Logic that used to require a round-trip to a central server - authentication checks, simple transformations, personalization - runs at an edge location near the user. This removes the round-trip entirely for those operations.

9. Reduce round-trips and payload size. Every network round-trip costs latency, so the fewer the better. Techniques include combining multiple small requests into one, using HTTP/2 or HTTP/3 to multiplex many requests over a single connection, keeping connections alive instead of re-establishing them, and paginating large responses so you fetch only what's needed. A related win is sending less data - return only the fields the client needs rather than entire objects.

10. Compression. Compressing data before sending it over the network means fewer bytes to transmit, which means faster transfers - especially valuable for large text responses and for users on slow connections. The trade-off is CPU time spent compressing and decompressing, which is almost always worth it for anything beyond tiny payloads.

Compute-layer strategies

11. Connection pooling. Opening a new database connection is surprisingly expensive - it involves a handshake and setup that can take milliseconds every time. A connection pool keeps a set of open connections ready to reuse, so requests grab an existing connection instead of paying the setup cost. This is one of the easiest, highest-return optimizations in any system that talks to a database.

12. Avoid the N+1 query problem. A classic, sneaky latency killer: code that fetches a list of items with one query, then makes a separate query for each item's details - turning one operation into hundreds of database round-trips. The fix is to fetch the related data in a single query (a join or a batched lookup) instead of looping. This single anti-pattern is responsible for an enormous share of "why is this page slow" mysteries.

13. Batching. Group many small operations into one larger operation. Instead of writing a thousand records one at a time (a thousand round-trips), write them in a single batch. Instead of sending a notification per event, batch events and send periodically. Batching dramatically reduces per-operation overhead. The trade-off is a slight delay while the batch fills - a latency-versus-efficiency balance you tune to your needs.

Architecture-level strategies

14. Asynchronous processing. Not everything needs to happen before you respond to the user. When a user uploads a video, you don't make them wait for it to be transcoded - you accept the upload, respond immediately, and do the heavy work in the background via a queue. Moving slow, non-essential work out of the request path is one of the most powerful latency strategies, because the user's perceived latency drops to just the fast part. The cost is added complexity: queues, background workers, and handling work that fails after you've already responded.

15. Load balancing. Spreading requests across many servers prevents any single server from becoming overloaded - and an overloaded server is a slow server, because requests pile up in queues behind the work in progress. A load balancer distributes traffic so each server stays within its comfortable capacity, keeping per-request latency low and predictable. It's foundational infrastructure for any system serving meaningful traffic.

Interview: When an interviewer says "this is too slow, how would you speed it up?", don't list every technique you know. Diagnose first: where is the latency? If it's read-heavy, talk caching and replicas. If users are far from servers, talk CDN and geographic placement. If a slow operation sits in the request path, talk async processing. Matching the strategy to the actual bottleneck - rather than reciting all fifteen - is what signals real engineering judgment.

A worked example: diagnosing a slow endpoint

Strategies are only useful if you apply them to the right bottleneck, so let's walk through how diagnosis actually works. Imagine an API endpoint that loads a user's dashboard, and it's taking 1,800 milliseconds - painfully slow. Where's the time going?

You add timing measurements and discover the breakdown: 50ms in the application code, 1,400ms in database queries, 200ms rendering the response, and 150ms in network transfer. Immediately the picture is clear - the database is responsible for nearly 80% of the latency. Optimizing the application code or compressing the response would barely move the needle; the entire problem lives in the data layer.

Digging into the queries, you find two issues. First, the dashboard makes one query to fetch the user's projects, then a separate query for each project's details - a textbook N+1 problem turning into forty database round-trips. Second, one of those queries filters on a column that has no index, forcing a full table scan. The fixes map directly onto strategies from this guide: collapse the N+1 into a single query with a join (strategy 12), add an index on the filtered column (strategy 3), and cache the assembled dashboard for users who reload frequently (strategy 1). After these changes, the same endpoint responds in under 200 milliseconds - a 9x improvement, achieved by targeting only the part that was actually slow.

This is the entire discipline in miniature: measure, find the dominant cost, apply the specific strategy that addresses it, and ignore the parts that were never the problem.

Note: From the field - Emeka (Netflix), Head of Data Infrastructure: "The most expensive latency mistakes I've seen weren't exotic. They were a missing index nobody noticed until traffic grew, or a service quietly making the same call in a loop. When something's slow, resist the urge to add infrastructure. Profile it first - nine times out of ten the fix is removing wasted work, not adding more machines."

Real systems combine many of these strategies in layers: a CDN in front, a cache ahead of the database, indexes and read replicas underneath, async processing for the heavy lifting, and load balancing across it all. Each layer shaves off a category of latency. But the order of operations is always the same - understand where the time goes before you spend effort speeding anything up.

A note on measuring before optimizing

The most important strategy isn't on the list because it isn't a technique - it's a discipline: measure before you optimize. Latency hides in non-obvious places, and engineers routinely optimize the wrong thing. Before applying any of the above, find out where the time actually goes - which query, which service call, which round-trip is the real cost. The biggest latency win is almost always eliminating the single slowest step, and you can't find it without measuring. Optimizing blind wastes effort on parts that were never the bottleneck.

A final caution worth internalizing: latency optimizations can fight each other, so they're not a checklist to apply all at once. Aggressive caching improves read latency but risks serving stale data. Adding read replicas spreads load but widens the consistency gap. Batching raises efficiency but adds a small delay. Every speed-up here is a trade against something - staleness, complexity, cost, or memory - which is precisely why measuring first matters: it tells you which trade is actually worth making for the bottleneck you have, rather than paying all of them blindly.

Latency reduction is rarely about one clever trick. It's about understanding where time is spent in your system and applying the right tool to the right bottleneck, accepting the trade-off each one carries. We work through these strategies - and how to choose between them under interview pressure - across System Design Fundamentals.

Go deeper

• Latency (engineering) - the concept and its sources.
• Latency numbers every programmer should know - the classic reference table for the cost of memory, disk, and network operations.
• Amazon Builders' Library - real-world performance and latency engineering from production systems.
• web.dev performance - practical guidance on measuring and improving latency, especially on the front end.

More articles