Free lesson · System Design Fundamentals

What is Caching?

Medium10 min readsystem-designhlddistributed-systems

10.1 What is Caching?

Interactive animation: The two journeys every read takes: a hit answered from the cache, or a miss fetched from the database and stored.

The Principle That Makes Everything Fast

Open any website. Click a link. The page loads in under a second. Behind that one second, dozens of things happened - DNS resolution, TCP connection, TLS handshake, HTTP request, server processing, database query, response serialization, network transfer, browser rendering. Many of those steps involve fetching data from somewhere. And in a well-designed system, most of that data was not fetched from the original source at all. It was fetched from a cache - a copy of the data stored closer to where it was needed, ready to be served instantly without repeating the expensive work of computing or retrieving it from scratch.

Caching is the most universal performance optimization in all of computing. It exists at every layer of every system. Your CPU has L1, L2, and L3 caches that store recently accessed memory so the processor does not have to wait for RAM. Your operating system caches file system data in memory so disk reads are avoided. Your browser caches images, stylesheets, and JavaScript files so it does not download them on every page load. DNS resolvers cache domain-to-IP mappings so they do not query the root name servers on every request. CDNs cache static content at edge locations around the world so users get responses from a server 50 miles away instead of 5,000 miles away.

And in application architecture - the layer this course focuses on - caching means storing the results of expensive database queries, API calls, or computations in a fast, in-memory store (typically Redis or Memcached) so that subsequent requests for the same data can be served in microseconds instead of milliseconds. The difference between "query the database" (5-50ms) and "read from the cache" (0.1-1ms) does not sound like much for a single request. Multiply it by 10,000 requests per second, and caching is the difference between a system that responds instantly and one that collapses under load.

The Core Idea: Trade Space for Time

Interactive animation: The eighty-twenty rule: when 20% of the data serves 80% of requests, caching it removes 80% of database load.

Every cache makes the same fundamental trade-off: it uses extra storage space (memory, disk, bandwidth) to save time (latency, computation, I/O). You are storing a copy of data that already exists somewhere else, paying for the storage of that copy, in exchange for not having to retrieve or recompute the original every time someone needs it.

This trade-off is worth it because of a property called locality of reference - the observation that access patterns are not uniform. In almost every system, a small fraction of the data receives the vast majority of the requests. On Twitter, a tiny percentage of tweets (from celebrities, viral content) account for the majority of reads. On an e-commerce site, a small fraction of products (bestsellers, deals of the day) receive the majority of views. In a database, a small fraction of rows are queried far more frequently than the rest.

If 20% of your data serves 80% of your requests (often called the "80/20 rule" or Pareto principle), then caching that 20% eliminates 80% of your database load. You do not need to cache everything - you need to cache the hot data.

graph LR
    Client["Client<br/>Request"] --> Cache{"Cache<br/>Has data?"}
    Cache -->|"Yes (cache hit)<br/>~0.5ms"| Response["Return<br/>cached data"]
    Cache -->|"No (cache miss)<br/>fetch from source"| DB["Database<br/>~20ms"]
    DB --> CacheStore["Store in<br/>cache"]
    CacheStore --> Response2["Return<br/>data"]

    style Cache fill:#1e293b,stroke:#f59e0b,color:#e2e8f0
    style DB fill:#1e293b,stroke:#6366f1,color:#e2e8f0

This diagram shows the basic caching flow. A client sends a request. The cache checks whether it has the data. If yes, this is a cache hit, and the data is returned in approximately half a millisecond. If no, this is a cache miss - the data is fetched from the database in approximately 20 milliseconds, stored in the cache for future requests, and then returned to the client.

Hit Rate, Miss Rate, and Why They Matter

The effectiveness of a cache is measured by two numbers:

Cache hit rate - the percentage of requests served from the cache. If 90 out of 100 requests find the data in the cache, the hit rate is 90%.

Cache miss rate - the percentage of requests that must fall through to the original data source. If 10 out of 100 requests do not find the data in the cache, the miss rate is 10%.

Hit rate = hits / (hits + misses). Miss rate = 1 - hit rate.

These numbers matter enormously for system performance. Consider a system handling 10,000 requests per second where each database query takes 20ms and each cache read takes 0.5ms:

With no cache (0% hit rate): Every request hits the database. You need enough database capacity to handle 10,000 queries per second, each taking 20ms. That requires significant database resources, and under load, latency climbs as the database queue grows.

With 90% hit rate: 9,000 requests are served from cache (0.5ms each). Only 1,000 requests hit the database (20ms each). Average latency drops from 20ms to ~2.5ms. Database load drops by 90%. You can serve the same traffic with a much smaller (and cheaper) database.

With 99% hit rate: 9,900 from cache, 100 from database. Average latency drops to ~0.7ms. Database load drops by 99%. The database barely works.

The difference between 90% and 99% hit rate is not 9 percentage points - it is a 10x reduction in database load. In system design, hit rate is one of the most important metrics to design for and monitor.

graph TD
    subgraph HitRate90["90% Hit Rate"]
        H90["9,000 from cache<br/>1,000 from DB<br/>Avg latency: ~2.5ms"]
    end

    subgraph HitRate99["99% Hit Rate"]
        H99["9,900 from cache<br/>100 from DB<br/>Avg latency: ~0.7ms"]
    end

    subgraph NoCache["No Cache"]
        NC["10,000 from DB<br/>Avg latency: ~20ms"]
    end

    style NoCache fill:#1e293b,stroke:#ef4444,color:#e2e8f0
    style HitRate90 fill:#1e293b,stroke:#f59e0b,color:#e2e8f0
    style HitRate99 fill:#1e293b,stroke:#10b981,color:#e2e8f0

This diagram compares three scenarios for a system handling 10,000 requests per second. With no cache, all 10,000 requests hit the database with an average latency of 20 milliseconds. With a 90% hit rate, 9,000 are served from cache and 1,000 from the database, reducing average latency to about 2.5 milliseconds. With a 99% hit rate, 9,900 are from cache and only 100 from the database, reducing average latency to about 0.7 milliseconds.

What Gets Cached (and What Should Not)

Not all data is a good candidate for caching. The best cache candidates share specific characteristics:

Frequently read. Data that is read many times for every write is ideal for caching. A user's profile, a product listing, a configuration value - these are read thousands of times between updates. Caching them absorbs the read load.

Expensive to compute or retrieve. If generating the data requires a complex database query (multiple joins, aggregations), an expensive computation (machine learning inference, image processing), or a slow external API call (third-party service with 500ms latency), caching the result saves that cost on every subsequent request.

Tolerant of staleness. Cached data is a copy, and copies can become stale - the original data changes but the cache still holds the old version. Data that can tolerate being a few seconds (or minutes, or hours) out of date is a good cache candidate. A product's average rating can be a minute old. A user's follower count can be a few seconds old. A news article's view count can be 30 seconds old. Users do not notice or care about these small delays.

Data that should NOT be cached (or cached very carefully):

Rapidly changing data. If data changes on every request (a real-time stock price, a live auction bid), caching it creates a stale-data problem. By the time the cache is populated, the data has already changed. Caching can still work here with very short TTLs (seconds), but the hit rate will be low.

Data where staleness is dangerous. A bank account balance, an inventory count that determines whether you can sell an item, a security permission check - these are cases where serving stale data can cause real harm (double-spending, overselling, unauthorized access). These can be cached, but the invalidation strategy must be precise and the TTL must be very short or zero.

Data that is unique per request. If every request asks for different data (a search query with unique terms, a one-time verification code), caching does not help because the data is never requested again.

Where Caches Live in an Architecture

Caches exist at multiple layers. Understanding where each lives helps you design systems that cache effectively at every level:

Client-side cache (browser cache). The browser stores static assets (images, CSS, JavaScript) locally. On subsequent page loads, these are served from disk without any network request. Controlled by HTTP headers (Cache-Control, ETag, Last-Modified). This is the fastest cache - zero network latency - but only helps the individual user.

CDN cache (edge cache). Content Delivery Networks like CloudFront, Fastly, and Cloudflare cache content at edge locations around the world. When a user in Tokyo requests an image, it is served from a CDN node in Tokyo rather than from the origin server in Virginia. CDN caching is effective for static content and can also cache dynamic API responses with short TTLs.

Application-level cache (Redis, Memcached). This is the cache most people mean when they say "caching" in system design. A fast, in-memory key-value store that sits between the application and the database. The application checks Redis before querying the database. Redis stores data in RAM, which is 100-1000x faster than disk-based database queries.

Database query cache. Some databases (MySQL, PostgreSQL with extensions) cache the results of recent queries internally. If the same query is executed again and the underlying data has not changed, the cached result is returned. This is transparent to the application but limited in flexibility - the cache is invalidated whenever the underlying table is modified, even if the modification does not affect the cached query's result.

Operating system page cache. The OS caches recently read file system pages in RAM. Database reads that seem to hit "disk" often actually hit the OS page cache, which is in memory. This is invisible to the application but can significantly affect performance - a database that fits entirely in the OS page cache performs much differently from one that does not.

graph TD
    User["User"] --> Browser["Browser Cache<br/>(local disk, 0ms)"]
    Browser --> CDN["CDN Edge Cache<br/>(nearest PoP, ~5ms)"]
    CDN --> AppCache["Application Cache<br/>(Redis/Memcached, ~1ms)"]
    AppCache --> DBCache["Database Query Cache<br/>(internal, ~2ms)"]
    DBCache --> Disk["Database on Disk<br/>(~20ms)"]

    style Browser fill:#1e293b,stroke:#10b981,color:#e2e8f0
    style CDN fill:#1e293b,stroke:#3b82f6,color:#e2e8f0
    style AppCache fill:#1e293b,stroke:#f59e0b,color:#e2e8f0
    style DBCache fill:#1e293b,stroke:#6366f1,color:#e2e8f0
    style Disk fill:#1e293b,stroke:#ef4444,color:#e2e8f0

This diagram shows the caching layers from fastest to slowest. The browser cache is on the user's local disk with zero network latency. The CDN edge cache is at the nearest point of presence with about 5 milliseconds latency. The application cache using Redis or Memcached has about 1 millisecond latency. The database query cache is internal to the database at about 2 milliseconds. Finally, reading from the actual database disk takes about 20 milliseconds. Each layer closer to the user is faster but has less data.

The Two Hard Problems

Caching introduces two fundamental challenges that every system designer must confront:

Cache invalidation. When the source data changes, the cached copy becomes stale. How do you know when to update or remove the cached copy? If you invalidate too aggressively (removing cached data on every write), you lose the benefit of caching. If you invalidate too slowly (keeping stale data too long), users see outdated information. Phil Karlton famously said: "There are only two hard things in Computer Science: cache invalidation and naming things." He was not entirely joking.

The strategies for invalidation are covered in detail across the next four lessons (cache-aside, read-through, write-through, write-behind). Each pattern handles invalidation differently, and choosing the right pattern is one of the most important caching decisions.

Cache stampede (thundering herd). When a popular cache entry expires or is invalidated, all the requests that were being served by the cache suddenly hit the database simultaneously. If 1,000 requests per second were being served by a cached product page, and the cache entry expires, all 1,000 requests flood the database at once. The database, which was happily handling 10 requests per second for cache misses, is suddenly hit with 1,000. It slows down, timeouts increase, and the system can cascade into failure.

Mitigation strategies include: locking (the first request that finds a cache miss acquires a lock, fetches from the database, and populates the cache; subsequent requests wait for the lock instead of all hitting the database); staggered TTLs (adding random jitter to TTL values so not all cache entries expire at the same moment); background refresh (refreshing popular cache entries before they expire, so the cache is never empty for hot keys).

Interactive animation: A popular entry expires and every request misses at once - then a lock lets only the first one refill it.

Caching in System Design Interviews

Interview: Caching comes up in virtually every system design interview. The interviewer expects you to identify which data to cache, which caching pattern to use, where to place the cache in the architecture, and how to handle invalidation. A strong answer addresses all four. A weak answer says "we'll add a cache" without specifying the pattern or the invalidation strategy.

When discussing caching, always mention: what you are caching (specific data, not "everything"), what hit rate you expect and why, what TTL you would set and the staleness implications, and what happens on a cache miss. These four points show the interviewer you understand caching as a design decision, not just a buzzword.

Six Takeaways

Tip:

Caching trades space for time. Store a copy of data closer to where it is needed to avoid expensive retrieval or computation on every request.

Hit rate is the most important caching metric. The difference between 90% and 99% is a 10x reduction in backend load. Design for and monitor hit rate.

Cache the hot data, not everything. Locality of reference means a small fraction of data serves the majority of requests. Cache that fraction.

Caches exist at every layer - browser, CDN, application (Redis), database, OS page cache. Each layer absorbs load before it reaches the next.

Cache invalidation is genuinely hard. Stale data is the price of caching. The patterns in the next four lessons each handle invalidation differently.

Cache stampedes can take down a system. When a hot cache entry expires, hundreds of requests flood the backend simultaneously. Locking, jitter, and background refresh mitigate this.

Exercises

Exercise: Warm-up: Your e-commerce application has a product catalog with 1 million products. Analysis shows that 5,000 products receive 85% of all page views. You have a Redis instance with 2 GB of memory. Each cached product is about 2 KB. Can you cache all the hot products? What hit rate would you expect?

Design challenge: You are designing a social media news feed. Each user's feed is computed by aggregating posts from people they follow, sorted by relevance. This computation takes 200ms and involves 5-10 database queries. How would you use caching to make the feed load in under 50ms? What do you cache - the computed feed, the individual posts, or both? What happens when someone the user follows publishes a new post?

Deep dive: Your system uses Redis as a cache with a 5-minute TTL. A cache stampede occurs on a popular product page during Black Friday - 10,000 requests per second suddenly hit the database when the cache expires. Design three different mitigation strategies and explain the trade-offs of each.

Resources

Redis Documentation - https://redis.io/docs/
Memcached Wiki - https://github.com/memcached/memcached/wiki
AWS ElastiCache Documentation - https://docs.aws.amazon.com/elasticache/
Facebook Engineering, "Scaling Memcache at Facebook" - https://research.facebook.com/publications/scaling-memcache-at-facebook/

Back to System Design Fundamentals