Glimpse.
Hello Interview
30 min video
3 min read
Caching for System Design Interviews
You just saved 27 min.
The big takeaway
Master caching fundamentals for system design interviews: understand where to cache (external, in-process, CDN, client-side), learn the four main architectures (cache-aside, write-through, write-behind, read-through), know eviction policies (LRU, LFU, FIFO, TTL), and be prepared to discuss common issues like cache stampedes, consistency problems, and hot keys. Always justify caching by identifying a real bottleneck before introducing it.
Why Caching Matters
Speed Difference Between Storage Layers
Accessing data from disk (SSD) takes about 1 millisecond, while accessing from RAM takes about 100 nanoseconds—roughly 10,000 times faster. This gap compounds quickly when serving thousands of requests per second, making caching a critical optimization.
Disk (SSD)
1000000 nanoseconds
RAM
100 nanoseconds
Access latency comparison: RAM is ~10,000x faster than disk
Cache Trades Storage and Complexity for Speed
Caching keeps copies of frequently used data in a faster layer (usually memory) so systems don't have to reach back to slower sources every time. This trades off additional storage space and system complexity for significant performance gains.
Where to Cache: Four Layers
External Caching (Redis, Memcached)
A dedicated caching service runs on its own server with its own memory, separate from the application and database. Multiple application servers share this global cache, so once one server caches data, others can reuse it instantly. Cache hits return data immediately; cache misses trigger a database fetch, which is then cached for future requests.
In-Process Caching
Data is cached directly in the application server's memory, avoiding expensive network hops and providing the fastest caching possible. However, each server has its own isolated cache, so data cached on one server won't be visible to others, leading to inconsistencies and potential memory waste. Best used for low-latency optimizations like config data or lookup tables that every request depends on.
CDN (Content Delivery Networks)
Geographically distributed servers cache content closer to users, optimizing for network latency rather than memory-vs-disk speed. A request to an edge server a few miles away (20-40ms round trip) is far faster than traveling to an origin server thousands of miles away (300-350ms). Modern CDNs can cache static media, API responses, HTML pages, and run edge logic, though media delivery is the most common use case in interviews.
Without CDN (Virginia to Australia)
325 ms
With CDN (edge server nearby)
30 ms
CDN reduces round-trip latency by ~90% for distant users
Client-Side Caching
Data is stored directly on the user's device (browser local storage, app memory, or local disk), avoiding network costs entirely and providing instant access. The downside is less control over data freshness and validation. Relevant mainly for offline functionality or client-heavy workloads like browser image reuse or mobile app sync-on-reconnect patterns.
Cache Architectures
Cache-Aside (Most Common)
The application checks the cache first; if data is found (cache hit), it returns immediately. On cache miss, the app fetches from the database, stores it in the cache, and returns it to the user. This keeps the cache lean by only caching requested data, but cache misses incur database latency. This is the default pattern you should use in interviews.
1
Application checks cache
2
Cache hit: return data immediately
3
Cache miss: fetch from database
4
Store result in cache
5
Return data to user
Cache-aside flow: check cache first, fall back to database on miss
Write-Through Caching
The application writes directly to the cache first, then the cache synchronously writes to the database before returning to the user. The write is only considered complete once both are updated. Requires a caching library or framework (like Spring Cache or Hazelcast) to handle this automatically, since Redis and Memcached don't natively support it. Trade-offs include slower writes, cache pollution with data never read again, and the dual-write problem where cache and database can enter inconsistent states if one succeeds and the other fails.
Write-Behind (Write-Back) Caching
Similar to write-through, but the cache writes to the database asynchronously in batches rather than synchronously. The application only writes to the cache, making writes much faster. However, if the cache crashes before flushing, data loss occurs. Use only when high write throughput matters more than immediate consistency, such as analytics or metric pipelines where occasional data loss is acceptable.
Read-Through Caching
The cache itself handles database lookups instead of the application. On cache miss, the cache fetches from the database, stores the result, and returns it to the app—essentially acting as a proxy. This is how CDNs work. For application-level caching in interviews, cache-aside is preferred because it doesn't require special frameworks.
Naming Doesn't Matter in Interviews
Interviewers care about understanding behavior, not memorizing exact terms. If you forget the name, simply describe the flow: 'I'll check the cache first, and if it's not there, I'll go to the database and update the cache.' Clear explanation of how caching works beats terminology.
Cache Eviction Policies
Least Recently Used (LRU)
Evicts items that haven't been used recently. Implemented with a linked list or priority queue tracking access order, though implementation details are out of scope for interviews. LRU is the most common default eviction policy in system design interviews.
Least Frequently Used (LFU)
Evicts items based on access frequency rather than recency. Items accessed least often are removed first, even if they were used recently. Makes sense when access patterns are highly skewed, with a few items read far more often than others.
First In, First Out (FIFO)
The oldest item is removed to make space for the newest. Simple but rarely the right choice in system design interviews.
Time To Live (TTL)
Each cached item has an expiration time (e.g., 5 minutes). Once the time passes, the cache automatically removes it. Ideal for data that can go stale, like user sessions or API responses. TTL is super common when freshness matters more than recency or frequency.
Common Caching Issues
Cache Stampede (Thundering Herd)
When a popular cache entry expires via TTL, a flood of requests simultaneously try to rebuild it from the database. Even a 1-second window can turn one query into thousands or millions, overwhelming the database. Example: a homepage feed cached for 60 seconds with 100,000 requests per second; when it expires, all 100,000 requests hit the database at once.
100,000
simultaneous database requests on cache expiry
Cache stampede: all requests hit database at once when popular key expires
Cache Stampede: Request Coalescing Solution
When multiple requests try to rebuild the same cache key, only the first one proceeds; the rest wait for results and read from the cache. Also called 'single flight,' this prevents duplicate database queries during a cache miss.
Cache Stampede: Cache Warming Solution
Proactively refresh popular keys before they expire (e.g., refresh at 55 seconds for a 60-second TTL) to prevent expiry and the resulting thundering herd. This keeps hot data fresh without ever triggering a stampede.
Cache Consistency
The cache and database can return different values for the same data because most systems read from cache but write to the database, creating a window where stale data persists. Example: a user updates their profile picture in the database, but the old image remains in the cache until evicted, so other users see the outdated version.
Cache Consistency: Invalidate on Write
When data is written to the database, proactively delete that key from the cache. The next read will miss the cache, fetch fresh data from the database, and update the cache. This ensures the latest data is served for the most part.
Cache Consistency: Short TTLs
If some staleness is acceptable, use short TTLs (e.g., 60 seconds for a newsfeed). Data will be automatically evicted and refreshed frequently, limiting the window of inconsistency.
Cache Consistency: Accept Eventual Consistency
For feeds, analytics, metrics, and other non-critical data, accepting eventual consistency is valid. Users might see stale data for a period (e.g., 5 minutes), but it's acceptable because the delay doesn't break functionality.
Hot Keys
A single cache entry receives far more traffic than others, becoming a bottleneck even if overall cache hit rate is excellent. Example: Taylor Swift's profile on Twitter receives millions of requests per second, overloading a single Redis node or shard. Caching increases overall read throughput but doesn't solve the problem if one piece of data is overwhelmingly popular.
Hot Keys: Replication Solution
Replicate hot keys across multiple cache shards or instances. Instead of all traffic hitting one cache node, load balance evenly across all nodes, each holding a copy of the hot key (e.g., Taylor Swift's profile on every shard).
Hot Keys: Local Fallback Cache Solution
Use in-process caching to keep extremely hot values in the application server's memory. Repeated requests for hot keys don't even need to hit Redis; they're served from local memory, further reducing latency and load.
Discussing Caching in Interviews
Don't Add Cache Without Justification
Never introduce caching just for the sake of it. Lack of justification is a red flag, even if caching is ultimately correct. Always identify a real bottleneck first.
When to Bring Up Caching
Introduce caching when one of four conditions is true: (1) read-heavy workload straining the database, (2) expensive queries requiring joins across multiple tables, (3) high database CPU usage, or (4) latency requirements that database queries alone cannot meet.
1
Read-heavy workload
2
Expensive queries
3
High database CPU
4
Latency requirements
Four scenarios where caching is justified
Five-Step Framework for Introducing Caching
Follow this order to impress interviewers: (1) Identify the bottleneck with rough numbers, (2) Decide what to cache (frequent, expensive, infrequently-changing data), (3) Define cache keys explicitly, (4) Choose a cache architecture (e.g., cache-aside), (5) Specify eviction policy and address potential downsides (stampedes, consistency, hot keys).
1
Identify bottleneck with metrics
2
Decide what to cache
3
Define cache keys explicitly
4
Choose cache architecture
5
Specify eviction policy and address downsides
Five-step framework for introducing caching in interviews
When Caching Comes Up in Interviews
Caching typically emerges during deep dives when discussing scale or latency. When you reach non-functional requirements and talk about scale or latency constraints, that's the appropriate time to introduce caching and explain where it fits into your system.
Worth quoting
"There's only two hard problems in computer science: naming things and cache invalidation."
— Evan (citing famous computer science saying), at [17:42]
"Interviewers don't care if you remember these exact names. What matters is you can describe the behavior clearly."
— Evan, at [14:46]
"Caching helps us scale reads, but it doesn't make your system magically infinite."
— Evan, at [25:28]
Try this
Memorize the cache-aside architecture and be able to describe it without looking up the name.
Practice identifying when caching is justified by quantifying bottlenecks (e.g., 2 billion reads per day, expensive joins).
For each caching scenario you discuss, explicitly state: what you're caching, the cache key structure, the eviction policy, and at least one potential issue (stampede, consistency, or hot keys).
When discussing consistency, be ready to justify your choice: invalidate-on-write for critical data, short TTLs for acceptable staleness, or eventual consistency for non-critical data.
Practice the five-step framework (identify bottleneck, decide what to cache, define keys, choose architecture, specify policy and downsides) until it becomes second nature.
Prepare examples of hot keys and their solutions (replication and local fallback caching) for your specific domain (e.g., celebrity profiles, trending topics).
Made with Glimpse by Wozart
glimpse.wozart.com/v/31zbl0e4
Share this infographic
Read this infographic as text

Caching for System Design Interviews

Summary of the video “Caching in System Design Interviews w/ Meta Staff Engineer” by Hello Interview.

Master caching fundamentals for system design interviews: understand where to cache (external, in-process, CDN, client-side), learn the four main architectures (cache-aside, write-through, write-behind, read-through), know eviction policies (LRU, LFU, FIFO, TTL), and be prepared to discuss common issues like cache stampedes, consistency problems, and hot keys. Always justify caching by identifying a real bottleneck before introducing it.

Why Caching Matters

Speed Difference Between Storage Layers

Accessing data from disk (SSD) takes about 1 millisecond, while accessing from RAM takes about 100 nanoseconds—roughly 10,000 times faster. This gap compounds quickly when serving thousands of requests per second, making caching a critical optimization.

Cache Trades Storage and Complexity for Speed

Caching keeps copies of frequently used data in a faster layer (usually memory) so systems don't have to reach back to slower sources every time. This trades off additional storage space and system complexity for significant performance gains.

Where to Cache: Four Layers

External Caching (Redis, Memcached)

A dedicated caching service runs on its own server with its own memory, separate from the application and database. Multiple application servers share this global cache, so once one server caches data, others can reuse it instantly. Cache hits return data immediately; cache misses trigger a database fetch, which is then cached for future requests.

In-Process Caching

Data is cached directly in the application server's memory, avoiding expensive network hops and providing the fastest caching possible. However, each server has its own isolated cache, so data cached on one server won't be visible to others, leading to inconsistencies and potential memory waste. Best used for low-latency optimizations like config data or lookup tables that every request depends on.

CDN (Content Delivery Networks)

Geographically distributed servers cache content closer to users, optimizing for network latency rather than memory-vs-disk speed. A request to an edge server a few miles away (20-40ms round trip) is far faster than traveling to an origin server thousands of miles away (300-350ms). Modern CDNs can cache static media, API responses, HTML pages, and run edge logic, though media delivery is the most common use case in interviews.

Client-Side Caching

Data is stored directly on the user's device (browser local storage, app memory, or local disk), avoiding network costs entirely and providing instant access. The downside is less control over data freshness and validation. Relevant mainly for offline functionality or client-heavy workloads like browser image reuse or mobile app sync-on-reconnect patterns.

Cache Architectures

Cache-Aside (Most Common)

The application checks the cache first; if data is found (cache hit), it returns immediately. On cache miss, the app fetches from the database, stores it in the cache, and returns it to the user. This keeps the cache lean by only caching requested data, but cache misses incur database latency. This is the default pattern you should use in interviews.

Write-Through Caching

The application writes directly to the cache first, then the cache synchronously writes to the database before returning to the user. The write is only considered complete once both are updated. Requires a caching library or framework (like Spring Cache or Hazelcast) to handle this automatically, since Redis and Memcached don't natively support it. Trade-offs include slower writes, cache pollution with data never read again, and the dual-write problem where cache and database can enter inconsistent states if one succeeds and the other fails.

Write-Behind (Write-Back) Caching

Similar to write-through, but the cache writes to the database asynchronously in batches rather than synchronously. The application only writes to the cache, making writes much faster. However, if the cache crashes before flushing, data loss occurs. Use only when high write throughput matters more than immediate consistency, such as analytics or metric pipelines where occasional data loss is acceptable.

Read-Through Caching

The cache itself handles database lookups instead of the application. On cache miss, the cache fetches from the database, stores the result, and returns it to the app—essentially acting as a proxy. This is how CDNs work. For application-level caching in interviews, cache-aside is preferred because it doesn't require special frameworks.

Naming Doesn't Matter in Interviews

Interviewers care about understanding behavior, not memorizing exact terms. If you forget the name, simply describe the flow: 'I'll check the cache first, and if it's not there, I'll go to the database and update the cache.' Clear explanation of how caching works beats terminology.

Cache Eviction Policies

Least Recently Used (LRU)

Evicts items that haven't been used recently. Implemented with a linked list or priority queue tracking access order, though implementation details are out of scope for interviews. LRU is the most common default eviction policy in system design interviews.

Least Frequently Used (LFU)

Evicts items based on access frequency rather than recency. Items accessed least often are removed first, even if they were used recently. Makes sense when access patterns are highly skewed, with a few items read far more often than others.

First In, First Out (FIFO)

The oldest item is removed to make space for the newest. Simple but rarely the right choice in system design interviews.

Time To Live (TTL)

Each cached item has an expiration time (e.g., 5 minutes). Once the time passes, the cache automatically removes it. Ideal for data that can go stale, like user sessions or API responses. TTL is super common when freshness matters more than recency or frequency.

Common Caching Issues

Cache Stampede (Thundering Herd)

When a popular cache entry expires via TTL, a flood of requests simultaneously try to rebuild it from the database. Even a 1-second window can turn one query into thousands or millions, overwhelming the database. Example: a homepage feed cached for 60 seconds with 100,000 requests per second; when it expires, all 100,000 requests hit the database at once.

Cache Stampede: Request Coalescing Solution

When multiple requests try to rebuild the same cache key, only the first one proceeds; the rest wait for results and read from the cache. Also called 'single flight,' this prevents duplicate database queries during a cache miss.

Cache Stampede: Cache Warming Solution

Proactively refresh popular keys before they expire (e.g., refresh at 55 seconds for a 60-second TTL) to prevent expiry and the resulting thundering herd. This keeps hot data fresh without ever triggering a stampede.

Cache Consistency

The cache and database can return different values for the same data because most systems read from cache but write to the database, creating a window where stale data persists. Example: a user updates their profile picture in the database, but the old image remains in the cache until evicted, so other users see the outdated version.

Cache Consistency: Invalidate on Write

When data is written to the database, proactively delete that key from the cache. The next read will miss the cache, fetch fresh data from the database, and update the cache. This ensures the latest data is served for the most part.

Cache Consistency: Short TTLs

If some staleness is acceptable, use short TTLs (e.g., 60 seconds for a newsfeed). Data will be automatically evicted and refreshed frequently, limiting the window of inconsistency.

Cache Consistency: Accept Eventual Consistency

For feeds, analytics, metrics, and other non-critical data, accepting eventual consistency is valid. Users might see stale data for a period (e.g., 5 minutes), but it's acceptable because the delay doesn't break functionality.

Hot Keys

A single cache entry receives far more traffic than others, becoming a bottleneck even if overall cache hit rate is excellent. Example: Taylor Swift's profile on Twitter receives millions of requests per second, overloading a single Redis node or shard. Caching increases overall read throughput but doesn't solve the problem if one piece of data is overwhelmingly popular.

Hot Keys: Replication Solution

Replicate hot keys across multiple cache shards or instances. Instead of all traffic hitting one cache node, load balance evenly across all nodes, each holding a copy of the hot key (e.g., Taylor Swift's profile on every shard).

Hot Keys: Local Fallback Cache Solution

Use in-process caching to keep extremely hot values in the application server's memory. Repeated requests for hot keys don't even need to hit Redis; they're served from local memory, further reducing latency and load.

Discussing Caching in Interviews

Don't Add Cache Without Justification

Never introduce caching just for the sake of it. Lack of justification is a red flag, even if caching is ultimately correct. Always identify a real bottleneck first.

When to Bring Up Caching

Introduce caching when one of four conditions is true: (1) read-heavy workload straining the database, (2) expensive queries requiring joins across multiple tables, (3) high database CPU usage, or (4) latency requirements that database queries alone cannot meet.

Five-Step Framework for Introducing Caching

Follow this order to impress interviewers: (1) Identify the bottleneck with rough numbers, (2) Decide what to cache (frequent, expensive, infrequently-changing data), (3) Define cache keys explicitly, (4) Choose a cache architecture (e.g., cache-aside), (5) Specify eviction policy and address potential downsides (stampedes, consistency, hot keys).

When Caching Comes Up in Interviews

Caching typically emerges during deep dives when discussing scale or latency. When you reach non-functional requirements and talk about scale or latency constraints, that's the appropriate time to introduce caching and explain where it fits into your system.

Notable quotes

There's only two hard problems in computer science: naming things and cache invalidation. — Evan (citing famous computer science saying)
Interviewers don't care if you remember these exact names. What matters is you can describe the behavior clearly. — Evan
Caching helps us scale reads, but it doesn't make your system magically infinite. — Evan

Action items

  • Memorize the cache-aside architecture and be able to describe it without looking up the name.
  • Practice identifying when caching is justified by quantifying bottlenecks (e.g., 2 billion reads per day, expensive joins).
  • For each caching scenario you discuss, explicitly state: what you're caching, the cache key structure, the eviction policy, and at least one potential issue (stampede, consistency, or hot keys).
  • When discussing consistency, be ready to justify your choice: invalidate-on-write for critical data, short TTLs for acceptable staleness, or eventual consistency for non-critical data.
  • Practice the five-step framework (identify bottleneck, decide what to cache, define keys, choose architecture, specify policy and downsides) until it becomes second nature.
  • Prepare examples of hot keys and their solutions (replication and local fallback caching) for your specific domain (e.g., celebrity profiles, trending topics).

More like this