Concept 2: Caching Strategies

System Design Bible — File 02 of 10 Difficulty: ★★☆☆☆ Prerequisites: File 01, plus the memory-hierarchy and statelessness ideas unpacked in Section 0.

Table of Contents#

0Prerequisites — What You Must Understand First#

Caching is one idea — "keep a copy of expensive-to-get data somewhere fast" — but it only makes sense once five background facts are concrete. The first three are about why a fast copy is worth having; the last two are about the machinery that makes a cache work at all, and they are the ones that later sections (key design, serialization cost, the read-path race in §3.6) silently assume.

0.1The memory hierarchy (why "fast" and "slow" storage both exist)#

Computers don't have one kind of storage; they have a hierarchy, and every level trades speed against size and cost. At the top: CPU registers and caches (L1/L2/L3) — unbelievably fast (nanoseconds) but tiny (kilobytes to megabytes). Below that: main memory (RAM) — very fast (~100 ns) and moderately sized (gigabytes), but volatile (wiped on power loss) and relatively expensive per gigabyte. Below that: disk (SSD/HDD) — slow (microseconds to milliseconds) but huge (terabytes) and cheap and durable (survives power loss). And across a network: another machine's RAM or disk — add network latency on top. The universal rule: the faster a storage tier, the smaller and more expensive it is. You cannot simply "put everything in RAM" — you don't have enough RAM, and RAM loses data on restart, so the source of truth must live on durable disk (a database). Caching is the art of keeping a small, hot subset of that disk-based truth in a fast tier (RAM) so most requests never pay the slow-tier price. Everything in this file follows from this hierarchy.

0.2Why the database is the bottleneck (and can't just be scaled like app servers)#

In File 01 you scaled the application tier trivially: app servers are stateless (they hold no permanent data), so you can run 100 identical copies behind a load balancer and any one can serve any request. The database cannot be scaled this way, and understanding why is the whole motivation for caching. The database holds state — the actual, authoritative data — and that state must stay consistent. You can't just run 100 independent copies all accepting writes, because they'd immediately disagree about what the data is (which is the hard problem of File 08). So the database is a shared, hard-to-replicate chokepoint: every app server funnels its reads and writes into it, and it can only do so much before its disk, locks, CPU, and connection pool saturate. Since in most systems reads vastly outnumber writes (often 100:1 or 1000:1 — think how many times a popular tweet is read vs written), the highest-leverage move is to stop sending all those repeated reads to the database. That's caching: absorb the flood of reads in a fast tier so the precious database only handles writes and the occasional cache miss.

0.3Latency as a felt, monetized thing#

"Fast" isn't an abstract virtue — latency is felt by users and costs money. Studies at Amazon and Google found that every extra 100 ms of latency measurably reduced sales and engagement; users abandon slow pages. A database query that hits disk and crosses the network might take 20–100 ms, while the same answer from an in-memory cache takes 0.1–1 ms — a 100× improvement that turns a sluggish page into an instant one. So caching isn't just an efficiency tweak for the backend; it's directly a product/revenue lever. Keep this in mind: when you add a cache in an interview, you're buying both database relief and user-facing speed, and both are worth stating.

0.4The key–value model, serialization, and what "storing an object in Redis" actually costs#

Every cache in this file is fundamentally a key–value store: a giant dictionary mapping an opaque key (a string, usually) to an opaque value (a blob of bytes). It is the simplest possible database — no schema, no joins, no query language, no secondary indexes, no WHERE clause. You can only do three things: GET key, SET key value, DELETE key.

Why that restriction is a feature, not a limitation. Because there is exactly one access path — hash the key, jump to the slot — a lookup is O(1) and involves no query planning, no index traversal, no lock on a table, and no disk seek. A relational database's SELECT * FROM users WHERE id = 42 must parse SQL, plan the query, traverse a B-tree index, possibly fetch pages from disk, and honour transaction isolation. The cache's GET user:42 hashes 7 bytes and returns a pointer. That gap — a few hundred nanoseconds versus a few milliseconds — is the cache's value proposition, and it exists precisely because the cache refuses to do anything clever. The corollary you must internalize: a cache cannot answer a question you have not already turned into a key. There is no "give me all users in California" unless you explicitly computed that list and stored it under a key like users:state:CA. Designing what the keys are is designing the cache.

Serialization — the hidden cost. Your application holds a User object: a structure in memory with fields, pointers, and type information. The cache holds bytes. Converting between the two is serialization (object → bytes, for writing) and deserialization (bytes → object, for reading), and it is not free — it is often the single largest CPU cost in a cache hit, dwarfing the network time. The format choices:

Compression sits on top: LZ4 or Snappy for speed (compress at gigabytes per second, typical 2× reduction), zstd or gzip for ratio (3–4× reduction, meaningfully more CPU). The trade is always the same shape — CPU time to buy RAM and network bytes — and it is worth it exactly when your values are large and text-like, and not worth it when values are small (compression headers can make a 60-byte value bigger).

Hold this: a cache stores bytes under keys, so every hit pays serialization, and the format you choose is a real latency and memory decision, not a detail.

0.5Concurrency, atomicity, and why caches have races#

Everything in §3 and §6 is a concurrency story, so this must be concrete. Your service does not handle one request at a time; it handles thousands simultaneously, on many threads across many machines. Those requests interleave in unpredictable order, and any sequence of operations that is not atomic can be interrupted between its steps by another request doing the same thing.

Atomic means indivisible: an operation that either fully happens or does not, with no observable intermediate state, and which cannot interleave with another atomic operation on the same data. SET key value in Redis is atomic. But the pattern "read the DB, then write the cache" is two operations with a gap between them, and in that gap anything can happen — which is exactly the bug in §3.6.

Walk the classic lost update to make the shape unmistakable. Two requests both increment a view counter cached as a plain value:

   Request A                      Request B                    cached value
   GET counter -> 10                                                10
                                  GET counter -> 10                 10
   compute 10+1 = 11                                                10
                                  compute 10+1 = 11                 10
   SET counter = 11                                                 11
                                  SET counter = 11                  11   ← should be 12

Two views recorded, one counted. Nothing crashed, no error was logged, the number is simply wrong — and it will be wrong more often the more traffic you have, which is the worst possible property. The fix is to make the whole read-modify-write one atomic operation: Redis's INCR does the increment inside the server, where commands are executed one at a time, so no interleaving is possible. This is why "use the data structure's atomic operation instead of read-modify-write" is a rule rather than a preference.

The three tools you will see used against races in this file:

Hold this: any cache operation that is not a single atomic command has a gap in it, and at scale every gap is eventually exploited by an unlucky interleaving.

1Architectural Definition & "The Why"#

1.1The Plain Definition#

A cache is a high-speed data storage layer that stores a subset of data — typically the results of expensive computations or frequently-accessed records — so that future requests for that data are served faster than recomputing or re-fetching from the original slower source (the "source of truth" / "origin" / "backing store", usually a database or a downstream service).

Caching is fundamentally a space-for-time trade: you spend memory to buy latency reduction and load reduction.

1.2"The Why" — The Latency Pyramid#

Caching exists because of the brutal, non-negotiable physics of the memory hierarchy. Different storage media differ in access speed by orders of magnitude. Every senior engineer should have these "latency numbers every programmer should know" burned in:

L1 CPU cache reference ............... ~1 ns
L2 CPU cache reference ............... ~4 ns
Main memory (RAM) reference .......... ~100 ns
Read 1 MB sequentially from RAM ...... ~10 microseconds
SSD random read ...................... ~16 microseconds
Round trip within same datacenter .... ~500 microseconds (0.5 ms)
Read 1 MB from SSD ................... ~1 ms
Disk (HDD) seek ...................... ~10 ms
Read 1 MB from disk .................. ~20 ms
Network round trip CA -> Netherlands . ~150 ms

The punchline: RAM is ~1000× faster than disk, and a same-datacenter network hop is ~300× faster than a cross-continent one. A database query that hits disk and crosses the network might take 20–100 ms; the same answer served from an in-memory cache takes 0.1–1 ms. That's a 100× improvement.

1.3The Problems Caching Solves#

Historically, applications hit two walls:

Connecting to File 01: We ended load balancing by saying "make your servers stateless — externalize session state to a shared store." That shared store is very often a distributed cache (Redis/Memcached). Caching is what enables the stateless-server design that makes horizontal scaling clean.

2The Recursive Sub-Concept Tree#

2.1Sub-Concept: Cache Hit, Cache Miss, Hit Ratio#

Why the hit ratio matters more than it looks — the arithmetic that surprises people. Hit ratio does not improve latency linearly; it improves it hyperbolically, because the misses dominate the average. Suppose a hit costs 1 ms and a miss costs 50 ms. Average latency is hit_ratio × 1 + (1 − hit_ratio) × 50:

Read that table twice, because two lessons live in it. First, going from 90% to 99% halves your database load again — the last few percent are where the remaining load lives, so "we're already at 90%, why bother" is wrong. Second, the average is a liar: at 95% hit ratio your median user sees 1 ms and your p99 user sees the full 50 ms miss, so the cache improves the typical experience far more than it improves the worst one. When an interviewer asks about tail latency, this is the point to make.

Two more metrics that matter and are usually forgotten. The miss penalty — how expensive is a miss, really? A cache in front of a 5 ms query is worth much less than one in front of a 2-second report. And the cost of a hit that shouldn't have been — a stale hit is not a success, it is a correctness failure that your hit-ratio dashboard will happily report as a win. Hit ratio measures performance, never correctness.

How to actually read it. In Redis, INFO stats reports keyspace_hits and keyspace_misses; the ratio is computed by you, and note it is server-wide, not per key-prefix — so one badly-behaved key pattern can be hidden by a well-behaved one. Instrument hit ratio per cache-use-case in your application, not only at the server.

2.2Sub-Concept: TTL (Time To Live)#

TTL is an expiration timer attached to a cached entry. After TTL seconds, the entry is considered stale and is either evicted or refreshed on next access. TTL is your primary defense against serving forever-stale data: even if you never explicitly invalidate, the entry self-destructs after the TTL. Short TTL = fresher data but lower hit ratio + more origin load; long TTL = higher hit ratio but staler data. This tension is central to caching.

How expiration is actually implemented — and why it is not a timer. The naive mental model is that each key has a countdown and something fires when it hits zero. Nothing does that, because a server holding fifty million keys cannot afford fifty million timers. Real caches use two mechanisms together:

The consequence you can be asked about: memory usage does not drop the instant a TTL passes, and a key can be logically expired but physically present. If you are debugging "why is memory still full," this is the mechanism to name.

Choosing a TTL — the actual reasoning, not a guess. The right TTL is derived from how stale the business can tolerate this data being, bounded by how much origin load you can afford. Work it in that order:

TTL is not the only expiry knob. Absolute TTL (expire at a fixed moment, EXPIREAT) is right for data that becomes invalid at a known time — a daily report, a token with a known expiry. Sliding TTL (the timer resets on every access, which you implement by re-issuing EXPIRE on read) keeps hot data alive indefinitely and is right for sessions, where "expire 30 minutes after last activity" is exactly the semantic you want. Confusing the two produces sessions that log users out mid-work, or reports that never refresh.

2.3Sub-Concept: Eviction#

Caches are bounded (limited RAM). When the cache is full and you need to insert a new entry, you must evict (remove) an existing one. The eviction policy decides which entry to sacrifice (LRU, LFU, etc. — full treatment in Section 4). Note: eviction (capacity-driven removal) is different from expiration (TTL-driven removal) and different from invalidation (correctness-driven removal because the underlying data changed). Interviewers test whether you conflate these three.

Say the distinction precisely, because it is a genuine tell:

The consequences differ sharply. Eviction reduces your hit ratio but never serves wrong data. Expiration is a blunt correctness instrument — it bounds staleness but does not eliminate it, since a value can be wrong for the entire TTL window. Only invalidation is targeted correctness, and it is the hardest of the three because it demands you know every key affected by a write (§5).

A fourth removal you should know: memory-pressure eviction of the wrong thing. If your cache holds both session data (loss = users logged out) and cached query results (loss = a slow page), an ordinary LRU eviction under memory pressure will happily throw away sessions. The fix is separate caches or separate logical databases per workload — never mix data with different loss consequences in one eviction pool. Facebook's "pools" (§11.1) exist for exactly this reason.

2.4Sub-Concept: Cache Placement — Where Caches Live#

Caches exist at every layer of a system. Know the taxonomy:

A common design uses multiple tiers: local in-process cache (L1) backed by a distributed cache (L2) backed by the database (origin). This is a near cache pattern.

Those six bullets are the taxonomy; each one deserves its actual reasoning, because "where do I put the cache" is a design decision with different trade-offs at every level and interviewers probe exactly here.

Client-side (browser / mobile app). The fastest cache possible — a hit costs zero network, zero server CPU, zero money, and works offline. It is also the only cache you cannot reach: once a response is stored on ten million devices, you have no mechanism to delete it. This asymmetry drives the whole design of HTTP caching (§7): you control client caches in advance, by declaring at write time how long the response may be reused and by giving it a validator the client can use to cheaply re-check. The standard mitigation for un-invalidatable client caches is content-addressed URLs — name the file app.9f3c2a.js, where the hash is of the contents, and cache it for a year; when the contents change, the name changes, so the old cached copy is simply never requested again. This is called cache busting, and it converts an invalidation problem into a naming problem.

CDN / edge cache. A shared cache in hundreds of locations near users, so a hit avoids not just the database but the entire cross-continent round trip (§1.2: ~150 ms saved). Because it is shared across all users, one user's miss warms the cache for everyone nearby — an enormous multiplier on popular content. You can invalidate it (purge APIs, in seconds for Fastly, tens of seconds for most). Costs: it caches only what is safely shareable, so anything personalized needs careful Vary handling or it will leak one user's data to another — a real and catastrophic bug class. Full treatment in File 05.

Reverse-proxy cache (NGINX, Varnish, Apache Traffic Server). The same idea, sitting in your own datacenter in front of your app tier. It absorbs repeated identical requests before they ever reach application code, which means it saves not just database time but your app's template rendering, serialization, and framework overhead — often the larger cost. Best at: anonymous, high-traffic, identical responses (a homepage, a product page for logged-out users). Costs: it caches whole HTTP responses, so it is coarse — you cannot reuse a fragment of it, and any personalization defeats it entirely. Mitigation: ESI/fragment caching, or splitting pages into a cacheable shell plus a personalized fetch.

Application-local (in-process) cache — Caffeine, Guava, Ehcache, or a plain hash map. The value lives in your process's own heap, so a hit is a pointer dereference: tens of nanoseconds, roughly 10,000× faster than a network hop to Redis, with no serialization at all (you cache the object, not bytes — §0.4). That is a staggering advantage, and it comes with three sharp costs. (1) Every server has its own copy, so N servers means N potentially-divergent versions of the truth and no way to invalidate them all short of a broadcast (§2.9). (2) It dies on restart and on every deploy, so it is permanently cold in a frequently-deployed service. (3) It competes with your application for heap, which in a garbage-collected language means a large local cache directly lengthens GC pauses — you can make your p99 worse by caching more. Reach for it when: the data is small, extremely hot, and tolerant of a few seconds of divergence — feature flags, configuration, reference data, the results of a hot lookup.

Distributed / remote cache — Redis, Memcached, and the family in §10. One network hop (~0.2–1 ms in-datacenter), which is 1,000× slower than local memory and 50–100× faster than the database. In exchange you get the properties local caching cannot give: one shared copy (so invalidation is a single delete rather than a broadcast), survival across app restarts and deploys, capacity far beyond one process's heap (a Redis cluster can hold terabytes), and the ability to hold shared state like sessions and locks. This is the default meaning of "add a cache" in an interview, and the reason is that shared-and-invalidatable beats fast-but-divergent for almost all data.

Database-internal cache — the buffer pool. Every database already caches: the buffer pool keeps recently-used disk pages in RAM, so a "disk read" is frequently not a disk read at all. This matters for two reasons. First, it means your measured "database is slow" may be a cold buffer pool problem, fixable by giving the database more RAM rather than by adding a cache tier. Second, it means your application-level cache and the database's buffer pool are caching the same bytes twice — a genuine waste that argues for caching computed results (a rendered page, an aggregated total) rather than raw rows, since only the former saves work the database cannot save for you.

The multi-tier / near-cache pattern, made concrete. L1 local (nanoseconds, per-process) → L2 distributed (sub-millisecond, shared) → origin (milliseconds, authoritative). A read checks L1, then L2, then the DB, populating on the way back up. The benefit is not just speed — it is blast-radius containment: if the Redis cluster dies, L1 keeps serving hot keys and the database sees a survivable fraction of traffic instead of 100% of it (§6.2). The cost is coherence: you now have N + 1 copies, and the L1 copies are the ones you cannot delete directly, which is why §2.9 exists. The standard rule: give L1 a much shorter TTL than L2 (say 5 seconds versus 5 minutes), so local divergence is bounded by seconds without a coherence protocol at all. That single trick avoids an enormous amount of complexity and is the right first answer.

2.5Sub-Concept: Redis vs. Memcached (the two default answers)#

Those two paragraphs are the interview-answer version. The full landscape — including Valkey, Dragonfly, KeyDB, Hazelcast, Caffeine, Varnish, ElastiCache, MemoryDB, DAX, Momento — with the architectural reasons they differ and the decision rules for choosing between them, is §10.

2.6Sub-Concept: The Working Set and Why Caches Work At All#

A cache only helps if requests are not uniformly distributed over the data. If every one of your ten million users were equally likely to be requested on any given second, a cache holding one hundred thousand of them would hit 1% of the time and be worthless. Caching works because real access is savagely skewed.

The working set is the set of data actually being accessed during some window of time — the "hot" subset. Real workloads follow a power law (often summarized as the Pareto or 80/20 rule, though real systems are usually far more extreme): a tiny fraction of items receives the overwhelming majority of requests. On a social network, a few thousand accounts out of hundreds of millions generate a large share of all profile views. On an e-commerce site, the front-page products dwarf the long tail. Zipf's law describes this formally: the frequency of the item ranked k is proportional to 1/k — the second-most-popular item gets about half the traffic of the first, the tenth gets about a tenth, and so on.

The practical consequence, with numbers. Under a Zipf distribution, caching the top 1% of items by popularity commonly captures 60–80% of all requests, and the top 10% captures well over 90%. That is why a cache holding a small slice of your data can carry the great majority of your traffic, and it is the quantitative answer to "how big should the cache be?" — big enough to hold the working set, which you determine by measurement, not by fraction-of-total-data.

The corollary that decides real designs: past the working set, additional cache memory buys very little. Doubling from 1% to 2% of the dataset might take you from 70% to 75% hit ratio; doubling again buys less. So the right sizing procedure is: plot hit ratio against cache size on real traffic, find the knee of the curve, and provision a little past it. Buying memory beyond the knee is spending money on the long tail, which by definition is not being reused.

**When caching does not work — say this in an interview, it is a strong signal:** if access is genuinely uniform (random-ID lookups, a security scanner enumerating your ID space), or if data is written far more often than read (each write invalidates before the value is reused), or if every request is unique (a search over a huge query space with no repeats), a cache adds latency, memory cost, and a consistency hazard while buying nothing. Recognizing a workload that should not be cached is as valuable as knowing how to cache one.

2.7Sub-Concept: Key Design and Namespacing#

Since a cache can only answer questions you turned into keys (§0.4), key design is cache design. It is also where sloppiness produces outages.

A key is a contract. user:42:profile says: the entity type, the identity, and the aspect being cached. The conventional format is colon-separated segments, most-general to most-specific — app:entity:id:aspect:version. Concretely: shop:product:9981:detail:v3. The reasons this matters:

Everything that varies the answer must be in the key. This sounds obvious and is the most common cache bug in production. If a response depends on the user's locale, currency, feature-flag bucket, device type, or authorization level, and those are not in the key, then user A will be served user B's data. The failure is intermittent, user-specific, and often a privacy incident rather than a mere bug. Write the key as a pure function of every input that affects the value, without exception.

Key length is a real cost. A cache with 100 million entries and 60-byte keys spends 6 GB on keys alone, before a single value. Keep keys short but never at the expense of clarity — u:42:p saves bytes and costs you every future debugging session. If you must compress, hash the variable part (user:9f3c2a…) rather than abbreviating the readable part.

**The KEYS command is a production outage waiting to happen.** KEYS user:* scans the entire keyspace in one blocking operation; on a server with fifty million keys this blocks Redis's single thread (§0.5) for seconds, during which every other request from every client stalls. Use SCAN, which returns a cursor and walks the keyspace incrementally, yielding between batches. This is the single most-repeated piece of Redis operational advice in existence, and it is repeated because people keep doing it.

2.8Sub-Concept: Hot Keys and Big Keys#

These are the two ways a single key destroys a cluster, and they are distinct problems with distinct fixes.

The hot key problem. A cache cluster distributes keys across nodes by hashing the key (File 06 covers exactly how). That works beautifully — until one key becomes far hotter than any other. A celebrity's profile, the front-page item, a flash-sale product, a global feature-flag document: one key means one hash means one node. That node now receives a hundred thousand requests per second while its peers idle. You cannot shard your way out of it, because the unit of sharding is the key. Symptoms are unmistakable once you know them: one node at 100% CPU with the rest at 5%, and latency for all keys on that node degrading because they queue behind the hot one.

The fixes, in order of how often they are the right answer:

The big key problem. A single value that is enormous — a one-gigabyte list, a hash with ten million fields, a serialized blob of a whole catalogue. Three separate harms follow. (1) Blocking: Redis executes commands on one thread, so any operation touching that value — reading it, deleting it, serializing it for a replica — occupies the server for the entire duration, stalling every other client. Deleting a huge key with DEL can block for seconds; UNLINK exists precisely to free it in a background thread instead. (2) Network: every read ships the whole value across the network, so a 100 MB value read ten times a second is 8 Gbps of traffic from one key. (3) Imbalance: the node holding it uses vastly more memory than its peers, so cluster capacity planning becomes meaningless. The fix is always the same shape: split it — by field (use a hash and read single fields with HGET rather than HGETALL), by page (list:page:1, page:2), or by time bucket. Find them with redis-cli --bigkeys, which samples the keyspace and reports the largest key per type.

2.9Sub-Concept: Cache Coherence Across Tiers (the L1 invalidation problem)#

Once you put a local cache in every app server (§2.4), you have created copies you cannot delete with a single command, because they live inside N separate processes. Making those copies agree is the cache coherence problem — the same problem CPU designers solve in hardware for L1/L2/L3 caches, appearing again at the application layer.

Four strategies, from cheapest to most correct:

2.10Sub-Concept: Negative Caching#

Negative caching is caching the absence of a result — storing "this key does not exist" as a first-class cached fact. It is the primary defence against cache penetration (§6.1), and it is worth understanding as its own idea rather than as a trick.

Why it is needed. The ordinary cache-aside loop caches values. A lookup that finds nothing has no value to cache, so it caches nothing, so the next identical lookup also misses and also hits the database. For non-existent data — a deleted user, a mistyped ID, an attacker enumerating the ID space — the cache is completely transparent and the database absorbs 100% of the traffic. That is not a degraded cache; it is no cache at all, precisely when you are under attack.

The mechanism. Store a sentinel value (an empty marker, a null token — anything your code can distinguish from a real value) under the key, with a deliberately short TTL. Now the second request for that ID hits the cache and returns "not found" without touching the database.

The trade-offs, which are real. (1) Memory: an attacker sending random IDs is now filling your cache with negative entries, potentially evicting genuinely useful data — you have converted a database-load attack into a cache-pollution attack. Mitigate with short TTLs (30–60 seconds is typical) and, where possible, a separate memory budget for negatives. (2) Correctness at creation: if you cache "user 500 does not exist" and user 500 is created two seconds later, you will serve "not found" for the rest of the TTL. Mitigate by explicitly deleting the negative entry on creation — an easy step to forget, which is why the TTL must be short enough that forgetting it is survivable. (3) You must distinguish "not found" from "the database was down." Caching a failure as a negative result is how a thirty-second outage becomes a thirty-minute one. Only cache a negative when the origin authoritatively said the item does not exist.

The complementary defence is the Bloom filter (§6.1), which rejects known-absent keys before consuming any cache memory at all — the two are used together, not as alternatives.

2.11Sub-Concept: Cache Warming and the Cold-Start Problem#

A cold cache is an empty one — after a deploy that restarts your app servers (killing every L1 cache), after a Redis failover or restart, after adding a new node to a cluster, or after a FLUSHALL that someone will one day type in the wrong terminal. Every request is a miss, and your database instantly receives the full, unabsorbed production read load — traffic it has never handled, because the cache has been absorbing 95% of it since the day it was deployed.

This is the cold-start problem, and it is genuinely dangerous rather than merely slow: it is the single most common way a cache outage becomes a full site outage. The chain is: cache empties → database receives 20× normal load → database latency climbs → application threads block waiting on the database → thread pools exhaust → the application stops serving everything, including requests that would have been cache hits. That is a cascading failure, and it is why §6.2's avalanche and this are the same underlying risk seen from two directions.

Warming strategies, with their costs:

The measurement that tells you whether you are exposed: deliberately test it. Take a staging environment with production-like traffic, flush the cache, and watch what the database does. If the answer is "it falls over," you do not have a caching layer — you have a database that cannot serve your traffic, with a cache hiding that fact.

3Deep-Dive Mechanics: The Caching Patterns#

The caching pattern (a.k.a. caching strategy) defines how reads and writes flow between the application, the cache, and the database. This is the heart of File 02. There are five canonical patterns. Master all five and their trade-offs.

3.1Cache-Aside (a.k.a. Lazy Loading) — the most common#

The application is in charge; the cache is a dumb key-value store beside the DB.

Read path:

1. App asks cache for key K.
2. HIT  -> return value. Done.
3. MISS -> App reads K from database.
4. App writes K -> value into cache (usually with a TTL).
5. App returns value.

Write path:

1. App writes new value to the DATABASE.
2. App INVALIDATES (deletes) key K from the cache  (does NOT update it).
   -> next read will miss, reload fresh from DB, and repopulate.

3.2Read-Through#

Like cache-aside, but the cache library/service itself knows how to load from the DB on a miss (the app only ever talks to the cache).

Read path:

1. App asks cache for K.
2. HIT  -> return.
3. MISS -> the CACHE (via a loader function) fetches from DB,
           stores it, and returns it. App never touches the DB directly.

3.3Write-Through#

On a write, the app writes to the cache, and the cache synchronously writes to the DB before acknowledging.

Write path:

1. App writes K=V to the cache.
2. Cache writes K=V to the DATABASE (synchronously).
3. Only after DB confirms, the write is acknowledged to the app.

3.4Write-Back (a.k.a. Write-Behind)#

On a write, the app writes to the cache only, which acknowledges immediately, and the cache asynchronously flushes to the DB later (batched).

Write path:

1. App writes K=V to cache. Cache ACKs IMMEDIATELY.  (fast!)
2. Later, asynchronously (after a delay / in batches),
   the cache flushes dirty entries to the DATABASE.

3.5Write-Around#

On a write, the app writes directly to the DB, bypassing the cache entirely. The cache is only populated later on a read miss (i.e., write-around is usually paired with cache-aside reads).

Write path:

1. App writes K=V straight to DATABASE. Cache is NOT touched.
2. (If K was cached, either invalidate it or let TTL handle staleness.)

3.6The Read Path Race Condition (senior-level gotcha)#

Cache-aside has a famous concurrency bug:

Time →
Writer:  updates DB (K = new) ............ then deletes cache key K
Reader:  cache MISS, reads DB (K = old) ............... then writes cache K = old

If the reader's DB read happens before the writer's DB update, but the reader's "populate cache" happens after the writer's "delete cache," the cache ends up holding the old value permanently (until TTL). Mitigations: short TTLs (bounds the damage), delete-again after a delay (e.g., delayed double-delete), versioned keys, or using a proper write-through pattern. Knowing this race exists is a strong senior signal.

3.7Pattern Selection Cheat Table#

4Eviction Policies — Exhaustive Breakdown#

When the cache is full, which entry dies? This is a favorite interview topic (LRU implementation especially).

4.1LRU — Least Recently Used#

Evict the entry that hasn't been accessed for the longest time. Assumes temporal locality: recently used data is likely to be used again soon. The default policy in Memcached and a common Redis mode.

Implementation (know this — LeetCode-classic): a hash map (key → node) for O(1) lookup, combined with a doubly-linked list ordering nodes by recency. On access, move the node to the head (most-recent). On eviction, remove the tail (least-recent). Both operations O(1).

4.2LFU — Least Frequently Used#

Evict the entry with the lowest access count. Assumes popularity is stable. Keeps items that are accessed often, not just recently.

4.3FIFO — First In First Out#

Evict the oldest-inserted entry regardless of access. Simple queue. Rarely ideal (ignores usage), but cheap.

Mechanism. A single queue. Insert at the tail; evict from the head. Reading an entry does nothing — no reordering, no metadata update, no bookkeeping of any kind. That is the whole algorithm.

Worked example (capacity 3), inserting A, B, C then accessing A then inserting D:

   insert A, B, C   ->  queue: [A B C]        (A is oldest)
   access A         ->  queue: [A B C]        unchanged — FIFO ignores reads
   insert D         ->  evict A, queue: [B C D]

Compare that with LRU on the same trace: LRU would have moved A to the most-recent position on access and evicted B instead. FIFO discarded the item that had just proven it was still wanted — and that single difference is FIFO's entire weakness.

Why anyone uses it anyway. Because reads cost nothing. LRU must mutate a shared linked list on every read, which in a multi-threaded cache means taking a lock on the hottest path in the system — the read path — turning your cache into a contention point. FIFO's read path is lock-free by construction. Complexity: O(1) insert, O(1) evict, zero per-access work, and less metadata per entry than LRU (no prev/next pointers to maintain on read).

4.4Random Replacement#

Evict a random entry. Surprisingly not-terrible, needs no bookkeeping. Used where the metadata cost of LRU/LFU isn't worth it (some hardware caches).

Mechanism. Pick a slot uniformly at random; evict whatever is in it. No ordering structure exists at all.

Why it is not as bad as it sounds — the reasoning matters. Under a skewed (Zipf) workload (§2.6), hot items are a small fraction of entries but are re-inserted immediately after any unlucky eviction, while cold items are the majority and therefore are statistically more likely to be the random victim. Random replacement gets a surprising share of LRU's benefit for none of LRU's cost. Published comparisons typically put it a few percentage points of hit ratio behind LRU on realistic traces — worse, but not catastrophically so.

It also has a property LRU lacks: no pathological case. LRU can be defeated deterministically (a loop scanning N+1 items in a cache of size N produces a 0% hit ratio — every single access evicts the item that will be needed next). Random has no such adversarial trace, because there is no pattern to exploit. This is the same reasoning that makes randomized algorithms attractive elsewhere: predictable-average beats good-average-with-a-cliff when an adversary (or an unlucky access pattern) picks your input.

4.5MRU — Most Recently Used#

Evict the most recently used. Counterintuitive, but useful in specific patterns (e.g., a file being read once sequentially — once you've read a block you won't need it again soon).

Mechanism. Identical structure to LRU — a recency-ordered list — but evict from the head (most recent) rather than the tail.

Why it can possibly be right. LRU assumes temporal locality: what was just used will be used again. Some workloads have the exact opposite property, and for those, LRU is not merely suboptimal but actively worst-possible. The canonical case is a repeated sequential scan of data larger than the cache. Trace it with a cache of size 3 over a repeating scan A B C D A B C D…:

   LRU:  cache [A B C], next request D -> evict A (oldest)  -> cache [B C D]
         next request A -> MISS (just evicted!) -> evict B  -> cache [C D A]
         next request B -> MISS (just evicted!) ...
         every single access misses: 0% hit ratio.

   MRU:  cache [A B C], next request D -> evict C (newest)  -> cache [A B D]
         next request A -> HIT.  next request B -> HIT.
         retains a stable subset: ~50% hit ratio on this trace.

LRU achieves zero on a trace where MRU achieves half, because in a cyclic scan the most-recently-used item is precisely the one you will need last, and the least-recently-used is the one you will need next. This trace is worth memorizing — it is the cleanest demonstration that no eviction policy is universally correct, only correct for an access pattern.

4.5b Clock / Second-Chance — LRU's approximation, and why real systems use it#

Almost every operating system's page cache uses this rather than true LRU, so it belongs in the taxonomy.

The problem it solves. True LRU requires updating a shared ordering structure on every read. In an OS page cache, a "read" is a memory access happening billions of times a second — you cannot take a lock and splice a linked list on each one. Clock gets LRU-like behaviour with a single bit of work per access.

Mechanism. Arrange all entries in a circular buffer with a hand (like a clock face). Each entry has one reference bit. On access, hardware or software sets that entry's reference bit to 1 — that is the entire per-access cost, one bit write, no lock, no pointer surgery. On eviction, the hand advances: if the entry under the hand has reference bit 1, clear it to 0 and move on (the entry gets a "second chance"); if the bit is already 0, evict that entry.

   hand -> [A:1] [B:0] [C:1] [D:0]
   need to evict:
     A has bit 1 -> clear to 0, advance          [A:0] [B:0] [C:1] [D:0]
     B has bit 0 -> EVICT B

Why it approximates LRU. An entry accessed recently has its bit set, so it survives at least one full sweep of the hand. An entry not touched for a full revolution has a cleared bit and dies. The result orders roughly by recency at the granularity of "was this touched in the last sweep," which is enough.

4.5c Segmented LRU, 2Q, and LRU-K — making LRU scan-resistant#

LRU's defining weakness (§4.1) is that a single access promotes an item to the most-protected position. A one-time scan therefore evicts your entire hot set. This family of policies all fix it the same way: require evidence of reuse before granting protection.

SLRU (Segmented LRU). Split the cache into a small probationary segment and a larger protected segment. New items enter probation. An item accessed again while in probation is promoted into the protected segment; items evicted from probation are simply gone. Protected items that fall off the end demote back to probation rather than being evicted outright. Effect: a scan's items enter probation, are never re-accessed, and are evicted from probation without ever touching the protected hot set. Cost: one extra list and a tuning knob (the probation/protected split, commonly 20/80). This is a component of W-TinyLFU (§4.7).

2Q. The same idea with an explicit ghost queue: a FIFO queue for first-time entries, a second LRU queue for entries seen at least twice, and a small list of keys only (no values) for recently-evicted entries, so a returning key can be recognized as "seen before" and admitted straight to the hot queue. Cost: three structures and two tuning parameters; the ghost list costs memory proportional to key size, not value size — a good trade.

LRU-K. Track the timestamps of the last K accesses (typically K=2) and evict by the K-th most recent access time. An item accessed once has no second access, so its K-th access time is effectively infinitely old and it is evicted first, no matter how recent that single access was. This is the cleanest formulation of the idea: one access is noise, two accesses are a signal. Cost: K timestamps per entry and a priority queue rather than a list, so eviction becomes O(log n) rather than O(1) — which is why the approximations above are more common in practice.

4.6TTL-based / Volatile policies#

Redis offers policies like volatile-lru, volatile-ttl (evict the key with the shortest remaining TTL first), allkeys-lru, noeviction (reject writes when full). Choosing noeviction turns a full cache into write errors — a real outage source if misconfigured.

**All eight of Redis's maxmemory-policy values, since this is a concrete configuration decision you will actually make.** The policy applies once memory reaches maxmemory; the volatile-* family considers only keys that have a TTL set, while the allkeys-* family considers everything.

**The volatile-* trap you must know:** if you select a volatile-* policy and no keys have TTLs, Redis has nothing eligible to evict and behaves exactly like noeviction — writes start failing while memory sits full of keys it refuses to touch. This is a classic 3 a.m. incident, and the diagnosis is one line: check whether your keys actually have TTLs (TTL key returns -1 for "no expiry").

Redis's LRU is not LRU, and its LFU is not LFU. True LRU needs a global ordering structure; Redis refuses to pay for it. Instead, on each eviction it **samples maxmemory-samples random keys** (default 5) and evicts the best candidate from the sample, keeping a small pool of good candidates across calls. With 5 samples it approximates LRU closely; with 10 it is nearly indistinguishable from true LRU at modestly more CPU. This is §4.4's random-sampling insight applied deliberately. Likewise the LFU mode stores an 8-bit approximate counter per key that increments probabilistically (the higher the counter, the less likely a further increment — a logarithmic counter, so 8 bits can represent very large frequencies) and decays over time according to lfu-decay-time, which is how it solves the stale-popularity problem of §4.2. Knowing that these are approximations — and why the approximation is the right engineering choice — is a strong senior signal.

4.7Modern hybrids#

How ARC actually adapts, because "adaptively" is doing a lot of work in that sentence. ARC keeps four lists: T1 (items seen once — recency), T2 (items seen twice or more — frequency), and two ghost lists B1 and B2 holding the keys only (no values, so they cost almost nothing) of items recently evicted from T1 and T2 respectively. The adaptation is the elegant part: a hit in a ghost list is a signal that you sized wrongly. If a request hits in B1 — meaning "I evicted a recency item and immediately wanted it back" — ARC grows T1 at T2's expense. If it hits in B2, it grows T2 instead. The cache continuously re-tunes its own recency/frequency balance from live evidence, with no configuration knob at all.

4.8Comparison Table#

4.9The Decision Rule#

Say this, in this order:

5Cache Invalidation & Consistency#

"There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton

Invalidation is removing/updating cached data when the underlying source changes, so you don't serve stale data. Strategies:

Consistency reality: a cache introduces a second copy of data, so by definition you now have a consistency problem. Most caching accepts eventual consistency (the cache may briefly lag the DB). If you truly need read-your-writes or strong consistency, use write-through, or bypass the cache for critical reads, or version aggressively. State your consistency choice explicitly in interviews.

6The Three Cache Killers: Penetration, Avalanche, Stampede#

These three failure modes are the advanced caching interview questions. Know all three cold.

6.1Cache Penetration#

Problem: Requests for data that doesn't exist (in cache or DB). Every such request misses the cache and hammers the DB. Attackers exploit this by requesting random non-existent IDs (user id = -99999), and since the DB also returns "not found," nothing ever gets cached — every malicious request reaches the DB. The cache provides zero protection.

Solutions:

Sub-Concept: Bloom Filter. A Bloom filter is a space-efficient probabilistic data structure that answers "is X possibly in the set, or definitely not?" It's a bit array of m bits plus k independent hash functions. To add X, hash it k ways and set those k bits. To query X, check those k bits: if any is 0, X is definitely not in the set (no false negatives); if all are 1, X is probably in the set (false positives possible). It never says "no" wrongly, so it's perfect for "should I even bother querying?" It uses a tiny fraction of the memory of storing the actual keys. Trade-off: you can't delete easily (standard Bloom filters), and false-positive rate rises as it fills.

6.2Cache Avalanche#

Problem: A large number of keys expire at the same time (e.g., you loaded a million keys at startup all with TTL=3600, so they all expire in the same second), OR the cache cluster goes down entirely. Suddenly a flood of misses all hit the DB simultaneously → the DB is crushed → cascading failure.

Solutions:

6.3Cache Stampede (a.k.a. Thundering Herd / Dogpile)#

Problem: A single very hot key expires. In the instant it's gone, thousands of concurrent requests all miss simultaneously, and all of them try to recompute/refetch the same value from the DB at once — a "dogpile" that spikes the DB, even though they all want the identical value.

Solutions:

Distinction to keep straight: Penetration = data doesn't exist. Avalanche = many keys die together / cache down. Stampede = one hot key dies and everyone rushes it. Interviewers deliberately test whether you can tell these apart.

7HTTP Caching & Client-Side Semantics#

Everything so far has been your cache — one you deployed and control. But the single largest cache in front of your system is one you do not own: the tens of millions of browser caches, mobile HTTP caches, corporate proxies, and CDN edges that will store your responses whether you thought about it or not. HTTP has a complete, standardized caching protocol (RFC 9111), and it is a canonical part of this topic — an interview answer about caching that never mentions Cache-Control or ETag has a hole in it. This section teaches that protocol from scratch.

7.1The two questions every HTTP cache asks#

An HTTP cache holding a stored response must answer two questions, and every header below serves one of them:

Freshness avoids the round trip; validation avoids the payload. Real systems use both.

7.2Cache-Control — every directive that matters#

Cache-Control is the primary header, sent by the origin on a response (and sometimes by the client on a request). Its directives split into three groups.

Who may store it:

How long it stays fresh:

What happens when it goes stale:

7.3Validators — ETag and Last-Modified#

A validator is a token the origin attaches to a response so that a cache can later ask "still current?" cheaply.

**ETag (entity tag)** is an opaque identifier for a specific version of a resource — typically a hash of the content, e.g. ETag: "9f3c2a1b". On revalidation the cache sends If-None-Match: "9f3c2a1b"; the origin compares, and either returns 304 Not Modified (empty body — the cache keeps and refreshes its copy) or 200 with the new content and a new ETag. Strong vs weak: ETag: "abc" asserts byte-for-byte identity; ETag: W/"abc" (weak) asserts only semantic equivalence, which is what you want when a response is regenerated with a different timestamp or whitespace but the same meaning. Strong ETags are required for range requests (resuming a partial download) to be safe — a client resuming byte 5,000,000 of a file must be certain it is the same file.

**Last-Modified** is the cruder alternative: a timestamp. The cache sends If-Modified-Since: <date> and gets 304 or 200. Its weaknesses are why ETag exists: it has one-second granularity (so two changes in the same second are indistinguishable), it cannot express "changed and then changed back to identical content," and it depends on server clocks. Use ETag where you can; send both if you want maximum compatibility.

The cost of ETags nobody mentions: in a load-balanced fleet (File 01), every server must generate the same ETag for the same content, or a client hitting a different server on revalidation gets a spurious 200 with a full body and your bandwidth savings evaporate. If your ETag derives from an inode number, a build timestamp, or anything machine-local, you have built exactly that bug. Derive it from the content.

7.4Vary — the header that prevents serving the wrong user's data#

A shared cache keys stored responses by URL. But the same URL can legitimately produce different responses depending on request headers — a different language, a different compression, a different device layout. **Vary tells the cache which request headers are part of the cache key.**

Vary: Accept-Encoding means "store gzip and brotli variants separately." Vary: Accept-Language means "store per language." **Vary: Cookie means "store per cookie value"** — which, since cookies are per-user, effectively disables shared caching entirely, because every user has a distinct key.

Two failure modes, both severe and both common:

7.5Where this connects#

Every mechanism in §3–§6 reappears here in protocol form: max-age is TTL (§2.2), stale-while-revalidate is the stampede defence (§6.3), stale-if-error is graceful degradation (§8.2), Vary is key design (§2.7), and immutable + content hashing is the client-cache invalidation answer from §2.4. The one thing HTTP caching cannot give you is invalidation — you cannot reach into a browser cache — which is why the correct posture is: short freshness for HTML, long freshness plus content-hashed URLs for assets, and private on anything user-specific. File 05 (CDNs) picks this up from the edge's perspective.

8Pros, Cons & Trade-offs#

8.1Benefits#

8.2Costs & Trade-offs#

9Interview Diagnostic Framework (Symptom → Solution)#

10Real Implementations & Product Landscape#

"Add a cache" is not an implementation. This section is the catalogue: what each real caching product is, the architectural decision that distinguishes it from its siblings, what it is genuinely best at, and which one to pick when. §2.5 gave the two-line interview answer; this gives the actual landscape.

10.1Family 1 — Remote in-memory data stores (the "distributed cache" tier)#

Memcached. Released in 2003 by Brad Fitzpatrick for LiveJournal, and still the most-deployed cache you never hear about. Its defining property is deliberate minimalism: a flat key → opaque-blob store, nothing else. No data structures, no persistence, no replication, no cluster protocol — the client library decides which server holds which key (by consistent hashing, File 06), and the servers do not know about each other at all. That shared-nothing design is why it scales so cleanly: adding a server requires no coordination whatsoever.

Its second distinguishing property is that it is multi-threaded, so one Memcached instance uses every core on the box. Redis (classically) does not. On a 16-core machine serving a simple get/set workload, this is a real and measurable throughput advantage.

Its memory management is slab allocation: memory is carved into pages assigned to size classes (a 96-byte class, a 120-byte class, and so on, growing by a configurable factor), and each item is stored in the smallest class that fits. Benefit: no external fragmentation and O(1) allocation — critical for a process running for months. Cost: internal fragmentation (a 100-byte value in a 120-byte slot wastes 20 bytes) and slab calcification — once pages are assigned to a size class they are not easily reassigned, so a workload whose value sizes change over time can end up with memory stranded in the wrong classes and evictions happening while memory sits free. That is Memcached's signature operational surprise.

Redis. Released in 2009 by Salvatore Sanfilippo, and the default answer for good reason: it is not a cache but an in-memory data-structure server that is very frequently used as a cache. The architectural choice that defines it is a single-threaded command execution loop — every command runs to completion with no other command interleaving. That is the source of both its greatest strength and its sharpest limitation. The strength: every operation is atomic for free (§0.5), which is what makes Redis usable for counters, locks, rate limiters (File 03), and queues without any locking protocol. The limitation: one slow command blocks everything — a KEYS * (§2.7), a big-key DEL (§2.8), or a slow Lua script stalls every client on that shard. (Redis 6+ added threaded I/O for reading and writing sockets, and Redis 7 continues that, but command execution remains single-threaded by design.)

What you actually get on top of get/set: strings, hashes, lists, sets, sorted sets (the structure behind every leaderboard and every sliding-window rate limiter), bitmaps, HyperLogLog (§2.5), streams (a durable log, File 07), geospatial indexes (File 16), pub/sub (§2.9), Lua scripting, transactions, persistence (RDB snapshots and AOF, §2.5), replication, Sentinel (automated failover for a primary/replica pair, §6.2) and Cluster (16,384 hash slots sharded across primaries, §6.2).

Valkey. In March 2024 Redis Ltd. changed Redis's licence away from open source (to RSALv2/SSPL); the Linux Foundation forked the last BSD-licensed version as Valkey, backed by AWS, Google, and Oracle, and it is now the drop-in open-source continuation, wire-compatible with Redis and adding multi-threaded execution work of its own. You need to know this because the "just use Redis" answer now has a licensing dimension: managed services from major clouds have moved to Valkey, and if your organization cares about open-source licensing, Valkey is the answer to the same requirements. Naming this in 2026 signals you are current.

Dragonfly and KeyDB attack Redis's single-thread limit directly while staying wire-compatible, so existing clients work unchanged. KeyDB is a multi-threaded Redis fork. Dragonfly is a ground-up rewrite using a shared-nothing thread-per-core architecture with a modern hash table and a novel snapshotting algorithm, claiming order-of-magnitude throughput improvements on a single large machine. The trade you are making: far better vertical scaling and simpler operations (one big node instead of a cluster) in exchange for a smaller ecosystem, less battle-testing, and dependence on a smaller vendor. Reach for them when you have a genuine single-node throughput ceiling and would otherwise be forced into cluster complexity.

Hazelcast, Apache Ignite, Infinispan are the JVM-native distributed caches / in-memory data grids. They embed inside your Java processes (or run standalone), replicate and partition data across the cluster automatically, and add compute-near-data features (distributed execution, continuous queries, SQL over cached data). Best at: JVM shops wanting a cache that is also a distributed data structure library, with strong consistency options. Limits: heavy, JVM-only in practice, and operationally more complex than Redis. Reach for them when you are deep in a Java ecosystem and need embedded plus distributed in one; not when a language-neutral cache serving a polyglot fleet is the requirement.

10.2Family 2 — In-process (local / L1) caches#

Caffeine (Java). The current best-in-class local cache, and the reference implementation of W-TinyLFU (§4.7). It achieves near-optimal hit ratios, does its bookkeeping asynchronously in ring buffers so the read path is nearly lock-free, and supports size-based, time-based, and reference-based eviction plus automatic loading (read-through, §3.2) and refresh-ahead. Reach for it when you are writing JVM code and want an L1 cache — it is essentially the default, having superseded Guava Cache (its predecessor by the same author, still widely deployed, simpler, LRU-ish, and meaningfully slower).

Ehcache is the older JVM standard, notable for tiered storage (heap → off-heap → disk), which lets you hold far more than heap allows without GC pressure — its real differentiator, since a large on-heap cache directly worsens GC pauses (§2.4).

Outside the JVM: **lru_cache / cachetools in Python, go-cache/ristretto** in Go (Ristretto being explicitly TinyLFU-inspired), and in Node the lru-cache package. The common warning applies to all: an in-process cache in a multi-process server (Python/PHP under a pre-fork model, Node with cluster) is one cache per worker, multiplying memory use and divergence in a way people consistently under-estimate.

10.3Family 3 — HTTP / reverse-proxy caches#

Varnish is a caching HTTP reverse proxy whose distinguishing feature is VCL (Varnish Configuration Language) — a small domain-specific language, compiled to native code, in which you write the caching logic itself: how to build a cache key, when to bypass, how to handle cookies, how to purge. That programmability is why it powers large media sites. It also does request coalescing natively, so a stampede on a hot URL (§6.3) collapses into one origin fetch automatically. Limits: historically no TLS termination (you front it with something that does, File 01), memory-focused, and VCL is a real language to learn. Reach for it when you are caching HTTP responses at high volume with non-trivial rules.

**NGINX proxy_cache** gives you a competent disk-backed HTTP cache inside the load balancer you already run (File 01 §9.2), with proxy_cache_lock for coalescing and proxy_cache_use_stale implementing stale-if-error (§7.2). Reach for it when you already run NGINX and want caching without a new component — which is very often the right call.

Apache Traffic Server is the heavy-duty, disk-optimized option used by large CDNs; Squid is the veteran forward proxy, still relevant for outbound/corporate caching rather than serving your users.

10.4Family 4 — Managed cache services#

AWS ElastiCache is managed Redis/Valkey or Memcached: it handles provisioning, patching, backups, and failover, and is the default choice on AWS. AWS MemoryDB looks similar but is a different product: it is a Redis-compatible durable database that writes to a multi-AZ transaction log, so it is the answer when the in-memory data is the source of truth and cannot be lost — at meaningfully higher cost than ElastiCache. Confusing these two is a real design error: using ElastiCache where you needed durability loses data on failover; using MemoryDB as a plain cache overpays substantially. Amazon DAX is a third, narrower product — a write-through cache purpose-built for DynamoDB, sitting transparently in front of it and turning single-digit-millisecond reads into microsecond ones without any application-level cache-aside code.

Google Memorystore, Azure Cache for Redis, and Upstash (serverless, per-request pricing, HTTP API — notable because it works from edge/serverless runtimes that cannot hold persistent TCP connections) round out the managed options. Momento is a serverless cache API with no instances to size at all.

The universal trade with managed caches: you buy away the operational burden (failover, upgrades, monitoring) and give up some control (restricted commands, fixed configuration, no custom modules) at a price premium over self-hosting on raw instances. For nearly all teams that is the correct trade — the cost of a self-managed Redis cluster is not the instances, it is the 3 a.m. failover you have to debug yourself.

10.5The comparison table#

10.6Decision rules#

10.7Anti-recommendations#

11Real-World Engineering Scenarios#

11.1Facebook — Memcached at Massive Scale (the "scaling memcache" paper)#

Facebook runs one of the world's largest cache-aside deployments over Memcached, fronting MySQL.

11.2Twitter/X — Timeline Fan-out Cache with Redis#

Home timeline rendering is read-dominated and latency-critical.

11.3Reddit — Caching Rendered Content & Vote Counts#

12Interview Gotchas & Failure Modes#

12.1Classic Questions#

12.2Failure Modes & Mitigations#

13Whiteboard Cheat Sheet#

   READ (cache-aside):                        WRITE options:
   ┌──────┐  hit   ┌───────┐                  write-through: app->cache->DB (sync, consistent)
   │ App  │───────►│ Cache │                  write-back:    app->cache (ack), async->DB (fast, lossy)
   └──┬───┘        └───┬───┘                  write-around:  app->DB, cache filled on read
      │ miss           │ populate(+TTL,jitter) cache-aside W: app->DB, then DELETE cache key
      ▼                ▲
   ┌──────────────────┴┐
   │     Database      │   (source of truth)
   └───────────────────┘

Eviction: LRU (recency) | LFU (frequency, +aging) | W-TinyLFU (modern, scan-resistant) | TTL/volatile
Killers:  Penetration (missing data) -> cache nulls + Bloom filter
          Avalanche  (mass expiry/cache down) -> TTL jitter + cache HA
          Stampede   (hot key expiry) -> mutex/single-flight + stale-while-revalidate
Metric:   hit ratio = hits/(hits+misses)   -> aim high (>90%); jitter every TTL

One-sentence summary for an interviewer:

"Caching trades memory for latency by keeping hot data in RAM; pick cache-aside for general read-heavy loads (invalidate-don't-update on write), write-through when you need read-after-write freshness, and write-back for high-write loss-tolerant counters; bound staleness with jittered TTLs, choose an eviction policy deliberately (LRU/LFU/W-TinyLFU), and explicitly defend the database against penetration, avalanche, and stampede."

End of File 02. Next → File 03: Rate Limiting (token bucket, leaky bucket, sliding window, distributed limiting with Redis).