System Design: Modern Caching Strategies & Distributed Systems

In distributed systems design, data access speeds vary wildly: CPU registers operate in nanoseconds, RAM access takes ~100 nanoseconds, while disk-backed database queries and network hops take tens or hundreds of milliseconds. Caching is the primary architectural mechanism used to bridge this immense speed disparity.

However, introducing a cache layer brings significant architectural complexity. As Phil Karlton famously observed: "There are only two hard things in Computer Science: cache invalidation and naming things." In this comprehensive system design guide, we analyze the 4 core caching patterns, eviction strategies, and strategies to defend against catastrophic cache stampedes.

1. The Four Primary Caching Patterns

1. Cache-Aside (Lazy Loading)

The application coordinates both the cache and database. When data is requested, the app inspects the cache first. If a cache miss occurs, the app reads from the database, writes the result to the cache, and returns it to the client.

• Pros: Cache only contains requested data; resilient to cache node failures (falls back to DB).
• Cons: Initial cache miss penalty; potential stale data if DB is updated without invalidating cache.

async def get_article_data(article_id: str):
    cache_key = f"article:{article_id}"
    
    # 1. Attempt cache retrieval
    cached = await redis_client.get(cache_key)
    if cached:
        return json.loads(cached)
        
    # 2. Cache miss: Fetch from primary relational DB
    article = await postgres_db.fetch_one("SELECT * FROM articles WHERE id = $1", article_id)
    if not article:
        # Cache negative response for 60s to prevent Cache Penetration
        await redis_client.set(cache_key, json.dumps(None), ex=60)
        return None

    # 3. Populate cache with TTL (1 hour)
    await redis_client.set(cache_key, json.dumps(dict(article)), ex=3600)
    return article

2. Write-Through Caching

The application writes data to the cache, and the cache synchronously writes to the database before returning a success acknowledgment. Ensures the cache is always 100% consistent with the database, but adds write latency.

3. Write-Behind (Write-Back) Caching

The application writes immediately to the in-memory cache and acknowledges success. An asynchronous background worker batches updates and persists them to disk. Delivers massive write throughput (used in heavy logging, analytics, and gaming leaderboards), but risks data loss if the cache node crashes before flushing.

4. Refresh-Ahead Caching

The cache engine automatically reloads frequently accessed keys from the database before their TTL expires, ensuring hot keys never experience latency penalties on user queries.

2. Cache Eviction Policies

Because in-memory RAM is significantly more expensive than disk storage, caches operate with fixed memory allocations. When memory exhausts, the engine must evict items:

• LRU (Least Recently Used): Evicts the item that has not been read for the longest period. Ideal for general web feeds and e-commerce catalogs.
• LFU (Least Frequently Used): Counts query frequencies and evicts items with the lowest access counts. Great for stable, predictable reference data.
• FIFO (First In First Out): Simple queue eviction regardless of access patterns. Low CPU overhead, but less cache-hit efficient.

3. The Three Great Caching Failure Modes

Frequently Asked Questions (FAQ)

Conclusion

Caching is the most cost-effective lever to scale modern distributed systems. By selecting the appropriate caching pattern, configuring LRU/LFU eviction policies, and applying TTL jitter, you protect your core database while maintaining sub-millisecond API response times.

Production Redis Distributed Mutex (Locking) Script

Prevent cache stampedes under heavy traffic with atomic Redis operations:

import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL);

export async function fetchWithMutexLock(
  key: string,
  fetcher: () => Promise,
  ttlSeconds = 600
): Promise {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const lockKey = `lock:${key}`;
  const acquired = await redis.set(lockKey, 'locked', 'EX', 5, 'NX');

  if (acquired) {
    try {
      const freshData = await fetcher();
      await redis.set(key, JSON.stringify(freshData), 'EX', ttlSeconds);
      return freshData;
    } finally {
      await redis.del(lockKey);
    }
  }

  // Wait 80ms and retry reading warm cache
  await new Promise(r => setTimeout(r, 80));
  return fetchWithMutexLock(key, fetcher, ttlSeconds);
}