DNS Cache Shrink: How Cloudflare Saved 100TB of RAM on 1.1.1.1

The 100TB Problem Nobody Saw Coming

1.1.1.1 is not a small service. It handles a frankly absurd volume of DNS queries—over a trillion per day at peak. Behind that single IP sits a global anycast network of servers running a custom DNS stack. Every one of those servers maintains a local cache of DNS responses to keep latency in the single-digit milliseconds. The cache is the critical path. If it misses, the resolver has to walk the DNS hierarchy from root to authoritative, which burns time and compute. So you cache aggressively.

For years, Cloudflare used a standard std::unordered_map—the C++ standard library's hash table—to store cached records. It worked. Until it didn't.

When the engineering team profiled memory usage across the fleet, they found something alarming: the hash table alone was consuming over 100 terabytes of RAM globally. Not the cached data itself. The overhead of the data structure. Bucket pointers, empty slots, alignment padding, per-node allocation metadata. All the invisible tax that comes with a general-purpose container tuned for safety and flexibility, not for the specific access pattern of a DNS resolver.

To put 100TB in perspective: that's roughly 1,000 servers worth of memory at 128GB per box. Not chump change when you're running a free resolver at planetary scale. The fix wasn't to buy more RAM. It was to throw out the general-purpose data structure and build one that understood the domain.

The Architecture: From General-Purpose to Specialized

The team replaced std::unordered_map with a custom open-addressing hash table backed by an arena allocator. Let's unpack what that means and why each decision mattered.

Open Addressing vs. Chaining

std::unordered_map typically uses separate chaining: each bucket holds a linked list of nodes. Every node is a separate heap allocation. On a 64-bit system, each allocation carries 16-24 bytes of malloc metadata, plus the pointers linking the list. For a cache with hundreds of millions of tiny entries—many DNS responses are under 100 bytes—the overhead-to-payload ratio can exceed 2:1. You're storing more bookkeeping than data.

Open addressing flips this. There are no linked lists. The table is a flat array of slots. When a collision occurs, you probe linearly to the next slot. The entire table is one contiguous allocation. The cost? You need to be careful about load factor and deletion (tombstones). The win? Zero per-entry allocation overhead and excellent cache locality.

Arena Allocation: One malloc, Many Objects

Even with open addressing, you still need to store the variable-length DNS record data somewhere. The naive approach is to new each record's buffer. Cloudflare's insight: the cache has a well-defined lifetime per server process, and records are evicted in large batches based on TTL. Why pay per-allocation overhead when you can allocate a giant slab once and bump a pointer?

An arena allocator works exactly like that. You mmap a large contiguous region. To "allocate" a record, you bump a pointer forward by the needed size and hand back the old pointer. No free lists, no coalescing, no metadata per allocation. When a record expires and gets evicted, you don't even bother freeing it immediately—you just mark the hash table slot as empty. The arena is cleaned up in bulk when the entire region is recycled.

Domain-Specific Optimizations

This is where it gets good. The team didn't stop at generic data structure replacements. They baked DNS-specific knowledge into the design:

• Inline storage for small records. Most DNS responses are tiny—an A record is 4 bytes of payload. For records below a threshold, the data lives directly in the hash table slot, avoiding the arena entirely. One less pointer chase.
• TTL-aware eviction. Instead of a separate expiry thread constantly scanning, the lookup path itself checks TTLs. If a record is stale, the slot is reclaimed on the spot. This eliminates the need for a background sweeper and its associated metadata.
• Key compression. DNS names are hierarchical and repetitive. Storing the full "www.example.com" string for every record is wasteful. The cache uses a string interning scheme where domain suffixes are shared, drastically reducing the bytes spent on keys.

The Result

After the rewrite, per-entry overhead dropped from roughly 80-120 bytes to around 8-16 bytes. Globally, that freed over 100TB of RAM. The same servers could handle the same query load with substantially less memory, or handle more load with the same memory. Latency improved because the working set fit more cleanly in CPU caches—fewer L3 misses, fewer TLB misses.

Why This Matters for Forward Deployed Engineers

If you're an FDE, you live in the gap between prototype and production. You ship features into customer environments where you don't control the infrastructure. Memory is often the scarcest resource—especially when you're deploying LLM inference, vector databases, or real-time data pipelines alongside a customer's existing stack.

This Cloudflare story is a masterclass in three FDE-relevant skills:

1. Profile before you optimize. The team didn't guess where memory was going. They instrumented the resolver, broke down memory by component, and let the data point at the hash table. In your world, this might be a Python process ballooning because of a pandas DataFrame holding strings as Python objects instead of using the category dtype. Same pattern, different scale.

2. Understand your access patterns. A DNS cache has a specific shape: write-once (when a query resolves), read-many (until TTL expires), delete-in-bulk (when TTLs lapse). No random insertions, no resizing under load. Cloudflare exploited every one of those constraints. When you're building a feature that caches LLM embeddings, ask: are you ever updating an embedding in place? If not, you can use an append-only structure and skip the complexity of mutability.

3. General-purpose tools are a starting point, not the destination. std::unordered_map is a fine default. Redis is a fine cache. PostgreSQL is a fine database. But when you're deploying into a constrained edge environment—say, an on-prem GPU box running your LLM feature with guardrails—the overhead of a general-purpose solution can be the difference between fitting in memory and swapping to disk. Knowing when to replace the off-the-shelf component with something bespoke is a core FDE judgment call.

The lesson isn't "rewrite your hash tables." It's "your abstractions have a memory tax, and at scale, that tax dominates."

How to Apply These Patterns Today

You don't need a trillion-QPS resolver to benefit from these ideas. Here's a practical progression, from quick wins to full rewrites.

Quick Win: Audit Your Hash Maps

In any performance-critical service, grep for HashMap, dict, unordered_map, or your language's equivalent. For each one, ask:

• What's the average entry size?
• What's the insert/delete pattern?
• Is the key already a hash, or are we rehashing strings?

If entries are small and numerous, consider a specialized library. In C++, absl::flat_hash_map or robin_hood use open addressing and often halve memory vs. std::unordered_map. In Rust, hashbrown (which powers std::collections::HashMap) already uses open addressing, but you can tune the load factor. In Python, __slots__ on objects and array.array for numeric data can cut per-object overhead from 56 bytes to single digits.

Intermediate: Arena Allocation in Practice

Arenas are surprisingly easy to bolt onto existing code. The pattern:

// Simplified arena for fixed-size records
class Arena {
    char* region;
    size_t offset = 0;
    size_t capacity;
public:
    Arena(size_t size) : region(new char[size]), capacity(size) {}
    
    void* alloc(size_t n) {
        if (offset + n > capacity) return nullptr; // or recycle
        void* ptr = region + offset;
        offset += n;
        return ptr;
    }
    
    void reset() { offset = 0; } // bulk-free
};

The constraint: you cannot free individual allocations. This is perfect for request-scoped data in a server, or for cache entries that expire in batches. If you're building a resume tailoring agent that processes one job description at a time, an arena per request eliminates GC pressure and allocation churn.

Advanced: Domain-Specific Cache Design

If you're building a cache from scratch, steal Cloudflare's playbook:

1. Measure your key and value size distributions. Plot a histogram. If 90% of values are under 128 bytes, inline them.
2. Exploit TTL semantics. If your cache entries have known lifetimes, batch evictions. Don't build a general-purpose LRU if you don't need one.
3. Share common prefixes. DNS names share suffixes. Your cache keys might share namespaces, tenant IDs, or API version prefixes. A simple interning table (a set of unique strings, referenced by pointer) can deduplicate gigabytes of repeated strings.

For a concrete example: imagine you're caching embeddings from a vector DB. Each embedding is 1536 floats (6KB). The cache key is a hash of the input text. You don't need a general-purpose hash map—a flat array indexed by the lower N bits of the hash, with linear probing, is simpler and faster. The embeddings themselves go in an arena. When the cache fills, you don't evict one-by-one; you reset the entire arena and clear the hash table. Crude, but for many FDE use cases, perfectly adequate.

If you're instrumenting a pipeline that scrapes customer reviews for sentiment analysis—something like the G2/Trustpilot dashboard—pay attention to how intermediate DataFrames allocate. A df.copy() that doubles memory for a 10GB dataset hurts. Chaining operations in-place and using category dtypes for repetitive strings can recover gigabytes without a single algorithm change.

A Balanced Look: The Trade-offs

This isn't a "standard libraries are bad" sermon. Cloudflare's custom cache is faster and leaner, but it came with real costs:

Correctness burden. std::unordered_map has been battle-tested by millions of programs. A custom open-addressing table with tombstones, TTL-based eviction, and arena allocation is a breeding ground for subtle bugs. Dangling pointers, ABA problems on slot reuse, race conditions on TTL checks—these are footguns that a standard container handles for you. Cloudflare has the engineering muscle to test and verify this code. Your three-person FDE team might not.

Flexibility loss. The custom cache is tightly coupled to DNS semantics. If the resolver ever needs to support a new record type with different caching rules, the cache code might need surgery. A general-purpose map would absorb that change trivially.

Maintenance overhead. The next engineer who joins the team has to learn the custom cache. There's no StackOverflow for Cloudflare's internal data structures. Documentation, onboarding, and debugging all get harder.

The scale threshold. Below a certain size, these optimizations are negative value. If your cache holds 10,000 entries, the overhead of std::unordered_map is maybe 2MB. Who cares? The engineering time spent optimizing it would be better spent on features. Cloudflare crossed the threshold where 100TB of RAM was burning real money. Your threshold is different.

The engineer's judgment is knowing where that line sits. Profile first. If the overhead is under 10% of your memory budget, stop. If it's 50%, you have a mandate.

FAQ

Q: Could Cloudflare have just used Redis or Memcached for the DNS cache?

A: Not at this scale and latency requirement. General-purpose caches like Redis add network round-trips (even on localhost, that's microseconds) and serialize/deserialize overhead. At 1.1.1.1's query rates, every microsecond in the cache path translates to millions in tail latency. An in-process, memory-mapped data structure is the only way to hit sub-millisecond P99s.

Q: What language is the 1.1.1.1 resolver written in?

A: The core resolver is written in C++, which gives the team direct control over memory layout and allocation. Rust would be a strong candidate for a similar project today, with hashbrown and arena crates like bumpalo offering comparable control with memory safety guarantees.

Q: How does this relate to the "small allocations are expensive" advice I keep hearing?

A: It's exactly that principle, applied at scale. Every heap allocation carries 16-24 bytes of allocator metadata, plus fragmentation. When your average record is 50 bytes, that's a 30-50% overhead per record. Arena allocation collapses thousands of small allocations into one large one, amortizing the overhead to near zero.

Q: Should I rewrite my application's hash maps?

A: Almost certainly not as a first step. Profile. If memory overhead is genuinely a bottleneck, start by swapping in a specialized library (absl::flat_hash_map, Rust's hashbrown with tuned load factor, Python's __slots__). Only build a custom data structure when the domain constraints are so specific that no library fits. The full Cloudflare approach is warranted for maybe 0.1% of projects. But the thinking behind it—understand your access patterns, challenge general-purpose defaults, profile relentlessly—applies to every project.

Q: How do I learn to do this kind of systems-level optimization?

A: The best way is to build things that push against resource limits—embedded systems, game engines, high-frequency trading simulators. For FDEs specifically, the pattern is: deploy a feature, watch it strain under real customer load, profile the hot path, and optimize surgically. FDE Coach's handoff guide covers the full cycle of taking a prototype to production-grade, including when to hand off optimization work to core engineering. The Cloudflare DNS cache is what the end of that maturity curve looks like.

Further reading: The original Cloudflare blog post with full implementation details is at https://blog.cloudflare.com/dns-cache-memory-optimization-1111/.