Cloudflare recovers more than 100TB of RAM by optimizing consistent hashing in Rust

Cloudflare published details of an optimization that saved more than 100TB of RAM across its entire global network, achieved in a single internal service: the Pingora Backend Router (PBR), responsible for balancing cacheable requests across servers. The gain came from two fronts that rarely appear together in a technical post: an exact mathematical derivation of a statistical formula and a low-level adjustment to a Rust struct. Neither one alone would have solved the problem. The full account is on Cloudflare's blog.

The ticket that started it all

The story begins with an internal ticket opened by Ivan Babrou: excessive memory usage in the pingora-ketama library, used by the PBR for consistent hashing. In some cases, memory consumption reached 6GB for this structure alone. To understand why this happened, the team (Kevin Guthrie, Mariia Iurchenko, and Zaidoon Abd Al Hadi, along with Ivan himself) had to revisit how the consistent hashing algorithm is implemented in practice, not just in textbook theory.

Why consistent hashing, and why it bloats

Consistent hashing allows tasks (in this case, cache keys per URL) to be mapped to servers in a stable way: when a server joins or leaves, only a small fraction of the keys need to be remapped. In practice, each server and task hash falls on an integer (32, 64, or 128 bits), and the task goes to the first server to the right of that number on a line (or ring) of hashes.

The problem is that, with a single hash per server, load distribution is terrible. Cloudflare calculated the expected value and standard deviation of the fraction of load each server receives in a distribution with N servers: the expected value is 1/N, but the relative standard deviation (the so-called coefficient of variation, CV) is approximately equal to the square root of (N-1)/(N+1). With 100 servers, this gives a CV of about 99%, which in practice means some servers can receive twice the traffic they should while others sit nearly idle.

The classic solution: multiply hashes, with weight

The known workaround is to give each server multiple points on the ring, not just one. The more points, the more the law of large numbers balances the distribution. NGINX uses 160 points per server as a default, and Pingora followed the same number. With 160 hashes per server, the CV drops from 99% to about 8% in the 100-server example, a huge improvement.

Cloudflare also uses a variation called ketama to give each server a different weight: if a server should receive w times more traffic than another, it gets w times more hashes on the ring. In the PBR's case, the weight is proportional to each node's disk space, since the service deals with caching. It's this weighting scheme, multiplied by the need for separate rings for each combination of features (compliance, enabled cache features), that generates dozens of distinct rings held in memory simultaneously, and explains the 6GB that Ivan's ticket flagged.

The Rust trick that wasn't enough on its own

Each point on the ring is represented by a simple struct:

struct Point {
 hash: u32,
 index: u32,
}

Eight bytes: four for the hash, four for the index pointing to the server in a separate array. Zaidoon noticed that the index would never need more than 16 bits (the PBR doesn't coordinate more than 65,000 servers at once), so a u16 would suffice. But changing the type alone doesn't change anything:

struct PointV2 {
 hash: u32,
 index: u16,
}

Rust's memory alignment rules require that a struct's size be a multiple of the size of its largest field. Since the 32-bit hash is the largest field, the compiler still reserves 8 bytes, even with the smaller index. The solution wasn't to use #[repr(packed)] (which the team avoids for well-known security and portability reasons), but rather to store the raw bytes in an array and access them via getters:

struct Point([u8; 6]);

impl Point {
 fn hash(&self) -> u32 {
 u32::from_ne_bytes(self.0[0..4].try_into().unwrap())
 }
 fn index(&self) -> u16 {
 u16::from_ne_bytes(self.0[4..6].try_into().unwrap())
 }
}

The change alone reduced memory used for consistent hashing by 25%. But the bigger gain was still to come, and it depended on math, not code.

Why 90% of the hashes were waste

The coefficient of variation formula cited above only holds for a single hash per server. For k hashes per server, most sources only offer approximations. Cloudflare's team derived the exact formula (the full derivation is in a companion post linked in the original article) and reached a practical conclusion: each reduction in error requires, roughly, an order-of-magnitude increase in the number of hashes. With the average weight of 625 used internally, the PBR was generating 160 × 625 = 100,000 hashes per server, and the last 90,000 of those hashes reduced the error by only 0.7%.

Worse: since 32-bit hashes have a finite space, collisions become more frequent as the number of hashes grows (the classic birthday paradox). Simulations showed that, for data centers with 2048 servers, the actual error increases between 10,000 and 100,000 hashes per server, precisely the range in which the PBR was operating. With that data in hand, the team cut the number of hashes generated per server by 90%, with no perceptible loss of balancing precision.

How to migrate without taking down the origin

Swapping the hashing ring changes where each cache key points to. A single global cutover would invalidate almost the entire cache at once and send a flood of traffic straight to customers' origin servers. The solution was to keep both versions of the ring (the old one and the new, compact one) in memory at the same time during the transition, with the internal migration framework deciding, based on the request hash, which ring to use. This gave two advantages: the decision was stable per request, and there was a clean rollback path, without needing to redeploy the PBR.

The rollout was done in layers: first small validation locations, then progressively larger groups of data centers, controlling two variables separately: how much traffic used the new ring and in which data centers that happened. A global percentage rollout would have spread cache churn across the entire network at once; scoping by data center kept the blast radius small. The team monitored backend selection traces, ring version counters, PBR connection errors, process memory, startup time, cache behavior, and origin traffic throughout the entire process. Upon reaching 100%, the old path was removed, resulting in a drop of 100TB of memory used globally, a sum that adds to another 100TB that Cloudflare's DNS team had already freed up the previous month.

What changes for developers

The changes are now available in the pingora-ketama crate, behind a cargo feature not yet officially announced: the v2 ring brings the compacted storage format, a faster sorting method, and the ability to scale the base number of hashes per node. Since Pingora is open source, any team that already uses the framework for load balancing (or is considering building its own proxy in Rust) can test the gain directly in their own project, without waiting for Cloudflare to make any grand announcement.

But the point that matters beyond the code is the method: Rust's memory alignment rules aren't a compiler detail, they're the difference between a struct with 6 real bytes and one with 8 wasted bytes, and that matters in any system that holds millions of these structures in memory, whether an edge proxy or a game. And the migration strategy (two versions coexisting, deterministic per-request decisions, a rollout with blast radius controlled by location) is a reusable pattern for any routing or partitioning algorithm swap in production, inside or outside the CDN context.