Thermocline Proposes Tiering as an Alternative to Sharding for a Memory-Strained MongoDB Replica

The Replica That No Longer Fits in Memory

Every MongoDB replica set starts small: the working set fits in the WiredTiger cache, reads come from memory, and nobody worries about disk. The problem shows up later, when orders, events, and logs accumulate over years and the collection grows to ten times the size of the cache.

From that point on, every read of a document that isn't cached becomes a disk read, and latency starts to follow the disk, not the query's index. That's the scenario described by Zelmar Michelini, of Percona, in a post published on November 2 on the company's blog: in a test cluster, a 70 ms read p95 wasn't a symptom of a bad index, but of a disk at its IOPS limit.

The standard response to this picture is sharding. It works, but it's costly: a partition key that's hard to change later, mongos routers, config servers, and queries that need to include the shard key to stay fast. Michelini starts from an observation common to databases that grow by accumulating time:

most reads are for recent data, and old data is in the way

Zelmar Michelini, author of the test, Percona

The question that motivated the experiment: can you keep the replica set that already exists, put recent data in a fast place in front of it, and do that without touching the application?

Thermocline: Tiering by Age, Not by Partition

Thermocline, the name given to the project, is a storage tiering layer for MongoDB. The cluster that already exists becomes the archive, exactly as it is. A new, smaller replica set in front of it holds the recent documents. The application stops talking to the old cluster and starts talking to a proxy.

Four components keep the two layers consistent:

• Proxy: speaks the MongoDB wire protocol; it sends the query to the fast layer when it can prove it has the full answer, and to the archive when it can't. Writes go to the fast layer.
• Feeder: reads the fast layer's change stream and mirrors every write to the archive, ensuring the archive always has all the documents.
• Evictor: removes documents older than the age configured for the collection from the fast layer, after confirming the archive already has them.
• Warmer: fills the fast layer from the archive with everything newer than that age.

Nothing is loaded manually: the fast layer starts empty and fills itself, both through the warmer and through the documents accessed by traffic itself.

The Test: Deliberately Limited Hardware

Michelini ran the test on Hetzner Cloud, with a cluster designed to reproduce the problem, not to favor the result:

• The existing cluster: a three-member replica set with 40 million simulated orders, 47 GB of data. Each member limited to 2.5 GB of memory (1 GB WiredTiger cache) and 3,000 reads per second on the data disk, a dataset about 47 times the size of the cache.
• The fast layer: a second three-member replica set, empty at the start.
• The load: YCSB 0.17 with 64 threads, ten-minute runs, two rounds per side. Each document carries a date spread across the last year, so that "recent" means something measurable.

Three load mixes were tested, each stressing a different read pattern.

The Numbers: Gains Grow With Temporal Concentration

In the concentrated mix, p99 dropped from 92 ms to 8 ms. In the online store mix, tested at the same fixed load of 6,165 operations per second (the maximum the old cluster alone could sustain), the cluster alone delivered 6,086 operations with a p95 of 70.5 ms; behind Thermocline, it delivered 6,156 and 6,159 in two rounds, with a p95 of 2.8 ms and 2.1 ms. The archive disk, in this test, did 1,838 reads per second instead of 2,704: less pressure on the backend too.

There's also the cold-start scenario, which is the realistic one for anyone adopting this: an empty fast layer, filled only by traffic and the warmer. In 30 minutes of the online store mix at full tilt, the pair averaged 8,221 operations per second, 42% more than the old cluster alone with a cold cache (5,803). It surpassed the old cluster's number after 100 seconds, averaging 6,417 operations in the first five minutes and 9,146 in the last five, copying 271,271 documents from the archive to the fast layer in that interval.

Where the Gain Doesn't Show Up

The gain comes from the reads the fast layer can answer on its own. Everything else still falls on the existing cluster, and that's what defines the method's limits:

1. Writes still reach the archive. Every write is mirrored to the existing cluster, and mirroring an update means the archive first needs to read that document from disk. In the online store mix, writes ran at the old cluster's speed, and update p99 got worse with Thermocline (455 to 472 ms versus 104 to 189 ms alone). Michelini says he's working on this tail now.
2. Reads of old data remain as slow as before. A query looking for something the fast layer doesn't hold will end up at the archive. If the application's reads are spread evenly across the entire history, the gain is small.
3. Both layers need to run the same MongoDB version. The archive answers the same queries as the application, so it needs to understand them; adopting this over an old cluster means upgrading it first.
4. It's a proof of concept. It has an extensive test suite, according to the author, but has never run in production. Pointing real traffic at Thermocline today isn't recommended by the project's own author.

What This Changes for Those Deciding Between Scaling Up and Partitioning

For anyone keeping a replica set that outgrew its cache, Thermocline isn't a production option ready to use tomorrow, but it is a conceptual third path between buying a bigger machine and taking on the operational cost of sharding. The central difference: the application doesn't change a single line of code, because it always speaks the MongoDB wire protocol, just now with a proxy in the middle.

This has concrete implications for those who operate the stack. Any decision to adopt this pattern, whether Thermocline or an internally built equivalent, means monitoring two replicas instead of one, validating that the feeder doesn't accumulate lag in the change stream under write spikes, and designing the cutoff age per collection according to the actual access pattern, not an arbitrary value.

It's worth looking at the project with the right yardstick: it's open-source code under the Apache 2.0 license, available on GitHub at github.com/zelmario/thermocline, with a README explaining how to run it. Michelini explicitly asks for reports of workloads where the approach fails, which is a rare invitation: testing your own numbers before trusting the author's is exactly what the post's conclusion recommends.