Swapping mmap for io_uring made the Rust query engine 60% slower

There's an implicit consensus circulating in high-performance engineering forums: mmap is convenient but scales poorly, and io_uring is the future of I/O on Linux. Conviva, a company that analyzes trillions of events a day to diagnose user experience, decided to make the swap in its Rust-based query engine. The result, as told by engineer Evan Chan, was the opposite of what was promised: the io_uring version ended up about 60% slower than the mmap baseline.

The account is useful precisely because it contradicts the happy narrative that dominates the blogs. For those building data systems in Brazil, where more and more teams now run analytics engines on local NVMe, it's a reminder that swapping one I/O primitive for another without redesigning the surrounding architecture tends to backfire.

What was running before

Conviva's engine is built on DataFusion, Arrow, Rust, Rayon, and Tokio. Events are transformed, encoded in a proprietary, mostly numeric format, and stored in the cloud. For querying, files are copied to local NVMe and read as large Arrow IPC files of 3 to 5 GB.

The choice of Arrow IPC made sense: the in-memory and on-disk layouts are identical, so the decode cost is minimal, and mmap delivers zero-copy reads natively supported by arrow-rust. A typical query touches 6 columns across 8 batch files (one batch per file), about 1.6 GB per batch, roughly 13 GB of data per day.

Under light load, it worked well, responding in seconds. The problem showed up under heavy concurrency.

The symptom in production

Some increase in latency under load is expected, more queries competing for the same CPU. But p95 and p99 spiked far beyond what linear growth would predict. The symptoms, according to the account:

• The OS page cache shrank: each pod consumed more memory as private allocation and less as shared cache.
• A huge number of page faults.
• p95 jumped from about 30s to more than 150s under real concurrency.
• Adding pods made things worse, not better.

The diagnosis pointed to mmap page cache thrashing under memory pressure. A controlled test made this explicit: 1 pod beat 4 pods on the same machine, being 41% faster at peak and more than 20% faster at p95 for 14-day queries. perf record showed 100% kernel-level lock contention. The four pods weren't fighting over CPU, they were fighting over the page cache, which is implicit shared state: one cache, one lock hierarchy, one eviction policy for all processes on the host.

The pidstat numbers during a stress run illustrate the storm of faults:

23:27:09   1,255,709 minor faults/sec
23:27:41   2,124,327 minor faults/sec
23:27:46   2,354,383 minor faults/sec

At more than 2 million minor faults per second, each touching a cache line via atomics, L1/L2 gets destroyed, which is lethal for an application that depends on large lookup tables resident in cache. Context switches went from 14,000/s in a run with a warm cache to more than 2 million/s, 150x more.

What io_uring promised to solve

The hardware ceiling measured the size of the prize. An fio run with the io_uring engine, 4 processes, iodepth 32, 4 MiB blocks, and O_DIRECT hit 20.2 GiB/s with the 32 NVMe drives at 99.75% utilization. mmap, at peak, delivered 3.44 GB/s, about 16% of what the hardware could do.

The plan was classic: bypass the page cache with O_DIRECT, submit reads via io_uring, coordinate with Tokio, and decode Arrow inline. The implementation used compio, a Rust-native io_uring wrapper. The first version fired one future per column, with all 40 columns (8 batches × 5 columns) submitted concurrently.

The reality check on Linux

The initial test on macOS (without real io_uring) already wasn't encouraging, but it was on Linux that the bill came due:

| Metric | io_uring, cold | mmap, cold | |---|---|---| | Total query time | 21.8 s | 13.6 s | | Materialize time | 17.4 s | 0 (mmap reads are "free") | | Major faults | 3,647 | 128,957 | | Minor faults | 8.6 million | ~1 million |

io_uring solved exactly what it was supposed to: major faults dropped 35x, from 128,957 to 3,647. The kernel stopped thrashing on physical page-ins. But minor faults rose 8x, and total time went from 13.6s to 21.8s. One class of fault was traded for another, and the deal came out at a loss.

Where the minor faults were hiding

After turning on O_DIRECT (runtime dropped from 21.8s to ~19s, a real but modest gain), perf pointed to Arrow. Samples concentrated on Buffer::from_slice_ref, the standard way arrow-rs builds a Buffer from a byte slice: it allocates new memory and does a memcpy.

Every 4 KiB destination page that memcpy touches requires the kernel to zero and map it, one minor fault per page. The 8 million minor faults over ~13 GB of reads match that math almost exactly. In other words, the Arrow layer was forcing the kernel to redo the memory-management work the team thought it had bypassed by adopting io_uring. Building the Buffer directly from the bytes io_uring already owned brought the runtime down to ~16s, still worse than mmap.

The community showed no mercy (and pointed to the technical issue)

The Hacker News thread reacted on two fronts. The first, about the format: several readers accused the text of being LLM-generated. As muragekibicho summed it up: "Seeing a section titles "The Production Symptom" is a Claudism in itself."

The second front was more interesting for builders. samus pointed out that the title might have been aiming at the wrong primitive:

"It would have been more precise to say O_DIRECT instead of io_uring. The point of io_uring is to avoid syscall overhead. What they were after was actually managing a page cache on their own, and it turned out to be more complicated than they thought."

>

-- samus

And jandrewrogers went to the architectural core: "mmap and io_uring require fundamentally different software architectures in a performance context. You shouldn't swap them out." According to him, io_uring with O_DIRECT exists so you can design your own scheduler in user space; if you delegate scheduling to a runtime, you lose most of the advantage, and performance can even get worse. That's roughly what happened.

It's worth noting the caveat from laserbeam: there's a Part 2, linked at the end of the article, in which the team makes io_uring twice as fast as mmap. In other words, the title is the hook for a series, not the final conclusion.

What it means for builders in Brazil

The takeaway isn't "io_uring is bad." It's that the I/O primitive isn't a plug-and-play piece. Swapping mmap for io_uring while keeping the rest of the architecture unchanged (a single materialization thread doing five jobs, Arrow decode with a default copy, prefetch firing 40 SQEs at once) shifts the bottleneck instead of eliminating it.

For Brazilian teams running DataFusion, Arrow, or their own analytics engines on local NVMe, three concrete lessons emerge from the account: the real cost often isn't in the disk (NVMe I/O accounted for only 6.9% of the measured off-CPU time), the host's shared page cache becomes contested global state once you stack pods, and arrow-rs's default copy can reintroduce exactly the faults you tried to avoid. The rest of the story (the moment they realized that 40 concurrent SQEs were the problem) is left for Part 2, which Chan promises to detail at P99 CONF, online on October 21 and 22, 2026.