We Replaced MMAP with Io_uring in Our Rust Query Engine. It Got Slower
Engineers at Conviva detail their experience attempting to optimize a Rust-based query engine by replacing mmap with io_uring. They discovered that the change resulted in performance degradation due to unexpected page-cache thrashing under high concurrency.
Why it matters
This provides valuable technical insight into the limitations of low-level I/O optimizations in high-performance data processing systems.
In the beginning, there was mmap. It was convenient: it let us lazily read huge numbers of Arrow IPC files from disk without managing memory ourselves. It fit our file format perfectly — Arrow IPC’s layout is designed for zero-copy random access, and mmap gives you exactly that.
Then we deployed to production, ran real concurrent query loads, and mmap became a real problem.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in