Accelerated Out of Core Shuffling
This article explains "out-of-core shuffling," a technique used in data analytics to handle large datasets that exceed available memory, preventing "out-of-memory" errors. It details how RapidsMPF, a reusable shuffler, accelerates this process to 1.8 TiB/s, making crucial operations like joins and groupbys more efficient.
Why it matters
Efficient data shuffling is fundamental to modern data analytics, especially for large-scale distributed systems, as it directly impacts the performance and feasibility of complex data operations. This technology helps overcome memory limitations, enabling faster and more cost-effective processing of massive datasets.
Shuffling data at 1.8 TiB/s! RapidsMPF is a reusable, out-of-core shuffler that turns shuffling OOM headaches into a spill you can budget for.
Shuffling is the crux of structured data analytics distributed or otherwise. It's a core component of key data operations like: join, groupby, merge, sort, etc. A full distributed shuffle can move all the data from every process to every other process, an all-to-all. This is very costly and many sophisticated techniques have been developed to avoid this operation as much as possible. Shuffles aren't particularly computationally challenging: calculating the hashes to route data is fairly cheap. They are nonetheless expensive in a workflow, for a variety of reasons:
Because shuffling is hard, slow, memory-intensive, and critical, it's historically where RapidsMPF started.
A quick primer on joining tables. If we have two tables: partsupp and lineitem and we want to join them, what happens?
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in