Show HN: misa77 - a codec that decodes 2x faster than LZ4 (at better ratios)
This article introduces misa77, a new LZ-based data compression codec designed for "write-once, read-many" scenarios. It boasts significantly faster decompression speeds (up to 2x LZ4) and better compression ratios, achieving this by trading off slower compression times. The article provides detailed benchmarks across various platforms and compares its performance against other popular codecs.
Why it matters
This new codec could offer substantial performance improvements for applications requiring frequent data decompression, such as databases, log processing, or content delivery networks, by reducing CPU load and improving data throughput and storage efficiency.
misa77 is an LZ-based codec that targets the write-once, read-many niche. In particular, it aims to satisfy the following criteria:
Slow compression is the obvious tradeoff that one makes to achieve the above.
In addition, misa77 has a somewhat synergizing tendency to decompress highly compressed files faster, leading to the following results:
This makes high-effort compression particularly attractive for misa77, and inspires some experimental compression modes (refer to src/experimental/ ) that aim to spend more effort at compression time to produce a compressed stream that is friendlier to the microarchitectures of most CPUs when decompressing said streams.
misa77 also has two compression effort levels as of v0.2.0:
The underlying stream format (used by the library functions) and the container format for .misa77 files (produced by the CLI) can be found in docs/ .
Detailed results are listed ahead, but here's a terse summary:
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in