Btune: Making Compression Better
summary
Btune is a dynamic plugin for Blosc2 that finds codec and parameter combinations by running a Btune genetic mode search and optional neural‑network inference, measuring actual tradeoffs like ~1.5 GB/s compress with ~4× ratio on a laptop, ~4–4.8× best ratios with Zstandard, chunked processing that becomes ~6× faster, and up to ~20–22 GB/s on a 16‑core server. It tries codecs (LZ4, blosclz, LZ4, Zstandard), filters like bitshuffle, varies threads/levels, and produces usable recommendations but requires retrials and machine‑specific tuning.
AI-written summary · about
chapters (10)
- 0:00Btune and compression tuning
- 1:31Genetic tuning tutorial setup
- 5:45Btune’s state machine workflow
- 13:16First compression benchmark
- 21:37Speed vs compression ratio tradeoffs
- 31:06Compression speed in performance mode
- 41:13Balancing speed and ratio
- 48:35Maximizing Zstandard ratio
- 52:34Neural network inference mode
- 1:06:23Q&A
related talks
32mBeyond Zstandard: The Future of Data Compression
32mOn compression of everything in Postgres
09mCompressing the Library of Babel!
24mHow GZIP compression works
18mWhy Lossless Data Compression Is Important and Fun
27mCompression in Lucene #bbuzz
12mWhy is your encoder so slow? The curse of next-gen standards
52mI Want to Compress Everything