Lightning Talk: Model Checkpoint Compression With OpenZL
summary
OpenZL is an open source lossless compressor that uses tensor type information and a DAG of codecs to transform and entropy‑encode numeric data rather than treating it as bytes. For floats it separates exponent bits (then Huffman encodes) and leaves noisy bits uncompressed, yielding 33% for bfloat16, 21% for float32, 15% for float16. Deployed at Meta, it cuts checkpoint overhead by 15% and storage by 17%, shrinks embeddings by 30%, outperforms Zstandard on Nimble by 10%, and matches high‑compression ratios at far faster speeds.
AI-written summary · about
related talks
40mData compression with Go
PolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases
15mCLP: Efficient and Scalable Search on Compressed Text Logs
09mLogCrisp: Fast Aggregated Analysis on Large-scale Compressed Logs by Enabling...
22mLoopDelta: Embedding Locality-aware Opportunistic Delta Compression in Inline...
17mSparseZETA: Intelligent Auto-tuner for Designing High-Performance SpMV Programs
02mLightning Talk - Parquet: The Pillar of Modern Telemetry...
16mμSlope: High Compression and Fast Search on Semi-Structured Logs