Data compression with Go
summary
Shows concrete Go implementations and numbers: explains LZ77 with double-hash tables and a repeat-offset trick, outlines Asymmetric Numeral Systems and Huffman basics, and benchmarks compressors on real JSON/binary inputs. Reports gzip-style compression hitting ~200 MB/s compression, practical throughputs of 700–800 MB/s and peaks near 3 GB/s for decompression, and compares zstd, xz and legacy DEFLATE tradeoffs. Gives reproducible benchmarking tips (no allocations, real data sets) and warns that ~40‑byte records usually aren’t worth per-item compression.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:06Why compress data
- 2:10Compression algorithms basics
- 3:13How LZ77 works
- 4:19Hashing in compression
- 5:43Hash tables and role
- 7:14Double hash tables
- 8:40LZ77 repeat offsites
- 10:08Hash table optimization
- 11:11Entropy encoding
- 13:21Compression limits
- 15:41Huffman coding
- 17:07Numeral systems for compression
- 18:37Dictionary sharing
- 19:22Context mixing
- 20:07Benchmarking algorithms
- 21:53Data compression examples
- 24:38Real compression formats
- 26:54Format comparison
- 29:57Zstandard advantages
- 32:13Q&A
related talks
09mLightning Talk: Model Checkpoint Compression With OpenZL
PolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases
15mCLP: Efficient and Scalable Search on Compressed Text Logs
37mApproaches to online quantile estimation
36mIntergalactic dataspeak
44mAtomic Habits in writing high performing distributed code in the cloud
36mAn Exploration in Storing Telemetry in Cloud Object Storage
25mPayload formats for LPWAN Applications