Compression in Lucene #bbuzz
summary
Explains concrete encodings added and tuned for Lucene 5: a prefix FST maps terms to block offsets, postings switch from VInt vs packed ints (delta + bit-packing per block) to fixed-bit packing for sub-byte average, and norms gain seven encodings including constant, delta, table-compressed, indirect and patched to cut space. DocValues use GCD and monotonic models plus delta/table/prefix tricks for timestamps, offsets and ordinals. Stored fields use LZ4 by default and an optional deflate (LZ77 + Huffman) pass for higher compression.
AI-written summary · about
chapters (10)
- 0:00Introduction to compression
- 1:50Query processing in Lucene
- 3:58VInt compression for integers
- 7:26Delta and packed postings
- 11:17Scoring and Norms in Lucene
- 13:52Norms encoding techniques
- 19:38Doc values and their formats
- 23:31Returning stored fields
- 25:43LZ4 and Deflate compression
- 26:25Q&A
related talks
32mOn compression of everything in Postgres
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
34mIn core compression-How to shrink your database
52mI Want to Compress Everything
32mBeyond Zstandard: The Future of Data Compression
31mUnspoken overhead of the Inverted Index
24mHow GZIP compression works
18mWhy Lossless Data Compression Is Important and Fun