Employing Modern C++ for High Performance Delta-Coding Compression
summary
Shows that delta encoding can cut network bytes and end-to-end latency for time-series (market) data by applying information theory, Huffman-style coding and columnar homogeneous streams rather than row blobs; demonstrates a concrete pitfall: Google protobuf varint spreads unary bits which increases decoder entropy (varint can be up to nine bytes), proposes encoding the extra-byte count up front or using a small binary field instead, and warns that 100x-200x penalty for surprising branch predictors and that the CMD shuffle_bytes dead end limits SIMD scaling, offering production-tested alternatives that outperform the shuffle approach.
AI-written summary · about
chapters (7)
- 0:00Delta coding compression intro
- 11:06Information theory for Delta coding
- 22:03Entropy and performance
- 39:10Protocol Buffers encoding analysis
- 47:02Efficient variable-length codes
- 57:47Delta coding for time series
- 1:03:09Stepanov’s lessons
related talks
05mMaximising Throughput & Minimising Latency in High-Performance C++ Code
01h 04mBeyond the Floating Point Computations - What’s New, What’s Useful?
01h 00mMoving Faster: Everyday efficiency in modern C++
44mC++ Performance and Optimisation
01h 31mDesigning a Slimmer Vector of C++ Variants
57mThis Videogame Programmer Used the STL and You Will Never Guess What Happened Next
01h 06mModern C and What We Can Learn From It
01h 24mOptimising a small real-world C++ application