Adaptive Scrooge - Adaptive Thrift Decoding
summary
AdaptiveScrooge samples a few events to learn which Thrift fields are actually accessed and rewrites decoding and the in-memory representation to skip unneeded bytes, avoiding object creation and GC. Implementation uses a wrapper counter during a learning window and ASM bytecode manipulation to generate fast decoders that keep the original byte array for cheap serialization. Benchmarks show up to 20x faster than regular Scrooge and 10x faster than Lazy Scrooge in the hot case, with real-job gains around 10% CPU and a 25% spout CPU drop in a Heron topology.
AI-written summary · about
chapters (11)
- 0:00Introduction and context
- 3:10Problem with deserializing full payloads
- 6:41Why columnar formats don't always work
- 7:23Real-world use case example
- 9:38How adaptive Scrooge solves it
- 13:12How adaptive Scrooge works internally
- 16:42Bytecode manipulation with ASM
- 17:28Fallback and tuning parameters
- 20:14Trade-offs and limitations
- 21:38Usage and performance results
- 23:02Conclusion and comparison
related talks
35mScotty: Efficient Window Aggregation for your Stream Processing System
24mScio data processing nirvana at Spotify
20mHadoop Gets Groovy
18mPreparation Meets Opportunity: Enhancing Data Preprocessing for ML Training With Seneca
29mSliding Look-Back Window Assisted Data Chunk Rewriting for Improving Deduplication...
09mLogCrisp: Fast Aggregated Analysis on Large-scale Compressed Logs by Enabling...
01h 06mBest practices when accessing Big Data or any other data!
36mScalably crashing JVMs, or why binary data to content is hard