LogCrisp: Fast Aggregated Analysis on Large-scale Compressed Logs by Enabling...
summary
LogCrisp implements a two-phase pattern extraction that extracts a global Sketch offline and lightweight per-block Specs online, decoupling global description from local filtering and keeping local-filter effectiveness while retaining global aggregation. It adds an AVX SIMD algorithm that turns prefix/suffix full-text queries on numerically encoded units into high-performance range/point checks via vectorized right shifts and a squeezing-vs-compare-range table. Implemented system processes nearly 7TB (6TB eval), reports ~10× lower analysis latency, 3.8× higher ingestion, and nearly identical compression ratios.
AI-written summary · about
related talks
15mCLP: Efficient and Scalable Search on Compressed Text Logs
12mOn the Feasibility of Parser-based Log Compression in Large-Scale Cloud Systems
29mWide Event Analytics
30mData @Scale Seattle- Sundaram Narayanan
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
27mData @scale Seattle - Yucheng Low
23mDependence-Preserving Data Compaction for Scalable Forensic Analysis
24mCracking the Cost Code: A Cost Optimized Approach to Distributed...