Apache DataSketches
summary
An open source library in Java C++ and Python that implements streaming theta sketch and tuple sketches, KLL quantile and REQ quantile, CPC and HyperLogLog variants, frequent‑items, reservoir and VAROPT sampling, and a frequent‑direction SVD sketch, all designed to merge without loss and give provable error bounds. Claims include Yahoo reducing batch jobs from days to hours and enabling subsecond queries on real‑time streams. Recent release adds a memory module (2.0), Postgres and Hive integrations, and planned JDK 17, JPMS and tighter C++ interop.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:45The DataSketches Team
- 3:39A Sketch is a Streaming Algorithm that Solves a Specio Problem
- 8:38Counting Unique Things
- 11:11Typical Error: Theta Sketch
- 14:21Quantiles Sketch, PMF Example: Response Times
- 15:31KLL vs REQ Sketch, Error Properties
- 17:31Finding Frequent Things (Heavy Hitters)
- 18:48Frequent Items Error: Artistic Rendering
- 21:14Sampling
- 24:04Future Proofing the DataSketches Java Library
- 26:24Bright Future for Sketching Technology & Solutions..
- 27:20THANK YOU!
related talks
24mSketching Algorithms: Making Sense of Big Data in a Single Stroke
31mAnswering Multi-dimensional Queries on Massive Datasets
42mMergeable Summaries and the Data Sketches Library
01h 50mData sketching workshop
47mData Sketches — How to Eat an Elephant Whole (Even If It's Infinite)
01h 24mStreaming, Sketching and Sufficient Statistics I
28mBest of Stream + Batch: When Streaming Analytics Isn’t
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding