No, Maybe and Close Enough! Probabilistic Data Structures with Python and Redis
summary
Shows exact tradeoffs and runnable numbers: uses Python libraries and Redis data types to compare a set vs HyperLogLog and a Bloom Filter; adds 100,000 IDs and reports HyperLogLog ≈99,500 while using ~12 KB vs the much larger set, and configures a Bloom Filter for 200,000 items (adding 100,000) via pyprobables and Redis BF RESERVE, demonstrating false positives (no vs maybe), memory savings, hashing details and how to tune accuracy vs size.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:52Counting Things...
- 4:16Problems at Scale
- 5:35Use a Database: How Many Sheep?
- 8:41Tradeoff..
- 10:08Hyperloglog: Approximating Distinct Items
- 12:19Hyperloglog: Algorithm
- 14:22Redis: Approximately How Many Sheep Have I Seen?
- 15:55Bloom Filter: Set Membership (No, Maybe)
- 23:20When to use Probabilistic Data Structures
related talks
26mNo, Maybe and Close Enough: Using Probabilistic Data Structures in Python
51mProbabilistic Data Structures with Redis Bloom
41mProbabilistic Data Structures
37mWhen Probably is Good Enough
28mBloom filters with redis
01h 00mCount-min Sketch to Infinity
52mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
01h 00mHyper Speed: When Big Data Blooms