No, Maybe and Close Enough: Using Probabilistic Data Structures in Python
summary
Shows concrete trade-offs: use Hyperloglog to approximate distinct counts (100000 items produce ~99,565 with about 12k of memory vs a set at ~100000 and ~4.6MB), and use Bloom filter for membership queries that return definite negatives or probabilistic positives with tunable false‑positive rates. Examples use Python libraries and redis modules (pfadd/pfcount, BF.RESERVE), include a GitHub repo (https://github.com/simonprickett/python-probabilistic-data-structures), and demonstrates when accuracy versus memory wins.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 0:23The Problem
- 8:56Probabilistic Data Structures
- 9:39Hyperlog Log
- 11:50Hyperlog Log Algorithm
- 12:25Hyperlog Log Example
- 15:26Bloom Filter
- 20:33Python Code
- 22:51When to Use
related talks
26mNo, Maybe and Close Enough! Probabilistic Data Structures with Python and Redis
41mProbabilistic Data Structures
53mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
59mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
55mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
45mMost of the Time It Works Every Time: The Mindset Behind Using Probabilistic Data Structures
25mHigh on Cardinality, Low on Memory
51mProbabilistic Data Structures with Redis Bloom