Understanding Probabilistic Data Structures with 112,092 UFO Sightings
summary
Uses the 112,092-line UFO sightings CSV to demonstrate concrete implementations: Bloom filter for membership with tunable false‑positive rate and BF.RESERVE/BF.ADD, MinHash for Jaccard similarity on trigram shingles to detect near‑duplicate reports, and Top‑K (HeavyKeeper) to track trending words with erosive decay (decay ~1.05) and TOPK.RESERVE/TOPK.ADD. Demo pipeline uses Python (pandas, NLTK) to normalize text and Redis/Redis Stack to store and query approximate counts and top items in real time.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:28Why 112092 UFO sightings matter
- 7:02Probabilistic data structures
- 15:27Accuracy trade-offs at scale
- 22:29Bloom filters deep dive
- 31:59MinHash for set similarity
- 38:17HeavyKeeper for trending items
- 47:52Live demo with UFO data
- 52:30Resources
- 53:53Q&A
related talks
53mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
59mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
52mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
26mNo, Maybe and Close Enough: Using Probabilistic Data Structures in Python
26mNo, Maybe and Close Enough! Probabilistic Data Structures with Python and Redis
01h 00mHyper Speed: When Big Data Blooms
37mWhen Probably is Good Enough
51mProbabilistic Data Structures with Redis Bloom