Hyper Speed! When Big Data Blooms
summary
Built a telemetry pipeline that scales from a few hundred reports to 65,000 JSON payloads per second at peak, with growth milestones from 1M/day to 1M/hour. Uses heavy front-end normalization, deduplication and a 8.5 GB Redis Bloom filter (k≈7) sized for 7 billion distinct items to short‑circuit “have we seen this before” checks, cut DB queries and alert on true new events; also deploys Count‑Min sketches, top‑K and HyperLogLog for frequency and cardinality.
AI-written summary · about
chapters (21)
- 0:00Intro
- 0:50XSS (Cross-Site Scripting)
- 4:12Content Security Policy
- 4:56Controlling JavaScript
- 5:48Control Everything
- 6:47Knowledge is Power
- 8:02Violation Reporting
- 9:23Scale Up
- 11:32Handling Volume
- 15:21Bloom Filters to the Rescue!
- 16:00Hash Functions
- 20:33Probabilistic Data Structure
- 21:04Setting up your Bloom Filter
- 21:53Calculating Values
- 22:50Calculating w
- 25:40Real World Examples
- 32:14Counting Bloom Filter
- 33:18Cuckoo Filter
- 33:37Count-Min Sketch
- 35:59HyperLogLog
- 38:46The Bloom Filter
related talks
01h 00mHyper Speed: When Big Data Blooms
49mHyper Speed! When Big Data Blooms
22mIt’s all about the db - a journey to accommodate 100k events per second
01h 01mDealing With Data Panel
38mHow to handle 100k events per second
19mBig Data
52m100% Big Data. 0% Hadoop. 0% Java.
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns