Building domain specific databases using Python for prototypes during take off
summary
Prototype implements a Sliding HyperLogLog unique counter using Kafka & Python as a durable stream and in-memory prototype, then ports to RocksDB and C++ with trivial effort. Initial Redis approach stores buckets three times and costs hundreds of gigabytes in RAM; the production stack drops ~500 GB in memory to ~30 GB on disk. Prototype code is small (about 125 lines of algo-specific code), holds up for six months in production, and supports sharding, replication and quick recovery via Kafka offsets.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:03Trust and safety platform
- 3:53Why build our own DB
- 5:18Initial Redis solution
- 9:11Moving to RocksDB
- 10:56Kafka for data streaming
- 12:42Fault tolerance with Kubernetes
- 14:49Final architecture savings
- 15:34Extending for labels and search
- 16:16Accurate counters
- 16:57Docker and Kubernetes issues
- 18:23Future derived data work
- 24:00Team workflow
- 25:49Lessons from mistakes
- 27:18Data backup strategies
- 28:52Rule-based vs ML anti-fraud
- 31:58Data collection tactics
- 33:02Q&A
related talks
30mBuilding domain specific databases with a distributed commit log
22mSolving Everyday Data Problems with FoundationDB
33mThe rise and fall of the full stack database
24mBuild-a-Database with Python
14mYour favourite second database
38mDubious Database Design
29mBuilding a biological database with Python
26mNo, Maybe and Close Enough! Probabilistic Data Structures with Python and Redis