Solving Everyday Data Problems with FoundationDB
summary
Prototype replicates Aurora binlog into FoundationDB, stores compressed chunked bitmaps (2^18 chunks) keyed by rule, and uses the Roaring bitmap library plus parallel range reads and fork/join to AND/OR sets. Writes use version stamp keys to avoid manual counters. Result: queries that took minutes run under 100 ms for large customers; smaller ones are much faster. Implementation is ~3,000 lines of Node using FoundationDB bindings, high-availability via autoscaling load balancer, prototype not yet in production.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:39About Me • Independent software engineer
- 1:31About This Talk
- 2:32Status Quo
- 2:57Everyday Data Problems?
- 3:28Microservices
- 4:09Why is this a problem?
- 7:36Why is FoundationDB a solution?
- 9:11The Everyday Data Problem
- 10:12Breaking it down
- 11:41Bitmap Indexes 101
- 12:56New Possibilities
- 14:13How to get there. Step Two
- 16:38How to get there - Step Three
- 17:22Experimental Results
- 18:24YOUR Everyday Data Problems
related talks
15mTime Series and FoundationDB: Millions of Writes/s and 10x Compression in 2000 Lines of Go
46mFoundationDB: The Best-Kept Secret of New Distributed Architectures!
34mFoundationDB, the Black Knight
25mBuild Anything with FoundationDB
16mFoundationDB and Node.js: Year in Production
24mHow We Saved 5x Migrating from Aurora to FoundationDB
18mFoundationDB Record Layer: Open Source Structured Storage on FoundationDB
50mBuilding the Next-Generation Backend on FoundationDB