Data indexing with RGB (Ruby, Graphs and Bitmaps)
summary
Custom index stores respondent-level data as Roaring bitmaps and context as a property graph in Redis so queries become graph traversal + bitwise ANDs rather than deserializing many SQL pickles. Implementation uses redis roaring and RedisGraph; fetching an age bitmap for 3.6M respondents returns in ~9 ms, scoped product query shows ~495 ms versus 90 s previously (~180x speedup), and 40 studies compress from 3.5 GB to ~208 MB (~128x compression). The system runs in memory, supports bidirectional and directional harmonization edges, and serves real-time filtering and cross-measure joins.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:06Project origins
- 2:28Survey data processing
- 4:56Data indexing challenges
- 7:45Understanding harmonization in data relationships
- 9:32Measure Store solution
- 10:58Measure Store demo
- 13:49Cross-measure analysis
- 14:09Regional data harmonization
- 15:56Bitmap storage
- 19:03Graph database
- 23:16Redis tools
- 24:21Performance benchmarks
- 25:49Future plans
related talks
26mData indexing with RGB (Ruby, Graphs and Bitmaps)
49mCounting things at the speed of light with roaring bitmaps
42mii hidden gems
01h 02mStoring Non-Scalar Data
01h 01mDungeons, Dragons, and Graph Databases
25mHigh on Cardinality, Low on Memory
38mBuilding a Memex (with Ruby!)
27mUsing Ruby to build a modern Memex!