Data indexing with RGB (Ruby, Graphs and Bitmaps)
summary
A production in‑memory index called the measure store uses Ruby plus RedisGraph and redis roaring bitmaps to harmonize and scope survey measures by walking graph edges for context then fetching compressed bitmaps and combining them with bitwise ops. It replaces slow deserialize‑and‑concat SQL workflows: a 1.5M respondent query drops from 90 seconds to 495 ms (~180×), a 5M read returns in 9 ms, and 3.6 GB of serialized frames compresses to about 27 MB (~128×).
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:15Data Indexing with RGB
- 2:07The world before the MeasureStore
- 5:21Context
- 6:51Storage
- 10:00Introducing MeasureStore
- 11:12Cross market analysis
- 12:53Cross comparison
- 14:08Harmonisation
- 16:08The bitmap store
- 18:51The graph
- 23:06Architecture
- 23:59Performance
- 25:30Next steps
