Enabling Machine Learning through Real-Time Data Processing
summary
Rockset is a real‑time indexing database built on RocksDB that claims millisecond SQL queries over JSON/Parquet without schematization by converting incoming formats to a protobuf stream and then into RocksDB. Key features: Smart Schema auto-detects types across mixed records; a converged index (inverted, column, row) chooses indexes to cut CPU from ~15 seconds (scan) to hundreds of milliseconds; column clusters store ~3,000–6,000 values per key for mutable, low-latency updates. Also: Aggregator‑Leaf‑Tailer disaggregation, SIMD vectorized execution, and JavaScript UDFs for kNN/Faiss model serving.
AI-written summary · about
chapters (12)
- 0:00Real-time ML with Rockset
- 1:09Requirements for real-time machine learning
- 2:35Value of real-time data
- 4:20ML data exploration challenges
- 5:45Smart schemas for multi-dimensional data
- 8:53Dynamic type system for semi-structured data
- 10:16SIMD vectorized execution
- 12:01Supporting ML data sources
- 13:44Fast model serving with indexing
- 16:50Indexing vs scanning in ML
- 17:54Column-based clustering
- 19:19Key takeaways for ML
related talks
31mBe a Productive Dev with Real time Analytics as an API
19mGetting from Schemaless Ingest to Fast SQL
29mAccelerating Performance: Mastering RocksDB for High Speed Data Processing
57mRocksDB
37mMoving on from RocksDB to something FASTER
39mHow we isolate streaming ingest from search using RocksDB
43mRealtime Analytics with StarRocks in Wellness Booking
52mRocksDB Performance and Reliability Practics