Composable Data Management Systems
summary
Meta builds a reusable execution layer called Velox C++ execution library (open sourced in 2022) that executes fully optimized plans and integrates into engines like Presto (PRMO) and Spark via Gluten, and is already used for ingestion, real‑time pipelines and a time‑series DB. Production runs show about 3x performance and cost improvement; the project reports 200+ contributors and 20+ companies. They also open source a new Nimble file format for wide tables and argue hardware acceleration could add another ~10x.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:31Data infrastructure evolution
- 3:37Composable data management
- 8:14Vu execution engine
- 11:44Vu performance gains
- 14:12Future data stack innovations
- 16:40Hardware acceleration
related talks
35mIntroducing Prestissimo, Presto's Next-Gen Query Execution Engine
22mPresto for Your Fast, Open SQL Lakehouse
51mPresto Open Source SQL Engine for Massive Scale Data Lakehouses
14mReal-Time Data Processing for ML Feature Engineering
22mIntroduction to Presto: The SQL Engine for Data Platform Teams
24mData Mesh and Lakehouse
32mHow to Foster Cross-Community Collaboration – Lessons Learned
19mModern Data Stack Is Dead. Adopt Data Mesh – From Concept To Service