Efficient array (and data) processing by leveraging structure
summary
Claims a compilation-based DSL called SDQL that represents relations, vectors and sparse tensors as semiring dictionaries, implements ~44 algebraic rewrite rules and uses equality saturation to search plans, and generates C++ and GPU code. Prototypes in Python/Scala/Julia compile hybrid SQL+linear-algebra pipelines, match or beat DuckDB on relational workloads, outperform Taco and NumPy/SciPy on sparse tensor kernels (orders-of-magnitude gaps in some cases), and is the first system to support sparse autodiff at scale.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 5:51Data science pipeline challenges
- 11:06Unifying database and linear algebra
- 14:21Semiring dictionary query language
- 26:34Loop optimizations in SDQL
- 37:57Sparse tensor algebra optimizations
- 45:04Equality saturation for optimization
- 49:20Automatic differentiation for sparse tensors
- 51:31Structured tensor algebra future
- 53:21Conclusion and key takeaways
related talks
32mCompilation of Dynamic Sparse Tensor Algebra
07mdbverse: Composable database libraries for larger-than-memory scientific analytics
33mQUARK: A PURELY-FUNCTIONAL SCALA DSL FOR DATA PROCESSING & ANALYTICS
30mPanel 01 - APIs for Feature Stores
42mComposable Parallel Processing in Apache Spark and Weld
23mSpark Datasets: why they aren't great and what could be done
32mVectorized Query Processing for CPUs using Apache Arrow
39mComplex Data Transformations Made Easy