Vortex: LLVM for File Formats
summary
Vortex is an open source, Rust-based columnar file format built on Arrow with plug‑in encodings (FastLanes, ALP, FSST, PCODC, optional Zstd), extensible layouts that defer IO and arrays that defer compute, and a single optimized scan operator that pushes filters and expression evaluation down into file reads. Benchmarks claim 137x random‑access speedup versus Parquet, ~18x faster full‑scan decompression and ~5x faster writes, with mixed size results (geomean ~1.16× Parquet). Integrations: DataFusion, DuckDB, Spark, Python/Java/C++ bindings; GPU kernels via JIT (NVRTC) for streaming to device.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 1:03Talk overview
- 2:29Hardware trends in data systems
- 4:17Data systems and file formats evolution
- 6:22Limitations of Parquet
- 7:49Why Spiral created Vortex
- 9:15Vortex research and progress
- 10:40Vortex goals and Linux Foundation adoption
- 11:44Future-proofing design principles
- 13:51Vortex core concepts
- 15:15Logical types and arrays
- 16:39Layouts and deferred IO
- 18:43Extensibility in Vortex
- 19:27Encodings and compression
- 23:45Layout strategies and writing
- 24:47File structure and metadata
- 27:17Zone maps and data pages
- 29:05Reading and writing Vortex files
- 30:27Scan operator and query engine
- 31:52Push-down operations
- 33:59Compute kernels CPU/GPU
- 36:51GPU optimization and streaming
- 37:36Vortex in the wild
- 40:45Benchmarking methodology
- 44:01Vortex advantages
- 44:42Future vision and collaboration
- 45:45Q&A
related talks
30mIntroducing Spark Cyclone Accelerating Spark with...
31mWeld: An Optimizing Runtime for High Performance Data
57mHigh-Performance Logic Programming on Servers, Clusters, and GPUs
06mLightning Talk: A Quick Intro to Falcon
16mAccelerate HPC and AI Applications with RVV Auto-Vectorization
14mCompressed and Parallelized Structured Tensor Algebra
20mFlux: Liquid Types for Rust
25mFormat Abstraction for Sparse Tensor Algebra Compilers