Beyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
summary
Claims a new open columnar format and table layer called Lance that makes AI workloads like vector search, large multimodal training, and EDA far faster and cheaper by design. Benchmarks show 1–2 orders of magnitude faster than Parquet/ORC for random access; a demo reads single rows ~2,000× faster and searches a billion vectors on a laptop in milliseconds. Key techniques are separate offsets and data, late materialization, disk‑based secondary indices and time travel; integrates via Apache Arrow with DuckDB, pandas and object storage.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:32Parquet/ORC limits for AI
- 3:56Random access in AI
- 5:26Lance vs Parquet/ORC benchmarks
- 10:15Demo: Lance vs Parquet speeds
- 13:27Lance real-world use cases
- 14:54How Lance improves performance
- 18:14Versioning in Lance
- 19:38Lance CB vector DB
- 21:02Demo: Lance CB semantic search
- 23:32Scaling billion-vector datasets
- 24:35Future developments and ecosystem integrations
- 25:40Q&A
related talks
05mBringing AI to DuckDB with Lance columnar format for multi-modal AI
28mLanceDB: lightweight billion-scale vector search for multimodal AI
31mAgents Need More than Vector Search
31mTransforming Multimodal Data Management with LanceDB-Ray
16mThe Post-Modern Data Stack for AI: Open Source Foundations for a Multimodal La...
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
42mColumnar Storage: Redefining Data Management for the Modern Era
27mEmpowering PySpark with Lance Format for Multi-Modal AI Data Pipelines