Open-Sourcing Venice
summary
Venice is a derived data store built for very large, frequently rewritten datasets and low-latency AI inference. It handles ~1500 datasets, ingests about 14 GB/s average (peaking near 50 GB/s), and serves millions of key lookups and vector ops per second. Ingestion combines Hadoop full pushes, Kafka-backed buffers and Samza streams, supports partial updates and DaVinci eager caching, and runs multi-datacenter active-active CRDT replication.
AI-written summary · about
chapters (14)
- 0:00Intro
- 1:34Derived Data Store
- 7:05Hybrid Workloads
- 8:57Stream Processing
- 9:36Streaming Writes
- 13:06Partial Updates
- 14:34Correctness
- 19:36Scale
- 21:57Single Get Use Case
- 25:59Read Compute Use Case
- 28:20Eager Cache
- 31:42Da Vinci Use Cases
- 38:54Scalability
- 40:36What is Venice for?
related talks
25mData modeling, the secret sauce of building & managing a large scale data warehouse
31mBuilding large scale, transactional data lakes with Apache Hudi
23mThe Open Source Analytic Universe, Version 2022
50mApache Hudi : The Data Lake Platform
31mBuilding a Data Infrastructure for AI/ML
45mA gentle introduction to Vector Databases
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
30mMilvus Vector Database: Unlocking the Future of Open Source Vector Databases