Open Source Memory Speed Virtual Distributed Storage System
summary
Alluxio (formerly Tachyon) is an open‑source, memory‑first virtual distributed storage layer that sits between compute frameworks (Spark, Flink) and backend stores (S3, HDFS) and exposes a unified namespace with mountable stores. It manages RAM/SSD/HDD tiers with pluggable policies, ships as v1.0.1, and has ~250 contributors. Deployments: BYU runs it on 200+ nodes >2PB and reports 30× faster queries, Barclays avoids repeated ETL and cuts 30–60 minute reloads, and one China site cites up to 300× speedups enabling previously failing jobs.
AI-written summary · about
chapters (13)
- 0:00Introduction to Alluxio
- 1:13System overview and motivations
- 3:00Project history and growth
- 5:34Why memory speed storage
- 8:48Storage tiers and policies
- 11:37Virtual distributed storage benefits
- 15:56Use cases and integration
- 16:38Accelerating remote storage access
- 19:07Sharing data at memory speed
- 21:13Managing multiple storage systems
- 23:20Real-world deployments and results
- 28:20Summary and resources
- 29:22Q&A session
related talks
45mAlluxio Data Orchestration for Machine Learning
35mAccelerating Spark Workloads in a Mesos Environment with Alluxio
32mStreamlining Data Access for with
31mAccelerating Spark Workloads in an Apache Mesos Environment with Alluxio
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...
05mLightning Talk: How to Bring Data Locality to I/O-Intensive Workloads on Kubernetes
05m⚡ Lightning Talk: Using Decentralized File System to Optimize ML/AI Workloads on Kubernetes
28mOptimizing Latency-Sensitive Queries for Presto at Facebook