Fluid - Build Data Orchestration in Kubernetes
Talkat KubeCon + CloudNativeCon North America 2021 · on Thursday, 2021-10-14· at 15:25· Room 502 AB + Online·
summary
Fluid is an open source Kubernetes data-orchestration layer that exposes DataSet and Runtime CRDs, runs cache runtimes (Alluxio/JuiceFS-style) and adds a cache co-locality plugin plus prefetch hooks to the scheduler. Concrete claim: using Fluid’s distributed cache on object storage raises image-loading from ~638 to ~2,800 images/sec and cuts a training job from 1 hour to 24 minutes (4.5×) without hurting accuracy (92.26% → 92.5%). v0.6.0 is released and Alibaba Cloud and Tencent Cloud integrate Fluid in production.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:09Agenda
- 2:35Journey of A Data-driven Company
- 5:10Technical Challenges Blocking Faster Adoption
- 6:12st Challenge: Complicated Data Storage
- 7:39nd Challenge: 1/0 Bottleneck
- 9:02rd Challenge: Locality-ignoring Scheduling
- 9:49What is Fluid, indeed
- 11:44Fluid Architecture
- 14:30A Quick Start on Fluid: (1) Create a "Dataset"
- 17:18A Quick Start on Fluid: (2) Run Apps with Data Affinity
- 17:59Quick Start on Fluid: Auto-scale Dataset
- 18:43Demo
- 22:46Roadmap
- 24:28Plan
- 27:01Community & Join Us
related talks
35mFluid: Kubernetes Native Distributed Dataset Orchestrator and Accelerator
19mFluid: towards transparent, self-explanatory research outputs
26mPains and Gains: Lessons From Orchestrating Volumes for Remote Storage in Cloud Native AI Scenarios
37mComputational Fluid Dynamics (CFD) Analysis With Kubernetes, Kubeflow, And OpenFOAM
26mSpeed up Your ML Workloads With Kubernetes Powered In-memory Data...
34mOn Data Locality in Kubernetes
15mLightning Talk: Fluence: Approaching a Converged Computing Environm...
30mCloudflow: Spark, Flink, and Akka Working together on Kubernetes