Project Caerus: Compute Storage Coordination
summary
Project Caerus coordinates disaggregated compute and storage (Spark, TensorFlow, S3, Ceph) to squeeze real-world speedups by sharing semantics across layers. It builds a multi-layer cache colocated with compute, a Carries controller that extracts job DAGs (Spark, "Flink"/pink) to prefetch/eject, a mutable-volume layer over immutable backends with NVMe write‑back, and prototype near-data processing and multimodal caching; deployed on the OpenInfra Lab with >10,000 users and shows substantial acceleration in practice.
AI-written summary · about
chapters (13)
- 0:00Intro
- 4:00Welcome
- 7:08Open Infrastructure Labs
- 9:17Why Caerus
- 13:31Caerus Projects
- 17:04SemanticsAware CDN Caching
- 20:11Open Discussion
- 21:08Breaking Boundaries
- 23:53Pre standardization
- 24:39Early involvement
- 25:58NDP
- 28:01Open Source
- 29:15Experiments
related talks
36mAn almost perfect match: cloud native and scientific computing
36mOptimizing Ceph Object Storage for Analytics and AL/ML Workloads - Overview and Future Plans
19mKeynote: Building Bridges: Cloud Native and High Performance Computing
22mResearch SuperCluster(RSC): Accelerating AI Research at Scale
14mArchitecting Cloud Storage for AI Native Applications
26mUnlocking Scalable Distributed Training With Arrow Data Cache on Kubernetes
15mA Community Cache with Complete Information
27mFrom Supercomputing to Serving: A Case Study Delivering Cloud Native Foundation Models