Advancing HPC The Hybrid Scheduler Approach at the D.E.Shaw Group
summary
A hybrid scheduler bridges an on‑prem orchestration platform (maestro) and multiple clouds using the same CLI and Python APIs, letting workloads burst into the cloud to access thousands of VMs and GPUs while preserving on‑prem semantics. Security uses mTLS plus self‑managed keys and OCI encrypted containers for single‑use secret code. A fork of Dask runs across both domains. It works for large-scale experimentation and bursting, while data sync performance and cloud cost remain ongoing challenges.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:06Maestro orchestration platform
- 3:57Limitations of on-prem clusters
- 5:01Hybrid scheduler approach
- 7:28Principles for cloud integration
- 9:17Hybrid workload execution
- 10:42Information security compliance
- 13:11Application integration challenges
- 15:18On-prem cloud-like environment
- 16:41Avoiding vendor lock-in
- 18:06Cost and skill gap analysis
- 19:10Long-term challenges and conclusion
related talks
41mFrom Hardware to Advantage: Hybrid Infrastructure as Enabler for Industry Adoption
01h 03mScaling Web Operations with Distributed Cluster Schedulers
21mKeynote: CERN Experiences - Ricardo Rocha & Clenimar Filemon
37mUsing Elixir for scheduling cloudnative DAG workflows
16mPreview: Cloud Computing
31mAn Adventure in Multi Cloud with Kubernetes and Distributed Data
43mSystems @Scale Summer Q&A Week 2
27mKeynote Panel: Future of Cluster Management