Python Meets Heterogeneous Computing
summary
Claims a working Python SDK plus server called Covalent that orchestration-izes heterogeneous distributed runs, with AWS Batch and Azure Batch plugins, a desk executor, and language hooks for C/Fortran/Julia. It records execution metadata for reproducibility, offers real-time monitoring and native checkpointing, and treats workflow invocations as first-class. Examples show multi-cloud ML training, file transfers, and a UI that visualizes task graphs; notes about scale: Top500 aggregate compute ≈4× Frontier and quantum access costs can be tens of thousands per hour.
AI-written summary · about
chapters (11)
- 0:00Start
- 0:43Today's most important problems require high performance computing solutions.
- 1:57Battery simulation.
- 6:12Distributed computing is a necessity.
- 8:50Quantum is driving the need for more heterogeneity.
- 12:02Covalent is a workflow orchestration platform for heterogeneous computing.
- 16:58Core functionality and features.
- 20:03Where does Covalent fit in the stack?
- 22:26Concepts.
- 28:25Covalent supports a growing ecosystem of hardware and software.
- 29:20Covalent enables teams to work on high value, differentiated work.
related talks
01h 26mCovalent: A New Paradigm for High Compute Cloud Workloads
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
01h 01mParallel Python Analyzing Large Data Sets
01h 21mHPC in the cloud
17mR & Python playing nice, in Production
01h 30mDistributed Python with Ray-Hands on with the Ray 2.0 APIs for scaling Python Workloads
31m"Optimizing Scientific Python with C++, CUDA, and Serverless Compute"
20mGPU Cloud Deployment Without Leaving Your IDE