Homemade scientific clusters in Python
summary
Python orchestrates a working scientific cluster running on MAAS + KVM with orchestration via juju charms and reactive decorators, deploying master, login, storage and multiple compute nodes from a single codebase. The stack uses LXD or VMs, runs jobs under Grid Engine scheduler, and executes reproducible workloads in Singularity containers. The setup clones onto a few recycled desktops, deploys in minutes, runs CFD benchmarks and emits VTK result files.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 6:47Pay the bill
- 7:28Lowcost prototyping infrastructure
- 7:40Containerbased solution
- 7:52KVMbased solution
- 13:39Realworld example
- 14:34Trump tool
- 16:18Reactive pattern
- 20:14Summary
- 21:20Infrastructure
- 22:04Cluster
- 23:32Demo
- 32:47Questions
related talks
35mCluster as Code
32mDemocratizing Distributed Computing with Dask and JupyterHub
49mPython for High Performance and Scientific Computing
26mTerraforming Jupyter and Dask: How to Get Your Own Cloud Data Science Platform on the Cheap
36mBuilding a Scalable Data Science & Machine Learning cloud using Kubernetes
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
59mCloud Native Home Lab
16mHow We Built a Cloud On K3s: The Learnings Of Growing Up Fast