Distributed Sparse Computing with Legate Sparse
summary
Legate Sparse transparently distributes unmodified SciPy sparse programs across CPU/GPU clusters by composing with the distributed NumPy library cuPyNumeric, using the DISTAL compiler to emit distributed sparse kernels and partitioning code. The system generates about 50% of the C++/CUDA in the library, reaches performance competitive with PETSc, scales to multi‑GPU systems (demonstrated on 192 GPUs), and ships as a product used by a payments company to run large transaction‑graph SPMV workloads.
AI-written summary · about
chapters (11)
- 0:00Legate Sparse intro
- 1:04Scaling sparse code
- 2:49Legate Sparse development
- 4:32Distributed sparse challenges
- 6:59Compiler distribution
- 9:29Sparse matrix structure
- 11:11Partitioning sparse arrays
- 13:16Tensor partitioning workflow
- 13:57Automated code generation
- 15:23Future JIT scheduling
- 16:06Conclusion
related talks
17mZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization
19mEfficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance...
20mLegion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale...
14mAccelerating PyTorch FSDP Via Overlapping Collectives With In...
49mDistributed Deep Learning
15mDorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and
14mP3: Distributed Deep Graph Learning at Scale
32mFrom Sparse Matrices to Graphs and Back Again