NumS: Scalable Array Programming for the Cloud
summary
NumS compiles NumPy-like code to a Ray-backed runtime that schedules block-wise operations using futures and graph-arrays, trading element-wise overhead for coarse-grained block kernels and a placement optimizer called CRTS scheduler that uses a block-cyclic layout and an objective minimizing max memory and network load. Benchmarks report 10–20x speedup over Dask on linear algebra, 3–6x vs Dask and Spark for logistic regression, weak-scaling to scales to 1 TB, and optional GPU support via cuPy + NCCL; installable with pip.
AI-written summary · about
chapters (12)
- 0:00Introduction to NumS
- 2:31NumS architecture overview
- 3:14Solving dependency and parallelism
- 4:19Block arrays for coarse operations
- 7:59Logistic regression demo
- 9:04CRTS scheduler design
- 12:32Performance evaluation results
- 15:00Benchmark comparisons
- 16:26GPU support and scalability
- 17:50Real-world Higgs benchmark
- 18:13Installation and features
- 19:17Future work and conclusion
related talks
25mScale Smarter, Not Harder, with cuPyNumeric.
30mCuPy: A NumPy-compatible Library for GPU
01h 22mUsing Dask for Parallel Computing in Python
47mBig ideas shaping scientific Python: the quest for performance and usability
30mCUDA in Your Python: Effective Parallel Programming on the GPU
44mUsing Dask for Parallel Computing in Python
45mPython for Speed, Scale, and Science
43mKEYNOTE: Scaling Out PyData