keynote
summary
Claim: a usable out-of-core multicore numpy clone exists that turns array expressions into task graphs and runs them on one machine. It uses blocked algorithms and a lightweight task scheduler, reads netCDF/HDF5 slices, plugs into matplotlib, and often releases the GIL by running C/Fortran kernels. Most data fit in a gigabyte or terabyte (95% / 99% claimed), dask is pip/conda installable, and demos run on a multi‑core laptop without loading whole arrays into memory.
AI-written summary · about
chapters (19)
- 0:00Conference logistics
- 3:41Speaker introduction
- 5:06Continuum and open-source
- 7:37Data sizes and parallelism
- 10:25NumPy and Pandas limits
- 12:54Python's Global Interpreter Lock
- 14:59PyData parallelism
- 17:28GIL release and community
- 18:34Dask Array introduction
- 21:46Dask Array demo
- 26:29Dask task graphs
- 31:30Dask memory and execution
- 34:58Weather data use case
- 38:31Dask caching
- 41:24Extending Dask
- 42:31Dask graphs and scheduling
- 45:18scikit-image and GIL release
- 46:22Future of PyData parallelism
- 46:42Q&A
