Apache Beam on Dask- Portable, Scalable, Scientific Python
summary
Claims a working Beam-on-Dask translator that visits Beam pipelines and emits Dask high‑level graphs, with the change landed upstream in Apache Beam and a Dask runner that executes pipelines. Demonstrations compile Pangeo Forge recipes into Beam pipelines and run a two‑terabyte dataset in a local Jupyter notebook using xarray+Dask. Work is portable across HPC, Kubernetes and cloud, focused on scalability and portability, and remains experimental but upstreamed.
AI-written summary · about
chapters (21)
- 0:00Apache Beam on Dask intro
- 1:27Data engineering big picture
- 3:12Dask parallel computing
- 4:37Word count example
- 7:04Beam’s big data vision
- 9:12Beam key concepts
- 11:26Dask vs Beam paradigms
- 13:50Parallel frameworks case studies
- 15:58Data engineering climate crisis
- 17:00Weather data challenges
- 21:32Pangea Forge intro
- 24:21Pangea Forge demo
- 26:50Pangea Forge challenges
- 28:39Adopting Beam for Pangea
- 30:02Beam on Dask demo
- 31:05Open science with xarray-beam
- 32:52Key lessons future
- 33:58Q&A 1
- 36:24Q&A 2
- 37:07Q&A 3
- 38:50Q&A 4
related talks
38mData processing with Apache Beam
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]
41mTalk Python To Me: Stream Processing in your favorite language with Beam on Flink
01h 03mDask: Next Steps in Parallel Python
38mBeam on Flink: How does it actually work?
28mAdapting from Spark to Dask: what to expect
01h 23mData Science at Scale with Dask
01h 26mDask tutorial