Introducing the Dask Active Memory Manager
summary
A scheduler-side daemon called the Active Memory Manager runs every few seconds and enacts policy suggestions to replicate or drop in-memory task replicas; it is experimental and off by default and is enabled via client.amm.start or config (Dask 2022.2+). Built policies include remove-replicas (on by default) and retire-worker, with planned rebalance and replicate; demo shows total cluster memory capping at 46–47 GB versus much higher without the AMM, preventing hard OOMs and enabling graceful worker retirement.
AI-written summary · about
chapters (6)
- 0:00Start
- 5:14Issues in Memory Management
- 8:59Drop a Replica
- 13:53Amm Remove Replicas
- 17:51Re Reduce Replicas Policy
- 23:43How Do You Define a Custom Policy
related talks
26mSingle node shared memory comes to dask
18mUsing Dask to Distribute Machine Learning Jobs
44mDask for ad hoc distributed computing
38mScalable Machine Learning with Dask
01h 03mDask: Next Steps in Parallel Python
02h 18mHacking Dask: Diving Into Dask;s Internals
04mDask Powering lower end PC
27mData Processing on Ray