Scaling Python - Bank Edition
summary
Claims a working replacement of a legacy single‑threaded C++ engine by a distributed Python pipeline that actually handles 24.5 TB (140,000 parquet files, 6M rows each) and scales toward 1,400+ model sets. Experiments with GPUs, RAPIDS, BlazingSQL and OmnisciDB(Heavy.AI) reveal GPU IO overhead and cost outweigh compute gains, so the deployed solution uses Dask-based aggregation strategies with Argo Workflows for orchestration and Prefect patterns for reliability. Results: ~1700 aggregations across 140k files in ≈13 minutes on 20 large memory-optimized nodes.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:07RBC data processing
- 3:36GPU acceleration exploration
- 8:12GPU cloud challenges
- 10:00GPU compute limits
- 13:51CPU aggregation focus
- 14:12Problem and aggregation
- 15:39Memory chunking
- 17:03Throttling jobs with thread pools
- 19:51Group-by inefficiency
- 20:55Dask clusters
- 22:41Dask bags optimization
- 25:08Orchestration with Prefect
- 28:42Argo workflows
- 30:05Solution and outcomes
- 30:28Q&A
related talks
14mKeynote: Scalable Python We Can Afford
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
41mMassively Parallel Processing with Procedural Python
01h 22mHigh-Performance Data Science at Scale with RAPIDS, Dask, and GPUs
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
43mKEYNOTE: Scaling Out PyData
41mAccelerating Data Science with RAPIDS
31mScaling Python to thousands of nodes with Ray