Bodo: Automatic HPC Performance and Scaling for Data Processing in Python
summary
A compiler takes standard pandas/numpy Python and SQL (via Calcite), infers data distributions, and emits native MPI binaries that run SPMD; real users report about 20x speedup versus tuned Spark (12x on a retail ETL, 20x+ on large TPCx workloads, 4 TB sort 8x faster on 4,500 cores). It uses array-parfor fusion, bloom-filtered joins, and sampled parallel sort; limitations are that only supported APIs and stable schemas/types auto-parallelize and long jobs need checkpoints for spot instance loss.
AI-written summary · about
chapters (22)
- 0:00Start
- 2:46General Outline
- 6:59Halide
- 7:42Big Data Framework Approach
- 14:38Inferential Compiler
- 18:30Compiler Pipeline
- 22:36Automatic Parallelization
- 27:00Examples of Transfer Functions
- 27:39Join of Tables
- 32:26Results
- 37:36Limitations
- 45:10Example Code
- 47:53What Is Snowflake Python Code
- 49:43Performance
- 50:21Optimizations
- 50:32Compiler Optimizations
- 52:46Filter Push Down
- 53:25Parallel Computing Optimizations
- 54:32Implementation Efficiency
- 57:28Parallel Sorting
- 1:00:06How Do You Manage the Resource Management
- 1:00:47Resource Management
related talks
26mBodo: Supercomputing-Like Performance and Scale for Python/Pandas
40mScaling Python - Bank Edition
43mKEYNOTE: Scaling Out PyData
39mAccelerating Data Science with RAPIDS
41mMassively Parallel Processing with Procedural Python
33mLet's Run Python on a Supercomputer!
45mPython for Speed, Scale, and Science
30mSpeed is Not All You Need for Data Processing