Bodo: Supercomputing-Like Performance and Scale for Python/Pandas
summary
A platform compiles standard Python (pandas/numpy) into a inferential compiler that produces a native MPI parallel binary so every core runs the program instead of fanning out tiny tasks. Reported results: 20x faster than tuned Spark on a 40 TB TPCx run (4,500 cores, ~95% cloud cost savings), 23x average (up to 65x) on a 1 TB AWS cluster, and a retailer pipeline cut 1.1 hours to 6 minutes on 1.2 TB (25B rows). Limitations: dynamic typing and unknown dataframe schemas require refactor. 20x–65x speedups community edition (8 cores, conda)
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:46Scale-Out Analytics Computing Is Complicated
- 1:52Scale-Out Analytics Computing Is Simpler with Bodo
- 2:38Bodo Example Code
- 4:27The Bodo Parallel Compute Platform
- 5:06Customer Benchmark: TPCxBB Q26
- 6:35Simplicity vs. Performance Gap
- 7:49What is Our Inferential JIT Compiler?
- 8:46Bodo: True Parallel Architecture
- 10:33We're Closing the Data Analytics Simplicity-Performance Gap
- 11:06Python can be used for hard Data Engineering problems
- 11:52Scalable 1/0
- 12:40Distributed Fetch from Snowflake
- 14:20Bodo Limitation: type stability
- 16:38Bodo vs. Others using TPC-H
- 18:29Bodo vs. Spark (TPC-H)
- 20:02Bodo vs. Ray/Modin (TPC-H)
- 20:47Customer Use Case: Retail Product Analytics
related talks
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
43mKEYNOTE: Scaling Out PyData
40mScaling Python - Bank Edition
45mPython for Speed, Scale, and Science
14mKeynote: Scalable Python We Can Afford
33mLet's Run Python on a Supercomputer!
25mScale Data Science by Pandas API on Spark
30mSpeed Up Your Data Processing