HiPy: Extracting High-Level Semantics From Python Code For Data Processing
summary
HiPy converts annotated Python functions into program generators that, when executed, produce a statically-typed IR (SSA-like, n‑dimensional arrays and table types) enabling high‑level optimizations and C++ codegen via pybind11 and Arrow. Prototype handles real Python and common libraries (pandas, numpy, scikit‑learn) with a fine‑grained Python fallback; evaluation reports single‑threaded 2–20x speedups on data‑processing workloads and up to 150x on a few numeric cases.
AI-written summary · about
chapters (12)
- 0:00HiPy introduction and motivation
- 1:45Python challenges in data processing
- 3:56Existing Python optimization approaches
- 5:20HiPy typed IR transformation
- 7:48Handling complex Python code
- 9:33HiPy IR implementation and embedding
- 11:20Supporting pandas and NumPy
- 12:23Semantics for library functions
- 13:47End-to-end prototype and optimizations
- 14:48Evaluation and benchmarks
- 17:16Conclusion and future directions
- 18:19Q&A
related talks
27mAn Executable Semantics for Faster Development of Optimizing Python Compilers
16mPyHDL-IF: An Easy-to-Use Python/HDL Cross-Calling Interface
24mHPy: a better C API for Python
17mInside a PyPy JIT: is that even legal?
25mPython as a Hackable Language for Interactive Data Science
22mPyPy hands-on
22mPyPy hands-on
25mpypet: A Python Toolkit for Simulations and Numerical Experiments