Pushing Cython to its Limits in Scikit-learn
summary
scikit-learn shows exactly what was built and what worked: Cython + C++ stdlib + OpenMP for parallelism, Tempita code generation, and fused types produce concrete wins — a hist gradient boosting implementation matches LGBM performance, pairwise-distance work yields ~20% speedup across modules, and a rewritten target encoder is 4–5× faster with lower memory. techniques include memoryviews with no-copy view into C++ vector, releasing the GIL, build-time toggles so bounds checks enabled in CI but disabled in releases, and pragmatic leak detection via user reports.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:06Why Cython for performance
- 5:47Profiling and optimizing
- 8:58Cython basics
- 11:06Cython tips and tools
- 11:27Python-Cython interface
- 13:59Memory views for data
- 18:34Returning memory views
- 22:26Structs and memory views
- 25:57Extending with Cython classes
- 26:19Releasing GIL for parallelism
- 28:46Optimizing NaN checks
- 30:10OpenMP and Cython
- 31:56C++ in Cython
- 33:02Fused types support
- 34:30Code generation
- 35:14Summary of optimizations
- 35:34Q&A
related talks
42mLift your Speed Limits with Cython
25m"Boosting Python Numeric Computations with Dynamic C++ Integration"
21mKeynote: Easier Machine Learning Thoughts From Scikit-Learn
31mDo You Want To Build A Forest?
31m"Optimizing Scientific Python with C++, CUDA, and Serverless Compute"
40mGet native with Cython
33mEasy wins with Cython: fast and multi-core
27mSpeed up Python with hardware accelerators