Performance Pitfalls: Tales From a Python/OCaml Codebase
summary
Semgrep combines an OCaml matching engine with a Python frontend and experiences three concrete performance failures: segfaulting stack overflows due to ulimit misconfiguration, huge startup overhead from invoking the OCaml binary dozens to hundreds of times (example: 100 OCaml calls turning a 2‑hour timeout into a 20‑minute run), and runaway memory from a 200MB targets file being copied across forks. Fixes include rewriting Python in OCaml, compressing rule indices, and moving to shared memory parallelism, which eliminates repeated startup and huge copy‑on‑write overheads.
AI-written summary · about
chapters (15)
- 0:00Introduction and SEM grep
- 1:55Introducing OCaml to Python
- 3:18Stack overflow in OCaml
- 3:59Startup time overhead
- 7:31Solving startup time issues
- 9:18Memory usage problems
- 14:14Solving memory consumption
- 15:39Takeaways and lessons learned
- 18:07Q&A: Profiling end-to-end
- 20:59Q&A: Pressure for performance
- 23:02Q&A: Static analysis details
- 25:32Q&A: Fearless refactoring
- 27:21Q&A: CI scanning setup
- 28:49Q&A: Profiling challenges
- 29:32Q&A: Python to OCaml porting
related talks
59mMyths and fairy tales around Python performance
30mWrite faster Python! Common performance anti patterns
28m"Performant Python"
27mAn Executable Semantics for Faster Development of Optimizing Python Compilers
50mPerformance tricks learned from low level developers
23mMemoirs of a Python Object: Memory Management and Improving Performance
36mHow to Optimize your Python Program for Slowness
39mPython Performance Matters