Processing One Billion Rows in PHP
summary
The talk shows a concrete path from a naive PHP script that takes ~25 minutes on 1B rows to a production-ish solution under 13 seconds: replace fgetcsv with line parsing, cast temperatures to floats, avoid repeated hashmap lookups, enable opcache_enable_cli and JIT (7m19s), then split the file into byte-aligned chunks and run workers with the parallel extension (1m35s → 27.7s with an optimized build → 12.7s after stream_get_line and string-copy reductions).
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:281 billion row challenge overview
- 3:38Challenge rules
- 6:05Naive PHP solution
- 8:54Profiling initial solution
- 10:18Optimizing with fgets
- 12:50Reducing hashmap lookups
- 15:42Simplifying comparisons
- 17:08Fixing type juggling
- 19:41OPcache for CLI
- 21:05Parallel processing
- 27:47Debug vs optimized builds
- 29:18Final runtime
- 31:03Bonus optimizations
- 34:34Key takeaways
related talks
45mPHP's biggest bottleneck, MySQL
39mPHP Performance Demystified - Measure, Understand, Optimise
01h 02mSpeeding up the Web with PHP 7
52mPerformance optimisation: how do I go about it?
47mFun with BIG Data with various databases
52mPHP at the Firehose Scale
43mSpeeding up the Web with PHP 7
01h 12m25 Years of PHP