One Billion Row Challenge in Golang
summary
Builds a Go pipeline that processes a 16GB input file and iteratively improves throughput using Go CPU and flamegraph profiling. Starts with a scanner+map baseline at 6m13s → 28s by: replacing scanner with 64MB chunked reads, switching float parsing to ints, summarizing per-station stats instead of storing all values, adopting a map-reduce with 9 workers pattern, and shrinking channel messages from 10M items to 512. Shows profiles at each step and timing numbers.
AI-written summary · about
chapters (10)
- 0:00Introduction to talk
- 1:07What is 1BRC challenge
- 3:54Baseline implementation
- 6:46Adding concurrency
- 8:10Producer-consumer pattern
- 12:25Optimizing data structures
- 14:55Reading file in chunks
- 17:50Map-reduce pattern
- 20:37Final optimizations
- 21:22Further improvements
