Pushing Java to the Limits: Processing a Billion Rows in under 2 Seconds
summary
Claim: Java parses a billion weather rows in under two seconds (reference >4 minutes → under 2s). Uses Graal native image, sun.misc.Unsafe, memory-mapped files plus SWAR/branchless parsing, perf/hyperfine benchmarking and machine-level tuning. Builds a forward‑probing hashmap, an unsafe scanner, chunked worker segments and a subprocess trick to hide kernel unmap; result runs ~1.3s on 8 cores (~50 cycles/row, ~100 machine instructions → 117 later, branch misses dramatically reduced) and proves the approach works.
AI-written summary · about
chapters (21)
- 0:00Introduction to challenge
- 2:55Contest rules and setup
- 7:38Reference implementation analysis
- 10:10Optimization progress overview
- 16:23Native image compilation benefits
- 22:30Parallel processing and data structures
- 28:52Memory mapping files
- 32:23Unsafe memory access
- 40:24Long-based string comparisons
- 43:34SWAR scanning and bit manipulation
- 51:04Branchless temperature parsing
- 1:02:30Data distribution specialization
- 1:13:42Custom hash map implementation
- 1:21:33Linux kernel profiling
- 1:31:12Subprocess trick for unmapping
- 1:38:16Fine-grained parallel processing
- 1:49:07Perf profiling and machine code
- 2:01:40Compiler graph visualization
- 2:10:58Branch prediction and misses
- 2:25:55Vector API and future optimizations
- 2:36:11Q&A session
related talks
40mHow Fast Can You Parse 1 Billion Rows in Java? – Insane Speed Test
42mJava, How Fast Can You Parse 1 Billion Rows of Weather Data?
47mConquering a Billion Rows with Modern Java Concurrency
47mThe First 80% of Reading One Billion Rows Fast Enough
47mUltralow Latency with Java 11 and Terabytes of Data
51mSecrets of top performance in Java revealed
51mJava at Speed: Getting the Most Out of Modern Hardware
47mMore Action, More Overview – From 4k to 1M Threads