Conquering a Billion Rows with Modern Java Concurrency
summary
Takes the 1 Billion Row Challenge from a 77s one-thread baseline on cloud hardware and targets 8 cores using only the JDK (no Unsafe, no external libs) and shows concrete wins: replace String.split and Double.parse, move to byte-based parsing, use chunked ForkJoin and RandomAccessFile/mmap to scale to 8–16 threads, profile with async-profiler and Linux perf stat, and drop to ~11s end-to-end; a final open-address hash table plus Epsilon GC hits ~6–7s and the approach proves practical and reproducible.
AI-written summary · about
chapters (12)
- 0:00Introduction and motivation
- 6:35Baseline solution analysis
- 13:18Parallel processing attempts
- 18:23Fork join pool approach
- 26:35Profiling and performance bottlenecks
- 31:16Optimizing string splitting
- 34:21Replacing parseDouble
- 36:51Using integers instead of doubles
- 38:15Mutable state for performance
- 39:59Avoiding strings entirely
- 41:26Final solution with fork join pool
- 45:46Memory optimization with Epsilon GC
related talks
02h 57mPushing Java to the Limits: Processing a Billion Rows in under 2 Seconds
47mThe First 80% of Reading One Billion Rows Fast Enough
40mHow Fast Can You Parse 1 Billion Rows in Java? – Insane Speed Test
49m1BRC – Nerd Sniping the Java Community
46mLessons from 1BRC: Tricks and Approaches to Performance Optimization
47mMore Action, More Overview – From 4k to 1M Threads
42mJava, How Fast Can You Parse 1 Billion Rows of Weather Data?
52mSailing Modern Java