Java, How Fast Can You Parse 1 Billion Rows of Weather Data?
summary
Challenge: parse 1 billion rows (~15GB) of city + one-decimal temperatures with Vanilla Java and go from a 4+ minute baseline to sub-2-second results. Uses parallel segmented reads, memory-mapped files, low-level access via sun.misc.Unsafe, SIMD-as-register SWAR tricks and branchless parsing to avoid allocations, a custom forward‑probing hash table to aggregate without objects, Graal native-image and Epsilon GC for minimal runtime overhead, plus a worker-process unmap trick; final validated runs hit ~1.5s (winner) and several entries under 2s.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:49The challenge
- 6:07Watch, learn, adopt, experiment
- 8:00Mechanical sympathy
- 9:32Temperature as integer
- 10:37Memory mapped files
- 11:54Getting unsafe
- 13:31SWAR
- 17:22Stringless
- 18:18Branchless programming
- 20:35Parse the temperature
- 30:14Keeping track
- 36:22Which JVM?
- 37:21Graal (native-image)
- 39:38Summary
- 40:50Results
- 42:00Outro
related talks
40mHow Fast Can You Parse 1 Billion Rows in Java? – Insane Speed Test
02h 57mPushing Java to the Limits: Processing a Billion Rows in under 2 Seconds
47mThe First 80% of Reading One Billion Rows Fast Enough
47mConquering a Billion Rows with Modern Java Concurrency
49m1BRC – Nerd Sniping the Java Community
47mUltralow Latency with Java 11 and Terabytes of Data
51mSecrets of top performance in Java revealed
44mAtomic Habits in writing high performing distributed code in the cloud