1BRC – Nerd Sniping the Java Community
summary
A one‑billion‑row CSV (13 GB) is reduced to an aggregated min/max/avg in rules‑bounded runs using parallel chunked reads, memory‑mapped files, integer arithmetic for one‑decimal inputs, and either JVM Vector API/SIMD or SWAR bit‑tricks plus a custom open‑addressing map to avoid rehashing. Native image plus Epsilon GC and 2MB task chunks push throughput: 1.5 seconds on eight cores and ~300 ms on a 32‑core machine; some entries exploit the generator’s ~400 station names for extra speed.
AI-written summary · about
chapters (24)
- 0:00Introduction to the challenge
- 2:29Defining the challenge
- 5:17Setting up the environment
- 7:40Baseline implementation
- 9:34First optimizations
- 10:19Parallelization strategies
- 11:27Memory mapping and I/O
- 12:26Chunking and concurrency
- 14:41Addressing community feedback
- 16:19Parsing techniques
- 19:07SIMD and vectorized logic
- 19:56SWAR and bit-level hacks
- 22:18Branchless programming
- 23:51Advanced optimization risks
- 24:07Operational challenges
- 27:48Custom hash map structures
- 30:33Handling hash collisions
- 32:11JIT vs. AOT compilation
- 33:01Garbage collection and startup
- 35:40Advanced performance tricks
- 38:45Analyzing final results
- 39:35When to optimize
- 43:40Community influence and reach
- 46:09Lessons learned and conclusion
related talks
47mThe First 80% of Reading One Billion Rows Fast Enough
47mConquering a Billion Rows with Modern Java Concurrency
40mHow Fast Can You Parse 1 Billion Rows in Java? – Insane Speed Test
46mLessons from 1BRC: Tricks and Approaches to Performance Optimization
42mJava, How Fast Can You Parse 1 Billion Rows of Weather Data?
47mUltralow Latency with Java 11 and Terabytes of Data
45mScooby RAM, where are you?
51mJava at Speed: Getting the Most Out of Modern Hardware