A Java Implementer's Guide to Better Apache Spark Performance
summary
Java VM work focuses on concrete wins for Spark performance: JIT tweaks yield 1.35x faster on benchmarks versus OpenJDK8 and reduce JNI pain points around the Snappy serializer, Netty I O gets an RDMA plugin that cuts latency and shows 30% less CPU on shuffles, CAPI flash block storage replaces filesystem overhead, and GPU offload moves array ops at an arrays.sort offload threshold 20k and accelerates a Spark linear regression from 700s to 28s.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:47About Apache Spark
- 3:26What is Spark
- 7:00Master Node
- 9:27RDD
- 14:04JVM Performance
- 15:06Enhancements
- 17:39Logging
- 19:29Virtualisation Methodism
- 20:57JIT Improvements
- 22:32Serialization
- 25:20Networking
- 35:02GPU Implementation
- 36:40GPU Explicit
- 37:57Example
- 46:42Summary
related talks
30mImproving PySpark Performance: Spark performance beyond the JVM
51mJava at Speed: Getting the Most Out of Modern Hardware
35mJava at Speed: Building a Better JVM
43mHow to make your electrons do more in Java?
48mJust-In-Time Compilation for Java Performance: Recent and Ongoing Improvements
48mJava 9 Performance
51mJava Performance Update: From JDK 21 to JDK 25
55mThe Performance Engineer's Guide To Java HotSpot