Performance&Scale Domain Oriented Objects vs Tabular Data Structures
summary
Compares row‑oriented objects (records, boxed Integers, JDK collections) to columnar/tabular approaches and shows concrete memory wins using Java Collections, streams, Eclipse Collections, EC DataFrame, and the jaw Java object layout tool. Synthetic CSV workloads of 1M, 10M and 25M rows (2.2GB) reveal boxed Integer objects ~368B vs primitives ~72B (~5x), JDK HashSet ~640B vs Eclipse primitive set ~120B, and per‑row cost dropping from ~56B to ~40B after pooling and a twin date compression. A CSV run at 25M rows forces Jackson OOM; the columnar+pooling solution works and approaches raw CSV footprint.
AI-written summary · about
chapters (18)
- 0:00Domain vs tabular data structures
- 2:08Eclipse Collections origins
- 6:02Memory-optimized data structures in Java
- 9:50Simulating big data in Java
- 13:46Boxed vs primitive memory impact
- 17:21Mutable vs immutable sets memory
- 20:34Immutable sets optimizations
- 23:50Specialized Java list implementations
- 28:04Java Streams vs Eclipse Collections vs Data Frames
- 31:55Object-oriented to tabular data
- 36:33Mutable vs immutable memory impact
- 39:25Pooling for large datasets
- 41:12Data understanding for optimization
- 45:53Row vs column data structures
- 48:04Memory compression techniques
- 50:53Java memory optimization future
- 55:11Resources and code
- 55:52Key takeaways
related talks
51mIn-Memory Computing - The Big Picture
32mJava's Ignored Potential: Supercharge Database Apps with Java In-Memory Process
43mCaching and Beyond - Smarter Data Processing with Java
01h 12mHidden pearls for high-performance-persistence in Java
47mDesigning Great Java In-Memory Object Models
20mGive me the Data
40mFast Access To Your Complex Data #bbuzz
42mContemporary Approaches to Data at Scale