Oak: A scalable off-heap allocated key-value map
summary
A Java key-value map stores keys and values off-heap in preallocated 256MB blocks, exposes a zero-copy API via ephemeral OakBuffers, and uses Unsafe/direct ByteBuffer addresses plus a 64-bit per-value header for synchronization. Chunks hold ordered integer-entry arrays and split when full, enabling a backward-scan algorithm that walks anchored ordered ranges instead of O(log n) pointer rewinds. The API supports atomic compute/putIfAbsent and uses on-heap minimal metadata (~2.5–3%). Evaluations show about 2x throughput vs ConcurrentSkipListMap and successful integration into Apache Druid.
AI-written summary · about
chapters (14)
- 0:00Introduction to Oak
- 1:49Understanding off-heap
- 5:46Memory management basics
- 8:52Oak design and contributions
- 11:50Design principles
- 16:03Data organization
- 20:43Q&A session 1
- 26:04Zero copy API and scanning
- 30:06Concurrency and scan logic
- 35:37Advanced memory management
- 45:24Q&A session 2
- 53:14Performance evaluation
- 58:44Real-world deployment
- 1:04:45Conclusion and outlook
related talks
46mOn-heap cache vs Off-heap cache w Javie
01h 12mHidden pearls for high-performance-persistence in Java
45mEasy Scaling with Open Source Data Structures
01h 08mDistributed Java Databases Under the Hood
29mMapDB - taking Java collections to the next level
57mJava In-Memory Database-Apps with EclipseStore
34mLightweight Relations
48mHidden Pearls for Java Embedded High-Performance Persistence