The future of Lucene's MMapDirectory
summary
Lucene uses memory‑mapped files to avoid syscall and copy overhead but currently splits maps into 1 GB chunk workaround because Java ByteBuffer uses signed 32‑bit offsets, forcing many mappings and manual unmap hacks. Project Panama’s Foreign Memory Access API introduces shared memory segments and varhandles so Lucene prototypes map up to 16 GB default chunk (cutting mappings ~16x), support safe auto‑unmap via resource scopes, and run correctly on JDK 16–17+ prototypes. Windows stays unreliable; production use awaits JDK 18/19+ preview/stable releases.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:56T08 The "70s style" of file reads
- 2:52Problems
- 4:30With Java it gets worse!!!
- 5:31and why it's bad for Lucene: Multithreading
- 6:28The "90s style" of file reads
- 9:08Questions
- 9:57Memory Mapping: The "millenium style" of file reads
- 16:19Advantages
- 20:37Try-catch program logic
- 26:14The 19 year old feature request
- 32:08Key concepts
- 33:19Project Panama Incubation: JDK 15 (JEP 383)
- 36:04Thread confinement solution
- 38:51Why chunking?
- 42:10How about Lucene?
related talks
35mFuture of Lucene's MMapDirectory: Why use it & what's coming with Java 16 & later?
16mHeapBuffers: Why not just using a binary serialization format for your managed memory?
37mFast Decompression Lucene Codec #bbuzz
40mTrash Talk - Exploring the Memory Management in the JVM
39mMemory Efficient Java
51mIn-Memory Computing - The Big Picture
01h 32mTuning Large Scale Java Platforms
29mTowards an Alternative Memory Architecture