Updateable Fields and other Fun with Codec API
summary
Claims that Lucene 4’s Codec API can add updateable fields and other on‑the‑fly behaviors, and demonstrates a prototype that writes updates to separate stacked segments, stores a new→old internal‑ID mapping in memory, and merges data at read time using leapfrog enumerators for postings. Examples include an online filtering codec that prunes frequent terms and a single-pass index splitter that streams documents to shards. Implementation is incomplete: it works conceptually but costs random seeks at query time, doesn’t handle field deletion yet, and lives behind a JIRA branch.
AI-written summary · about
chapters (15)
- 0:00Introduction and welcome
- 0:26Codec API primer
- 5:10Custom codec examples
- 10:10Field level updates problem
- 15:19Holy Grail analogy
- 16:02IBM research paper
- 17:29Proposed update design
- 21:04Reading stacked updates
- 24:42Merging storage fields
- 27:34Merging inverted fields
- 31:12Implementation details
- 34:24Segment merging impact
- 34:46Design limitations
- 36:39Current status and call
- 37:26Q&A session
related talks
35mGetting Started with the OpenSearch Core Codebase
55mNew Bytecodes, New Objects
06mVerified Reversible Programming for Verified Lossless Compression
19mPanel: Compilation Frameworks
19mCompile-Time Java Stream Fusion via mapMulti
39mBytecode Pattern Matching
42mFrom Linear to Incremental
51mBeyond ByteBuffers