High-Performance Full-Text Search on Apache Iceberg Using Lucene-Integrate...
summary
Apache Pinot embeds Lucene inverted indexes into segment files (consolidated as byte ranges inside column.psf) and virtualizes segments over Iceberg Parquet so queries run against object storage with compute‑storage decoupling and tiered pinning of indexes to NVMe. Query flow uses broker/server pruning, bloom filters, request coalescing and async prefetching. Benchmarks show interactive results: 20 ms local vs 100 ms remote, and enabling per‑segment bloom filters cuts a 10s remote scan down to ≈1 s.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:51Analytics vs search systems
- 5:04Unifying search in Pinot
- 11:57Text search integration
- 12:17Cost challenges
- 16:37Segment format and index
- 19:27IO optimization
- 22:16Benchmark results
- 24:31Future work with Iceberg
- 25:58Summary
related talks
44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOT
49mIntroduction to Real-Time Analytics with Apache Pinot
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
31mHow Apache Iceberg Enables Multi-Engine Data Platforms
34mApache Iceberg.
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
39mFlexible OpenSearch Data Management With Apache Iceberg: Data...
39mIntroducing Apache Iceberg: Tables Designed for Object Stores