Search on the fly: how to lighten your Big Data
summary
Describes a system that avoids traditional ETL by fetching and indexing only minimal per-entity metadata at search time to deliver real-time results for live customers. It uses Apache Lucene with in-memory RAM indexes, a Guava time‑based cache for index lifetime management, Kryo binary serialization to reduce HTTP payloads, and Server-Sent Events to push incremental results; benchmarks and tuning show per-chain and overall throughput and describe scaling with clustered nodes and sticky sessions.
AI-written summary · about
chapters (14)
- 0:00Introduction and speakers
- 1:30Agenda and business needs
- 2:13Overview of Lucene and its features
- 3:37Business requirements and challenges
- 9:59On the fly data retrieval concept
- 14:37Architectural components in detail
- 21:26Cache implementation with Guava
- 23:50Server-Sent Events for data push
- 25:58HTTP data exchange optimization
- 27:04Full system workflow
- 28:49Performance testing setup
- 30:36Test results and scalability
- 33:26Optimizing search engine performance
- 34:52Future improvements and conclusion
related talks
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
43mCaching and Beyond - Smarter Data Processing with Java
47mNo DB: Use a Search Engine!
54mAdvanced (elastic)search for your legacy application
42mBuilding a real-time news search engine #bbuzz
25mApache Lucene: From Text Indexing to Artificial Intelligence
37mAlgorithms & data-structures that power Lucene & ElasticSearch
01h 00mAdvanced search for your legacy application