A complete Tweet index on Apache Lucene #bbuzz
summary
Claims a complete searchable index of every tweet ever (hundreds of billions, ~500M new tweets/day, ~2B queries/day) built on Apache Lucene variants with an Earlybird in-memory index for realtime, SSD-backed Lucene for larger tiers, and a rolling tiered layout to add capacity without repartitioning. Offline pipeline uses Mesos jobs to build segments, copy to HDFS and let replicas bootstrap via HDFS bootstrapping, with a segment builder and parallel index reader to append new fields without full reindexing; system scales and recovers under real spikes.
AI-written summary · about
chapters (13)
- 0:00Start
- 7:55Earlybird
- 11:27Offline Ingestion pipeline
- 16:51Performance tuning
- 19:41Time Tiers
- 20:27Search roots
- 21:22In-memory Real-time Index
- 22:07Outlook
- 23:02Segment Slices
- 23:51Parallel Ingestion pipeline
- 25:23Search Architecture
- 25:55Lucene Extension Library
- 27:06Demo
related talks
21mReporting Systems Built on Top of Hadoop & Lucene
40mSmarter Search with Spark, Solr and Machine Learning
29mFrom Bottlenecks To Breakthroughs: Optimizing Aggregation in OpenSearch for Lightning-Fas...
37mFrom Inverted Index to Columnar Vectorized Execution Search
26mHigh-Performance Full-Text Search on Apache Iceberg Using Lucene-Integrate...
39mLive-Hack: Analyzing 7 years of Buzzwords at Scale #bbuzz
30mData @Scale Seattle Samir Goel
20mBuilding a Personalised Home Feed #bbuzz