Realtime Search at Twitter
summary
Twitter runs a modified Lucene engine called Earlybird indexing ~340–400 million tweets/day and serving ~2.3 billion queries/day with <10s indexing latency and ~50ms average hits. It keeps live, in-memory time-slices of up to 60M docs, stores each posting as a 32-bit value (24-bit doc id + position byte) to allow backward reads and early termination, uses slice pools (2,16,128,2048) to grow posting lists lock‑free, relies on a volatile maxDoc publish barrier for safe reader snapshots, and reports 80–400% GC/indexing gains.
AI-written summary · about
chapters (11)
- 0:00Start
- 1:22Agenda
- 3:19Search Architecture
- 8:48Posting Lists
- 17:18Document Frequency
- 21:05The Data Structure
- 27:44Concurrency
- 29:05Why Is It Called Optimistic Locking
- 31:24The Program Euler Rule
- 39:35Optimistic Locking
- 41:09Encoding of the Doc Ids
related talks
42mBuilding a real-time news search engine #bbuzz
36mSearch on the fly: how to lighten your Big Data
34mBig and Small Data at @Twitter
41mInside Twitter's Realtime Delivery of Tweets for Enterprise Customers
09mHigh Order Bit: The In
37mAlgorithms & data-structures that power Lucene & ElasticSearch
17mHow Similarweb Serves 100s of TBs to Their Worldwide Users in Milliseconds
38mIntroduction to Search