Data @Scale Seattle Samir Goel
summary
Firefly is Dropbox’s from‑scratch full‑text search backing Firefly that indexes hundreds of millions of documents and over 10 trillion tokens, processes more than 1 billion updates/day, and serves queries under 250 ms while making edits searchable in under 10 seconds. Design uses namespace sharding for per‑user privacy, a hybrid instant‑indexing (mutation log pulled by leaf servers) plus batch rebuilds on HDFS for availability, and relies on Apache Tika, HBase, LevelDB and RabbitMQ; backend runs as a compact ~30k LOC MVP by three engineers.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:35Challenges
- 5:57Recap
- 7:17Indexing
- 10:15Private Space
- 13:00Serving System
- 17:45Batch Indexing
- 22:24Stats
- 24:10Summary
- 24:49Questions
related talks
20mScale By The Bay 2019: Sharanya Santhanam & Shiv Toolsidass
44mApache NIFi Deep Dive 300
34mBuilding a WebScale Search Engine with a Data Driven Architecture
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache Flink
35mConquering All Stores with Gimel – A Unified Data Processing Platform
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse
23mFeathr: An Enterprise-Grade, High Performance Feature Store