The History of Logging @ Facebook (Abridged)
summary
Scribe evolves from a C++ thrift collector into a multi-language pipeline that scales to thousands of categories and peaks of hundreds of GB/s; Calligraphers writes to HDFS for years, then a staged migration replaces HDFS with LogDevice (record‑oriented, append‑only, trimmable streams, distributed metadata) using a new Scribe router, double-write tests, a Ptail auto‑upgrader and Ztail replacement. Migration uncovers HDFS bugs, takes months of testing and automation, and finishes with real-time stream processing running on LogDevice.
AI-written summary · about
chapters (10)
- 0:00The legacy system
- 2:49Introduction of scribe
- 6:09Scaling with HDFS
- 10:34Limitations of HDFS
- 13:49Logdevice transition
- 15:48Migration strategy
- 21:22Migration execution
- 26:58Automation and completion
- 31:23Advanced use cases
- 39:05Q&A and conclusion
related talks
31mThe Long and Winding Road to Application Logging [A]
37mScaling Yelp’s Logging Pipeline with Apache Kafka #bbuzz
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
51mCase Study of Moving from CRUD to Log Based State Management
41mFrom Batch to Realtime with Hadoop
46mBuilding a scalable logging platform