PHP at the Firehose Scale
summary
Twitter publishes 500 million tweets a day and DataSift aims to route each relevant event to customers in ~1 second end-to-end. The stack actually used is PHP for ingestion and delivery, C++ for extreme-volume filtering (a custom goblin engine), Kafka for store/forward, Hadoop/HDFS for historic queries, and connectors to BigQuery, MySQL, MongoDB, S3 and Redis. Peaks force careful design: pipeline job-queue managers, long‑running workers, a Hornet firehose simulator, and measured claims like PHP handles all ingestion and delivery under production load.
AI-written summary · about
chapters (40)
- 0:00Introduction
- 4:30Scraping
- 7:50Firehose
- 12:47Architecture diagram
- 13:24Goblin
- 13:59PHP
- 14:25ETL
- 15:44URL
- 16:33Language Detection
- 18:15Architecture
- 18:36Supervisor
- 19:17Manager
- 20:36Pipelines
- 23:27Delivery
- 24:48Kafka
- 25:24Push Scheduler
- 28:21Load ETL
- 28:32More than one firehose
- 29:35Scale up
- 31:35Where PHP fits
- 36:10Summary
- 36:37History of PHP
- 37:03Why PHP
- 37:43PHP was no risk
- 38:17How well PHP works
- 38:32Other languages
- 39:35NoJS
- 40:04JSONDecode
- 40:35Behavior
- 40:57Scale
- 41:24String handling
- 42:16Encoding
- 42:30JVM
- 42:43Connectivity
- 42:55Bundled Extensions
- 43:14Liability in Production
- 43:43Quality Threshold
- 44:18PHP Philosophy
- 44:35Slide Summary
- 46:19Q&A
related talks
22mThe Open Source Technology Behind a Tweet
18mHigh-Load Systems: Challenges in Social Network Development
12mToday's LAMP Stack
45mPHP is the king ...
32mHow we abused PHP and Laravel to do Big Data.
46mScaling Pinterest
58mMassively Scaled High Performance Web Services with PHP
45mScaling: from 0 to 20 million users