Twitter Heron in Practice
summary
Heron runs in production at Twitter for nearly two years, processing billions of events and replacing Storm with a Storm-compatible topology model that runs as containers on Aurora. It builds a TopologyMaster in container zero plus per-node stream managers that provide backpressure (slowing sources rather than dropping data) and automatic reschedule of stragglers. Measured wins include 3–4x reduction in CPU/machine cost, with real jobs requesting 120 CPUs but using 30–50 and spouts consuming ~84% of runtime.
AI-written summary · about
chapters (28)
- 0:00Intro
- 0:53TALK OUTLINE
- 1:31TWITTER IS REAL TIME
- 3:05HERON TERMINOLOGY
- 4:24HERON TOPOLOGY BOLT
- 5:37WORD COUNT TOPOLOGY
- 7:28STREAM GROUPINGS
- 9:33HERON DESIGN DECISIONS PULLY A COMPATIBLE WITH STORM
- 10:42HERON ARCHITECTURE
- 12:00TOPOLOGY ARCHITECTURE
- 17:24TOPOLOGY EXECUTION
- 18:47HERON SAMPLE TOPOLOGIES
- 20:11HERON USE CASES
- 20:56HETEROGENOUS ECOSYSTEM
- 24:02HERON - MICROSTREAM ENGINE
- 25:13HERON ENVIRONMENT
- 26:56WHY AURORA?
- 27:50STRAGGLERS
- 29:41DROP DATA STRATEGY
- 30:34SLOW DOWN SENDER STRATEGY back pressure
- 33:44BACK PRESSURE IN PRACTICE
- 35:06LOAD SHEDDING
- 35:29HERON RESOURCE USAGE
- 36:46PROFILING SPOUTS
- 37:26PROFILING BOLTS
- 37:46RESOURCE CONSUMPTION - BREAKDOWN
- 38:02HERON EXPERIMENT RTAC topology
- 38:12HERON PERFORMANCE CPU usage - RTAC Topology
related talks
52mProcessing Billions of Events in Real-Time with Heron
50mProcessing Billions of Events in Real-Time with Heron
37mStorm, a Distributed Real-time Data Processing System
01h 06mThe Legends of Twitter and Beyond: Real-World Architectures at Scale Panel
20mBuilding Apache Heron
30mThe Distributed Database Behind Twitter
40mBrace yourself! Storm is coming!
27mHydra: a federated resource manager for data-center scale analytics