Processing Billions of Events in Real-Time with Heron

on YouTube ↗52m

" summary

Heron is an open‑source stream engine that Twitter runs in production for ~2.5 years to process billions of events, claiming 3x reduction in cores and memory versus Storm by moving to per‑task processes and container packing. Topologies run as independent task processes with per-task processes for isolation and integrate with schedulers like Aurora/Mesos. It supports at-most-once and at-least-once semantics, scales to 800-container topologies (~140 machines), and implements backpressure via stream‑manager buffers.

AI-written summary · about

" chapters (21)

1. 0:00Introduction
2. 1:10Agenda
3. 2:21Terminology
4. 3:39DAG
5. 4:36Word Count
6. 6:12Parallelization
7. 7:20Grouping
8. 8:48Issues with Storm
9. 12:48Design Decisions
10. 14:27Heron Architecture
11. 15:24Containers
12. 16:18Topology Master
13. 24:29Topology Architecture
14. 28:01Use Cases
15. 29:59Big Data Evolution
16. 30:47Micro Kernels
17. 31:20Streaming Systems
18. 35:37Environments
19. 40:53Handling Stragglers
20. 49:51Load shedding
21. 50:56Summary

" related talks

• 50mProcessing Billions of Events in Real-Time with HeronKarthik Ramasamy · LinuxCon + ContainerCon North America 2016
• 39mTwitter Heron in PracticeKarthik Ramasamy · Data By the Bay 2016
• 37mStorm, a Distributed Real-time Data Processing SystemMariusz Gil · Confitura 2013
• 36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzzAmeya Kanitkar · Berlin Buzzwords 2015
• 20mBuilding Apache HeronJosh Fischer · BazelCon 2018
• 46mClojure Scaling the Event StreamDerek Troy West · Clojure/conj 2017
• 30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring SolutionMarcus Crestani · code.talks 2022
• 01h 06mThe Legends of Twitter and Beyond: Real-World Architectures at Scale PanelEvan Weaver, Ben Hindman, William Morgan, Mark McBride, Helena Edelson, Marius Eriksen · Scale By the Bay 2017