ZenMonitor: Scaling Distributed Monitoring
summary
ZenMonitor is a drop-in replacement for BEAM monitors that reduces network traffic and prevents thundering-herd failures by batching, throttling and localizing monitor logic. It uses GenStage producers/consumers and connector/proxy batching to replace millions of remote monitor messages with compact death certificates, truncates huge stack traces with truncate-er, and controls dispatch rate to avoid 100% scheduler spikes (observed peaks fall under ~10%). Deployed since Feb 20, 2018 and open sourced, running ~250 million local monitors monitoring ~150 million remote processes as ZenMonitor on Hex.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:11Spawning
- 2:07Linking
- 3:21Monitors
- 4:31Context
- 5:34Examples
- 9:42Thunder
- 10:49starvation
- 12:23amplification
- 15:21throttling
- 16:22batch
- 18:07calming
- 19:28interlacing
- 20:09preprocessing
- 21:24ZenMonitor
- 22:36Instruments
related talks
43mDistributed Monitoring: How to Understand the Chaos
34mMonarch, Google’s Planet Scale Monitoring Infrastructure
18mDDMon: a Monitoring Tool for Distributed Deadlock Detection
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
01h 03mZenoh: The Edge Computing Data Fabric
01h 00mSystems @Scale Summer Q&A Week 4
47mMonolith to Microserv. at 20M+ requests per second: Latency and Scale Challenges
40mDecomposing a live application with confidence