Keeping the lights on: Application monitoring with Sensu and BatsD
summary
Sensu and BatsD form a runnable stack that executes checks, pipes metrics through RabbitMQ and the statsd protocol, and alerts via handlers (PagerDuty, email) or stores time series for queries. Examples include a load‑average check, memcached stats, and revenue counters with hourly, percentage, and linear‑forecast rules. The stack operates in production at scale — over 1,000 checks/minute, thousands of metrics/sec on hundreds of servers — and is implemented in Ruby and works at Tapjoy.
AI-written summary · about
chapters (39)
- 0:00Intro
- 0:52What to take away...
- 3:14Growth comes with a cost
- 3:45This is downtime, right?
- 4:44Business metrics
- 5:29Application Performance
- 6:00System metrics
- 6:15Existing monitoring systems
- 8:00Overall Architecture
- 9:21Sensu: Server
- 10:07Sensu: Message Bus
- 11:47Sensu: Client
- 12:24Sensu: Handlers
- 13:06Sensu: Dashboard (Admin)
- 13:34Sensu: Architecture
- 14:00BatsD: Architecture
- 14:47BatsD: Storage
- 15:26BatsD: Data Types
- 16:24BatsD: Configuration
- 18:05BatsD: Rails Integration
- 19:47Checks: Load Average
- 20:45Checks: Memcached Metrics
- 21:48Checks: Config
- 23:09Checks: Community
- 24:24Business checks: Patterns
- 26:11Business checks: % Change
- 26:49Business checks: Forecast
- 27:53Handlers: Stats D
- 28:47Handlers: Pagerduty
- 29:58Handlers: Config
- 30:44Handlers: Community
- 31:14Tapjoy Monitoring
- 32:05Development process
- 33:33Automated Instrumentation
- 34:21Alerting is still hard
- 34:38Measuring is the easy part...
- 34:49Seriously, though, alerting...
- 36:07Final thoughts
- 37:47Just a reminder...
