Monitoring MySQL at scale
summary
Lithium uses Datadog agents and performance_schema to collect and correlate ~350 MySQL metrics (about 460 metrics per host including OS and CloudWatch), added 150+ metrics for 5.7, and integrates MHA helper events so failovers show up as timeline events. Hosts are ephemeral—containers last ~3 days, VMs ~12—so monitoring is agent-based and service‑centric. Root causes include optimizer changes that drop indexes (detected as spikes in handler_read_next and per‑schema latency); fixing an index immediately restores throughput, proving the approach.
AI-written summary · about
chapters (14)
- 0:00Introduction and background
- 1:12Scaling MySQL infrastructure
- 3:53DevOps principles and culture
- 7:26Monitoring-first strategies
- 10:54Adapting to dynamic environments
- 12:55Tools and monitoring challenges
- 15:41Moving to a service-centric model
- 19:51Transitioning from legacy systems
- 24:25Categorizing metrics
- 27:25Alerting and diagnosis workflow
- 29:35Deep dive into MySQL metrics
- 34:25Dashboard design examples
- 37:32Case study in troubleshooting
- 41:04Conclusion and resources
related talks
35mBest Practices for MySQL Performance
24mMySQL Performance Schema
49mMySQL monitoring and debugging: maximum insight with minimal cost
53mMySQL DevOps
58mBetter MySQL With Percona PMM
19mMariaDB Observability
43mMySQL Like We Don't Know It
40mImproving Observability, Reliability, and Security of Relational Database...