Venmo's Experiences With Aurora Failovers Using ProxySQL
summary
Aurora clusters route through RDS Proxy for day-to-day ops and spin up a ProxySQL fleet via CloudFormation + Auto Scaling, Ansible and Secrets Manager for maintenance failovers. A Python failover script swaps ProxySQL host groups, validates replication, flips clusters to read‑only, and can roll back; observed downtime is typically 5–15 seconds. Recipes cover per-service cutovers, blackhole-table migrations, and automation so failovers become Infrastructure as Code driven with ProxySQL controlling traffic and safety checks.
AI-written summary · about
chapters (13)
- 0:00Start
- 4:07Proxy Options for Aurora MySQL
- 6:36RDS Proxy
- 12:38CloudFormation ProxySQL Stack
- 15:27Before Failover
- 15:42During Failover (Maintenance Window)
- 16:32Recipes for painless failover
- 20:443 Ps of a painless failover
- 29:01Failover is a balance between art and science
- 31:31Data Decoupling for microservices
- 34:18Before Database Decoupling
- 37:15Post Failover Set Up
- 38:34Typical Cluster Set Up
related talks
34mScaling Venmo for High Volume Events
55mProduction with ProxySQL and Orchestrator
37mScale MySQL beyond limits with ProxySQL
55mAmazon Aurora: A deep-dive into what's new
27mThe State of ProxySQL, 2020 Edition
39mAWS Aurora
35mAmazon Aurora DSQL: Scalable, Distributed Relational Databases
54mBreakout Session