High Availability at Braintree
summary
Braintree handles roughly $10B in processing (~19,000 transactions per minute) and aims for four to five nines by using concrete techniques: switching MySQL to postgres concurrent index for millisecond DDL and transactional migrations, running pre/post migrations with a deleted_columns shim, pausing traffic via a redis queue broxy and dispatchers to avoid failed requests, using LVS/ipvs with BigBrother and LitmusPaper plus pacemaker for failover, Mallory proxies for ISP-specific processor routing, and heavy automation with Puppet and Capistrano.
AI-written summary · about
chapters (18)
- 0:00Intro
- 3:09Ruby on Rails
- 5:43Refined deploy process
- 6:46Rails caches columns
- 8:15We run multiple versions of the code at once
- 9:51Limitations
- 12:30Lightweight rack adapter
- 16:53Build our own
- 19:42Stateful services
- 21:37Networking - inbound
- 26:55Processor proxies
- 27:27Connection failures
- 28:40Summary - unplanned failures
- 29:05People make mistakes
- 30:35Reduces errors
- 32:03Safeguards
- 33:34Monitoring Nagios
- 35:40Summary - people Automation
related talks
46mManaging a Billion Transactions with Tech, Tactics, and Teamwork
48mFault Tolerance on the Cheap Making Systems That Probably Won't Fall Over
55mFrom the engine to the bridge - building systems that scale
31mBeyond the Code
43mJust Use Postgres.
29mScaling Rails to 2M MySQL requests per second
01h 03mMoving Fast - without crashing
33mHerding Elephants