The Evolution of Databases at Dropbox

on YouTube ↗01h 00m

" summary

History of Dropbox’s database infrastructure evolution from a single global MySQL deployment to about 6,000 shards. Covers practical engineering: per-type configuration across hardware and MySQL versions, integration tests for all cluster operations, running multiple database instances on one host and why, and fast backup-and-restore procedures that complete in tens of minutes.

AI-written summary · about

" chapters (41)

1. 0:00Introduction and speaker
2. 0:49Dropbox architecture overview
3. 3:34Database growth and sharding
4. 5:19Introduction of Shtor graph system
5. 6:23Database reliability principles
6. 7:47Master-slave topology design
7. 9:29Server hardware and configuration
8. 10:32Problems with current shard topology
9. 11:57Multi-instance topology benefits
10. 12:38Discovery service evolution
11. 15:46Database operations automation
12. 16:50Promotions and failovers
13. 17:31Flower operation details
14. 19:00Cluster synchronization process
15. 21:27Backup importance and guarantees
16. 24:39Backup process and tools
17. 27:10Future backup plans
18. 29:15Automation and monitoring systems
19. 31:44Automatic recovery manager
20. 33:10Operation execution framework
21. 34:17Incident response and alerting
22. 35:23Monitoring and metrics collection
23. 37:09Testing strategies
24. 38:13Integration and unit testing
25. 39:17Staging environments and traffic replay
26. 40:22Disaster recovery testing
27. 43:34Automation and testing principles
28. 45:02Q&A on shard load balancing
29. 46:11Q&A on alerting system
30. 47:12Q&A on incident stress
31. 48:15Q&A on service failures
32. 49:00Q&A on master-master switching
33. 52:01Q&A on automation scripts
34. 52:42Q&A on testing infrastructure
35. 53:45Q&A on server memory efficiency
36. 54:26Q&A on data storage scale
37. 55:09Q&A on shard scaling limits
38. 55:32Q&A on shard splitting
39. 56:20Q&A on server failure statistics
40. 57:49Q&A on master lifespan
41. 58:10Q&A on file versioning

" related talks

• 27mDistributed MySQL Architectures: Past, Present and FuturePeter Zaitsev · Distributed SQL Summit 2019
• 57mThe Evolution of a MySQL Database SystemFrederic Descamps · DevClub.lv 114th Frontend focused event
• 23mStructured Data at Box: How We're Building for ScaleTamar Bercovici · @Scale 2014
• 54mThe Evolution of a MySQL Database, From a Single Instance to HA With Disaster RecoveryFrederic Descamps · Percona Live 2022
• 11mDatabase Evolution: Partitioning and IndexingPurushotham Jinka · Conf42 Cloud Native 2026
• 40mThe Mysteries of DropboxJohn Hughes · Lambda Days 2016
• 01h 01mPanel: Facebook's Distributed Database EvolutionKarthik Ranganathan, Dhruba Borthakur, Harrison Fisk, Jeff Rothschild, Victoria Dudin, Vishal Kathuria · Distributed SQL Summit 2019
• 25mData in the CloudMarten Mickos · O'Reilly MySQL Conference & Expo 2011