Spotify: Horizontal scalability for great success
summary
Spotify runs a Python-heavy backend for over 10 million users and 15 million songs, using many small services, one process per core, and per-service DNS SRV records to distribute load. Playlist storage moves toward Cassandra for eventual consistency with configurable quorum writes, while relational databases handle global uniqueness and transactions. Operational tools include a homegrown supervisor, memcached and page cache tuning, and a DNS SRV-based DHT for key assignment and replica placement.
AI-written summary · about
chapters (7)
- 0:00Spotify and scalability
- 7:26Vertical vs horizontal trade-offs
- 12:42Scalable microservices design
- 23:05Supervision tool for services
- 24:32DNS load balancing & discovery
- 27:41Hash tables for data partitioning
- 33:26Q&A
related talks
35mDiversity and Data: Cases in the Music Industry
38mScale out from the very beginning
22mIt’s all about the db - a journey to accommodate 100k events per second
25mDeploying Python at Scale with Dask
29mBowerbirds of Technology: Architecture and Operations at Less-Than-Facebook Scale
33mLessons learned building Python microservices
25mA microservices architecture with Python and Docker containers
45mPython at Massive Scale