Hot reloading Python web-servers at scale
summary
Instagram implements uWSGI master-fork to spawn a new master that execs updated code and then handsover workers using cheaper mode and controlled FIFO commands, adding batching and retry logic to avoid dropped requests and handle thunderlock blocking. Fixes include multi-minus FIFO handling, re-exec to refresh LD_LIBRARY_PATH, and fallbacks when many masters exist. Result: no capacity loss during pushes, much faster rollouts compared to 15–20s restarts, and about 70 deployments/day with peaks of 120.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:36Continuous Deployment
- 3:37Problems with Continuous Deployment
- 6:13Can we do better
- 14:26Master
- 17:45Cheap mode
- 18:45Slow deployment
- 19:43Patching
- 23:42Reliability
- 25:44Environment variable
- 26:59End result
related talks
58muWSGI, the Swiss Army (MacGyver) knife of deployment
21mHigh-Availability Django
31mConfiguring uWSGI for Production: The defaults are all wrong
42mFree Threaded Django with Micah Lyle
39mHigh Performance Django: From Runserver to Reddit Hugs
36mUnicorn Unix Magic Tricks
47mBuilding Scalable Web Apps
41mDjango Deployments Done Right