From Flying Circus to Holy Grail: Migrating the Twilio API from PHP to Python

on YouTube ↗38m

" summary

Twilio’s REST API migrates from a 500k‑line PHP monolith to a fleet of Python services behind a starship proxy (Flask + Flask‑RESTful + requests) that maps public routes to internal REST services. Starship implements auth, validation, rate limits, instrumentation (Datadog), and async I/O with gevent, and handles ~5,000 req/s. Migration proceeds endpoint‑by‑endpoint using nginx routing, the proxied resource pattern, unit/integration/cluster tests, response diffing, shadowing and canary rollouts to validate behavior without downtime.

AI-written summary · about

" chapters (29)

1. 0:00Start
2. 4:34Difficult deploys
3. 5:09Hard to understand
4. 5:36Operationally awful
5. 6:49What to do?
6. 7:08Refactor and replace!
7. 8:50Starship: API Federation (The Next Generation)
8. 10:32Proxied Resource
9. 11:43Benefits
10. 12:21Flexibility
11. 12:49Operational excellence
12. 15:10Planning
13. 15:56Start small
14. 16:41Architecture
15. 17:09We can solve any problem by introducing a layer of indirection
16. 18:18Keep it simple
17. 18:43Testing
18. 19:30Test in layers
19. 21:31Starship's Test Pyramid Unit tests: auth functions, transformations, serializations, etc. • Integration tests: mocked-out requests • Cluster tests: black box requests to live systems • Manual tests
20. 22:15Starship's Test Pyramid Unit tests: auth functions, transformations, serializations, etc. • Integration tests: mocked-out requests Cluster tests: black box requests to live systems • Manual tests
21. 22:39Reduce overhead: find common cases
22. 24:35Throwaway tests
23. 25:02Response diffing
24. 26:46Small Fast Simple
25. 27:31Shadow deployments
26. 28:56Canaries
27. 29:26Batch
28. 30:09Cheat
29. 38:02NEXT DAY VIDEO

" related talks

• 21mThe great SOA migrationCharles-Axel Dein · DevFest Nantes 2017
• 27mHow We Migrated 3.8 Million Lines of Python 2 Without Interrupting Deve...Benjamin Bariteau · PyCon US 2022
• 39mA scalable blueprint for product customization with Python 3Chinab Chugh, Ikhsan Rosyidin · PyCon APAC 2018
• 36mA Python TakeoverMark Ramm-Christensen · EuroPython 2011
• 36mA Python TakeoverMark Ramm-Christensen · EuroPython 2011
• 44mPython in the MicroServices worldHari Krishnan · PyCon Singapore 2016
• 31mFighting the Good Fight: Python 3 in your organizationJason Fried · PyCon 2018
• 30mHow To Build a Python Microservice Without Losing a JobAnton Caceres · EuroPython 2019