Python at Massive Scale

on YouTube ↗45m

" summary

Athena runs a mono repo of 35m LOC/150k modules served from a custom Hydra object DB with import hooks and continuous deployment, supporting about 1,600 active committers and 60k commits/month. A 5 GB binary Python distro (rolled weekly as prod/prod_new/prod_old) and staging stacks produce fast rollouts and rollbacks; the platform powers thousands of developers and trading systems reliably. Perspective visualization is open sourced (browser-side aggregation via WebAssembly). Migration off Python 2.7 is urgent because of pickled objects and many replicas.

AI-written summary · about

" chapters (19)

1. 0:00Introduction
2. 1:10JP Morgan’s Python scale
3. 2:35Athena trading platform
4. 3:41Athena’s early development
5. 5:49Python stack choices
6. 7:56Mono repo lifecycle
7. 8:59Athena’s growth
8. 10:48Data science expansion
9. 11:56Managing massive codebase
10. 13:21Developer productivity
11. 18:48Python code deployment
12. 19:32Python version management
13. 22:48Hydra database deployment
14. 25:16Database unions testing
15. 28:52Tooling and culture
16. 31:23Athena Research
17. 33:08Athena platform summary
18. 33:51Perspective visualization engine
19. 36:01Q&A

" related talks

• 36mA Python TakeoverMark Ramm-Christensen · EuroPython 2011
• 40mScaling Python - Bank EditionAndrew Fulton, Adam Lewis · PyData NYC 2022
• 30mEmpowered by Python - A success storyJens Nie, Peer Wagner · PyCon.DE 2017
• 36mA Python TakeoverMark Ramm-Christensen · EuroPython 2011
• 42mBuilding a hosting platform with PythonAndrew Godwin · EuroPython 2011
• 42mBuilding a hosting platform with PythonAndrew Godwin · EuroPython 2011
• 24mBuilding a Sustainable Python Package IndexDustin Ingram · PyCon AU 2019
• 30mAn Admin's Cornucopia - Python Is More Than Just A Better BashChristian Theune · PyCon.DE 2017