Engineers to Data-Scientists with Python

on YouTube ↗27m

" summary

A MATLAB monolith with 50,000 lines and zero tests migrates incrementally to Python using Docker, AWS Batch/ECS, S3 and Airflow while calling between MATLAB and Python via a wrapped serialization layer; an initial one‑month refactor named Opportunity October breaks production but leads to formalized heuristic tests with pytest, parameterized fixtures, and faster iteration. Half the analysis pipeline ends up 50% rewritten, production bug rate falls ~30%, feature velocity temporarily slows, and training uses pairing, code reviews and lunch‑and‑learns.

AI-written summary · about

" chapters (40)

1. 0:00Introduction
2. 0:31What will I talk about
3. 1:12About me
4. 1:34What we do
5. 2:14Technical context
6. 2:37Deployment
7. 2:52Business Context
8. 3:14Analysis Pipeline
9. 3:33Reliability Analysis
10. 4:09Advantages of Python
11. 4:47Risks of Python
12. 5:49Project Pigeon Rat
13. 6:10Preparing for Python
14. 6:33Separating Core Analysis from Abstract Data
15. 6:52Pure Functions
16. 7:20Abstracting Data
17. 7:35Data Access Layer
18. 7:48Christmas Eve
19. 8:05Why not just roll back
20. 8:33Postmortem
21. 9:40Heuristic Tests
22. 10:20Inversion Speed
23. 12:23Managing Configuration
24. 13:34Global State
25. 14:02Retro
26. 14:38Empathy
27. 15:38Reframing
28. 16:49Moving to Python
29. 17:53Python rewrites
30. 18:41Pain points
31. 20:34Whats a mechanical engineer
32. 20:49Whats a data scientist
33. 21:17Being an effective data scientist
34. 22:01Traditional approach
35. 22:35How do I make one
36. 23:15Whats separating engineers and data scientists
37. 24:18Our process
38. 25:11Some thoughts
39. 25:52Key bits
40. 26:23Wrap up

" related talks

• 30mEmpowered by Python - A success storyJens Nie, Peer Wagner · PyCon.DE 2017
• 46mThe Python Data Ecosystem: Navigating a fragmented landscapeUmare & Tong · PyData Seattle 2023
• 40mScaling Python - Bank EditionAndrew Fulton, Adam Lewis · PyData NYC 2022
• 39mHow Python Drives Modern Data EngineeringKiana Alessandra Villaera · PyCon Philippines 2023
• 26mCLI, CI, DIY: Crafting Python Based Dev Tools in a DevOps VacuumLiubov Burtseva, Uri Drori · PyCon Israel 2024
• 32mNot Invented Here: Porting Scientific Software to PythonAndrew Walker · PyCon Australia 2015
• 25mA microservices architecture with Python and Docker containersDiego Arenas · Docker Community All Hands #5 (March 2022)
• 32mNot Invented Here: Porting Scientific Software to PythonAndrew Walker · PyCon Australia 2015 Science/Data Miniconf