In house to open source Stitching the past to the futu...
summary
A legacy orchestration stack that writes about 50 million data points and must scale to billions gets split: scheduler and orchestration engine are replaced with Apache Airflow after evaluating NiFi and Argo. A lightweight Python shim called pi hero invokes existing job binaries as subprocesses, implements a mailx email plugin, uses pdb for live debugging, pytest for unit tests and virtualenv for isolated deployment. A pandas-based reconciliation framework verifies that old and new outputs match, so the migration succeeds without a big bang.
AI-written summary · about
chapters (33)
- 0:00Intro
- 0:36Agenda
- 1:00Legacy code
- 1:21Advantages of open source
- 2:24Things to keep in mind
- 3:04Case study
- 3:29Process
- 4:16Three Phase Approach
- 4:46Use Case
- 5:21MustHas
- 6:03Research
- 6:47Evaluation criteria
- 7:41Classification
- 8:36Apache Airflow
- 9:10Complex Framework
- 9:43Orchestration Engine
- 13:42Writing production quality code quickly
- 13:52Awareness when something goes wrong
- 14:24Group decorator
- 15:16Bug example
- 16:17Break point
- 16:43Configpy
- 17:34Advantages of Python
- 18:21Customization in Python
- 19:05Configuration modification
- 19:26Integration
- 20:04How Python makes it easy
- 21:31Data reconciliation
- 23:01Recap
- 23:25Integration with existing code
- 24:27Conclusion
- 24:50Bloomberg Memory Profiler
- 25:13Thank you
related talks
27mModern ETL-ing with Python and Airflow (and Spark)
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
34mData Ingestion in Python using Apache Airflow
38mUse Apache-Airflow to Build Data Workflows
32mBig Data With Open Source Solutions
45mDefining data pipelines
32mBuilding Analytics Workflow using Airflow and Spark
01h 25mApache Airflow in the Cloud: Programmatically orchestrating workloads w/ Py