Deploy & Monitor ML Pipelines with Python, Docker and GitHub Actions
summary
Shows a reproducible pipeline that pulls New York electricity data for 11 subregions, uses Python inside Docker images orchestrated by GitHub Actions + Docker + Python, runs model selection with backtesting (about 20 backtest partitions per series), saves CSV outputs, commits results, and renders a simple dashboard deployed to GitHub Pages. Includes pytest unit checks, cron scheduling (hourly), devcontainer development, image-size pruning, and notes limits (no GPUs, public-repo secret risks). The pipeline works end-to-end with automatic refreshes and logged metadata.
AI-written summary · about
chapters (13)
- 0:00ML pipeline deployment intro
- 6:35GitHub Actions and Docker
- 11:57Use cases for GitHub Actions and Docker
- 15:59ML pipeline architecture design
- 24:25Data pipeline environment setup
- 30:29Data refresh function
- 34:52ML model refresh component
- 37:22Monitoring dashboard deployment
- 42:09GitHub Actions workflow config
- 51:10Development environment setup
- 54:48Pipeline prototyping
- 1:03:46Running and monitoring pipeline
- 1:09:23Conclusion
related talks
45mWorkshop: Deploy and Monitor ML Pipelines with Open Source and Free Applications
23mGitHub Actions & Python
05mLightning Talk: Build a Reproducible ML Workflow with Kubeflow Pipelines
22mBuilding a Docker Image Pipeline Using GitHub Actions
33mDeploying and Scaling Spark ML and Tensorflow AI Models
39mEnd to End Machine learning pipelines for Python driven organizations
29mML Model Traceability and Reproducibility by Design
24mAutomating ML Pipelines With Kubernetes And Airflow