Scaling and automating R workflows with Kubernetes and Airflow
summary
A platform at UK Health Security Agency deploys R and Python tasks to HPC by packaging code in Docker container images, scheduling them with Airflow DAGs that launch a new pod per task, and isolating teams with Kubernetes namespaces on OpenShift. It replaces cron-on-laptops with git-driven CI/CD, uses rocker images and RStudio Package Manager for binaries, scales from development laptops to cluster nodes to process millions of COVID records, and reduces manual runs while centralizing logs and retries.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:23Farm analogy
- 1:40Who am I
- 3:30Agenda
- 4:02Context
- 7:31Where we came from
- 8:34What is Kubernetes
- 12:53What is Airflow
- 15:32How have we used these together
- 18:30How to get started
related talks
21mScaling and automating R workflows with Kubernetes and Airflow
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
24mAutomating ML Pipelines With Kubernetes And Airflow
36mBuilding a Scalable Data Science & Machine Learning cloud using Kubernetes
14mRobust R Deployments: Building a Pipeline from RStudio to Production
16mRevolutionizing Healthcare Data Processing with Kubernetes
08mProcessing Robot Data at Scale with R and Kubernetes
39mSell cron, buy Airflow: Modern data pipelines in finance