Enabling Multi-Language Programming in Data Engineering Workflows
summary
Python code reads clinical-trial metadata and auto-generate Snakemake workflows that orchestrate SDTM→ADaM→TFL runs, launching run SAS and R inside containers (Singularity/Docker) pulled from a local registry. Templates (Ginger) render snakefiles for hundreds of programs with explicit dependencies; execution happens in a GitLab CI pipeline with parallelism and timestamp-based reruns. Built artifacts live in Git, images include custom libs, visualization is minimal, and plans include dependency auto-detection and exploring Airflow for a dashboard.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:50Speaker background
- 3:13Clinical trials data context
- 6:05Multi-language challenges
- 8:17SnakeMake framework
- 12:35Docker and Singularity modes
- 15:24SnakeMake commands
- 16:26Wildcards and rule refactoring
- 18:55Current statistical setup
- 20:42Ginger metadata templating
- 23:33Solution benefits and limits
- 25:40Future exploration
- 27:29Q&A
related talks
17mR & Python playing nice, in Production
21mScaling and automating R workflows with Kubernetes and Airflow- 21mScaling and automating R workflows with Kubernetes and Airflow
34mData Products, From 0 to data science pipeline
27mDo the magic with DevOps, Open Source and All-remote teams (with Python, of course)
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
01h 02mDocker for Data Science: Reproducibility and Deployment
32mDeploying Python models to production