From DevOps to DataOps - Cloud based Software Development
summary
Research groups report that ~80% of data projects never deploy and up to 90% of effort is data preparation, so the work builds a cloud-focused pipeline and curriculum that moves notebooks into production. Experiments run on Amazon and Azure show SageMaker full-stack training, tuning and deployment and Azure ML/DS VMs integrate development with CI/CD. The team codifies practices in a DevOpsSE BoK extended for DataOps/MLOps, separates training/validation/production data to prevent model drift, and targets OpenStack for education and SRE-driven optimisation.
AI-written summary · about
chapters (8)
- 0:00Introduction to DataOps
- 2:12Research infrastructure challenges
- 6:03Data-driven research cycles
- 9:38Data science vs software engineering
- 12:07DataOps and DevOps integration
- 14:40Cloud platforms for DataOps
- 16:43Education and training in DataOps
- 17:47Q&A and future plans
related talks
29mDevOps for Data Science? - automate the boring stuff and leverage the OSS ecosystem
59mDevOps for Data Science
43mMachine Learning Ops: DevOps for Data Science
23mOpen Source DataOps and MLOps Strategies
44mDevOps for Machine Learning - why is it different?
36mDataOps and MLOps: NLP Development and Deployment in Kubernetes made easy
57mDevOps for Machine Learning
17mProductionizing Data Science Workloads