Scaling up to Big Data Devops for Data Science
summary
Claims that spinning analytic clusters is doable with scripts but the last 10% of configuration breaks most flows and requires system work; demonstrates using Amazon EMR or raw EC2 plus Ansible playbooks or bootstrap shell scripts to install R, RStudio, Anaconda, Spark and Hadoop; shows YAML and CLI glue, gives a reproducible pipeline that mostly works but still needs debugging and occasional manual fixes when packages or permissions fail.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:27Big data tools on cloud
- 7:29DevOps for data science
- 12:23Data science setup
- 17:17Ansible cluster config
- 21:30R and Python integration
- 26:24Infrastructure automation
- 33:33Debugging automation goals
- 34:36DevOps clarified
- 35:19Q&A
related talks
44mDeveloping for Big Data at the Intersection of Containerization and Infrastructure as Code
30mDevOps For Data Teams
39mFrom Data Science to Production - deploy, scale, enjoy!
28mDevOps in a Data Warehouse: Inside Out
34mKick-Ass Tools and Processes for Scalable Data Science
26mApplying DevOps practices in Data and ML Engineering
23mJust enough DevOps for data scientists.
36mMLOps from the edge to on demand visualization