DevOps For Data Teams
summary
Says that data scientists spend 80% of their time on collection and cleaning and that adopting DevOps practices reduces that overhead; shows concrete tooling and patterns — Airflow, Great Expectations, Kafka, CI/CD, feature flags, containers and Kubernetes — and gives examples of canary deployments, change data capture, ELT into an S3-backed data lake and one-team product-modeling for internal consumers; demos and postmortems describe what was built, what scaled and what failed.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:05What is data engineering?
- 3:56Role conflicts
- 5:19Tech conflicts
- 7:47Similarities
- 10:20Technical challenges
- 13:23Tools and tech we can borrow
- 19:56Lessons we can learn
related talks
59mDevOps for Data Science
28mDevOps in a Data Warehouse: Inside Out
29mDevOps for Data Science? - automate the boring stuff and leverage the OSS ecosystem
51mFrom DevOps for data to DataOps
26mApplying DevOps practices in Data and ML Engineering
30mDataOps: Much More Than DevOps for Data
30mDevOps to DataOps
22mDevOps in Data Science: What Works and What Doesn’t