Automating Data Analytics: Collect, Transform and Deploy - Lead With Change
summary
Claims full automation across the pipeline from pre-code checks to release management and monitoring, implemented with GitHub Actions, Snowflake, Airflow on Azure and generative AI for test and code generation. One small app runs more than 1,000 tests automated, dry runs replace risky pushes, incidents resolve in under 20 minutes, and the effort reductions amount to many saved person‑days, faster queries, fewer errors and minutes‑not‑hours maintenance.
AI-written summary · about
chapters (21)
- 0:00Automation in data engineering
- 1:10Current state of automation in data ops
- 4:05Data ops beyond profession
- 6:17Efficiency in data management
- 7:22Evolution of data engineering
- 9:32DevOps in data engineering
- 11:20CI in data pipelines
- 12:49Testing automated data solutions
- 14:16Orchestration challenges
- 15:42Config management for apps
- 16:49AI for review automation
- 17:32Planning in development
- 18:38Development to deployment
- 20:27Deployment vs release
- 22:36Monitoring production
- 24:01Data warehouse performance
- 25:27Scaling storage strategies
- 27:33AI in analytics
- 29:04Feedback in automation
- 30:30Automation impact results
- 32:59Closing remarks Q&A
related talks
01h 31mAccelerate Business Delivery With AI and DevOps
20mAutomatise everyday work
26mAccelerating DevOps with AI
38mKeeping pace with data infrastructure so it doesn't outrun you
27mEmpowering Innovation through DevOps & AI Integration
26mAutomate ALL the Things
30mA Devops Approach to Data: A Tale Through the Trenches in Search of a Cloud Data Revolution
31mExecuting An Effective Data Strategy: Why Automation Is No Longer A Choice