Reliable Data Processing with Minimal Toil
summary
Google and Slack implement automated global rollouts that promote batch-job versions through staged shards sliced by user id and traffic, with read-write staging and multiphase canaries (start at one percent then 19% etc.) to catch correctness and freshness regressions. Validation includes simple counters, file-size and row-count checks, runtime and resource spikes, and model-output checks; promotion is automatic when metrics stay within thresholds, markedly reducing manual rollouts and rollback pain during millions-of-user Workspace scans.
AI-written summary · about
chapters (11)
- 0:00Welkom en inleiding tot betrouwbare data processing
- 1:40Batch processing in Google Workspace
- 4:26Importance of reliable fresh results
- 8:01Correctness checks in batch jobs
- 10:07Graduele uitrol met veiligheidsniveaus
- 13:37Canary deployments for reliability
- 16:29Stabiliteit meten en valideren
- 19:19Kanarievalidatie voor user-facing services
- 21:47Splitting data for canary testing
- 23:31Freshness tracking en kritieke jobs
- 24:57Summary of the approach
related talks
25mOptimize your Data Pipeline without Rewriting it
01h 00mReliability and speed by 'push to main' mentality
33mMeasuring DevOps
11mRunning Jobs at Scale
27mDesigning Your Pipelines for Scalability and Resiliency
29mMaking tech boring to keep data exciting
45mResilient Predictive Data Pipelines
24mRollout on Reception: Progressive Delivery for GenAI Apps Using Real...