Validating Big Data Pipelines & ML (w Spark & Beam)
summary
Focus on concrete checks: run Spark & Beam examples that emit counters and compare counts or percentages against historical baselines; flag if input parse failures or unmatched records exceed a relative threshold (example: relative rules (e.g. 10% threshold)). Keep previous outputs for diffs, run a side validation pipeline instead of blocking main processing, and use property-based checks where feasible. For ML, monitor fixed test-set drift, convergence iterations and model weight deltas, and automate rollout with metric-driven rollback.
AI-written summary · about
chapters (40)
- 0:00Intro
- 0:32About Holden
- 2:04H1B Visa
- 2:57Outline
- 5:06Why should we test
- 5:35Why dont we validate
- 7:49Testing is hard
- 8:47We need better tools
- 9:48What happens when we dont
- 12:32Breakfast Sandwich
- 12:43Tests
- 13:50Property Checks
- 15:34Validation
- 16:57Parse Errors
- 18:03Empty Model
- 18:35Pager
- 19:00Writing Code
- 19:44Counters
- 20:28Processing
- 21:58Counters are unreliable
- 22:52Multiple processing
- 23:38Checking execution time
- 25:00Propertybased testing
- 25:35Recap
- 26:12Relative Rules
- 26:52Spark Metrics
- 27:44Alligator
- 27:58Match Words
- 29:55Software Changes
- 30:57ML Validation
- 32:12ML Deployment
- 32:39Human Cornered
- 33:43New Possible Practice
- 35:27Production Monitoring
- 36:00Extra Considerations
- 37:21CrossValidation Performance
- 37:48Conclusion
- 38:08Ending Notes
- 38:27Related Resources
- 38:59Distributed Computing for Kids
related talks
45mValidating Big Data Jobs - Stopping Failures before Production
42mDeciphering the way data is tested
30mHow to Properly Test ML Models & Data
39mModern Data Pipelines Testing Techniques: A Visual Guide
20mShift-left testing : Building reliable Data Pipelines
20mBuilding a Data Pipeline with Testing in Mind
25mTesting Data Pipelines
30mHow to Test BigData/ETL Solutions?