Building a ML Model Observability pipeline
summary
Claim: a production-ready observability pipeline is implemented using an open source Python stack and it actually scales. It profiles inference logs with y logs, serializes profiles to binaries stored in Hive, merges hourly profiles into daily aggregates that equal full reprocessing, and runs distributed transforms with Fugue. Anomaly detection uses stats forecast (MSTL, fitted in‑sample) and confidence bounds; SHAP explains root causes. Demo processes ~300k rows on a laptop Docker in seconds to minutes, so the approach is practical and repeatable.
AI-written summary · about
chapters (11)
- 0:00ML model observability intro
- 2:30Anomaly detection at Lyft
- 9:02Y Logs for inference profiling
- 17:57Feature engineering for logs
- 21:48Profiling logs with Y Logs
- 34:06Merging profiles at scale
- 39:50Anomaly detection with forecasting
- 51:34StatsForecast for forecasts
- 1:05:23Feature impact on predictions
- 1:15:53Key learnings
- 1:20:40Scaling with Fugue and Spark
related talks
10mML-based Log Analysis for Faster Debugging
18mDesigning a scalable ML Ops Pipeline: Insights and best practices
33mBuilding Resilient (ML) Pipelines for MLOps
18mBeyond Inference: Bringing ML into Production
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
18mAI Observability: How To Fix Issues With Your ML Model
25mBuilding Scalable MLOps Pipelines That Actually Work
53mExperience with an end-to-end MLOps pipeline: Practical insights and 3D architectural perspectives