Why Great Models Fail: Lessons From 9 Years of Deploying ML Models
summary
After nine years deploying ML models, the main claim is simple: high test metrics alone don’t get models into production. An NLP proof of concept builds topic models on archived social media in Databricks, scores well on topic coherence, but never scales because of minimal stakeholder consultation and an API pricing change that makes data acquisition unaffordable. The model sits in model purgatory—working technically but failing to deliver business value or justify maintenance.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 4:25Social media analysis failure
- 10:52Why great models fail
- 13:23Project scoping fundamentals
- 18:04Stakeholders and success
- 25:10Project definition mistakes
- 26:54Deployment constraints and risks
- 34:18Brainstorming solutions and building minimal viable products
- 40:02Maintenance and monitoring
- 49:17Model drift strategies
- 57:11Key takeaways
related talks
08mConsiderations for producing the best AI and Machine Learning models
21mWhy is Production ML so Hard?
25mWhy Good Models Fail After Deployment
42mDeploying ML models and all the things that go wrong
21mHow The Application Of Machine Learning Is Changing The Way Companies Use Data
27m10 Things That Can Go Wrong with ML Projects
48mPrinciples and Practices of Building Machine Learning Systems
56mData, AI, Hype and Fear - Recipe for Disaster?