Lessons Learned Running IBM Watson on Mesos
summary
Watson migrates ~40 microservices onto Mesos + Marathon and Docker, moving from ~50 nodes to ~1,000 in a year and running 25 services on Marathon. Solr needed local disks so the team uses SolrCloud mirroring and hits stateful pitfalls that require pinning workarounds. Outages include a 1 MB znode limit in ZooKeeper and leader split race conditions after network failures. Production stack includes Docker 1.7, Marathon 0.9, lacks auto scaling and persistent volumes, and relies on chaos testing and improved metrics.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 0:57China
- 1:15Jason
- 2:15What is Watson
- 3:41Brief History
- 4:07Commercialization
- 4:43Developer Portal
- 5:38Team
- 6:38Timeline
- 8:05Watson Developer Cloud
- 8:44Rank and Retrieve Service
- 11:17Marathon
- 13:08Moving containers
- 14:37Outages
- 16:55Limitations
- 18:14Mirroring
- 18:33Losing Contact
- 20:33Third Outage
- 22:36Lessons Learned
- 29:20Metrics
- 30:58Deployment Automation
- 32:41Chaos Testing
- 33:52Summary
related talks
53mHow We Built a Highly Scalable Machine Learning Platform Using Apache Mesos
43mLife of a startup...
48mA journey to micro-services on Mesos
53mDocker as the Building Block for Datacenter-Scale Applications
33mEnabling Microservices
42mLessons Learned: Deploying Microservices Software Product in Customer Environments
40mFrom Monolith to Docker Distributed Applications
38mHow We Almost Failed with Microservices