Distributed Scheduling with Apache Mesos in the Cloud
summary
Titan is a Mesos-based global resource scheduler that Netflix built to cut deployment latency (from ~15 minutes toward sub‑minute), using a custom Fenzo scheduler for bin‑packing and workload-aware placement, Docker for packaging, ZFS on Linux for persistent volumes and live cloning, and predictive plus reactive autoscaling across AWS ASGs. Job specs and containers replicate to multiple regions for active‑active multi‑region failover; several internal workflow systems already run on it in production.
AI-written summary · about
chapters (28)
- 0:00Start
- 0:24Who am I?
- 0:52A word about Netflix
- 1:57The Titan Framework
- 2:31Guiding Principles
- 7:08Active-Active Architecture
- 7:55Current Deployment Pipeline
- 9:20The Base AMI
- 11:03Need for a Distributed Scheduler
- 15:32Why did we write Titan
- 21:13Compute Resources as a service
- 22:49Things Titan doesn't solve
- 24:09Building blocks
- 24:45Resource Allocator
- 25:50How we use Mesos
- 28:25Packaging and Isolation
- 30:06Docker
- 31:28Volume Management
- 33:12Networking
- 35:15Logging
- 36:29Monitoring
- 43:39Reactive Auto Scaling
- 45:04Predictive Autoscaling
- 45:30Bin Packing for efficient Autoscaling
- 46:58Mesos Framework
- 48:59Globally Distributed
- 53:11Future
- 53:27Questions?
related talks
42mReliably shipping containers in a resource rich world using Titan
39mPractical Container Scheduling: Juggling Optimizations, Guarantees, and Trade-Offs at Netflix
19mKeynote: Making and Keeping Netflix Highly Available
27mKeynote Panel: Future of Cluster Management
41mIntroducing Flink on Mesos
40mNomad: Scheduling Applications at Scale
45mMesos, Then and Now
01h 01mTaming the Modern Public and Private Clouds with Nomad