Practical Container Scheduling: Juggling Optimizations, Guarantees, and Trade-Offs at Netflix

on YouTube ↗39m

" summary

Netflix runs Mesos frameworks Titus and Mantis and builds an open‑source Fenzo scheduler to juggle bin‑packing, locality, capacity guarantees and elasticity. Fenzo applies pluginizable hard constraints (GPU, reserved hosts), weighted fitness functions and DRF across two tiers so critical services launch immediately while Flex scales to demand; cluster sizing uses per‑app declared SLAs and avoids peak provisioning despite 5x peak-to-trough variation. Operators get JSON failure diagnostics, and planned features include evictions for noisy neighbors and automated agent rollouts.

AI-written summary · about

" chapters (20)

1. 0:00Intro
2. 1:54Reactive stream processing: Mantis
3. 2:45Container deployment: Titus
4. 3:27What the cluster needs to support - Heterogeneous mix of workload
5. 4:35Why juggle at all?
6. 7:24Scheduling challenge in large clusters
7. 8:42Our initial goals for a cluster scheduler • Multi goal optimization for task placement . Cluster autoscaling • Extensibility
8. 9:53Multi goal task placement
9. 13:10Security
10. 13:29Capacity guarantees
11. 16:36Fenzo scheduling strategy
12. 19:48Fitness functions we use • CPU, memory, and network in packing
13. 21:36Hard constraints we use • GPU server matching
14. 22:02Soft constraints we use • Specified by individual jobs at submittime • Balance tasks of a job across availability zones
15. 22:58Mixing fitness with soft constraints
16. 23:48Our queues setup
17. 26:49Sizing agent clusters for capacity
18. 28:05Reasoning about allocation failures
19. 29:41What's next?
20. 32:31Questions?

" related talks

• 58mDistributed Scheduling with Apache Mesos in the CloudDiptanu Gon Choudhury · Philly ETE 2015
• 19mKeynote: Making and Keeping Netflix Highly AvailableKatharina Probst · MesosCon Europe 2017
• 35mSystems Scale Summer Q&A Week 1Michael Chang · Systems @Scale Summer 2021
• 21mHandling Traffic Spikes Through Automated Prioritized Load SheddingBenjamin Fedorka · gRPConf 2025
• 44mTechniques Netflix Uses to Weather Significant Demand ShiftsJoseph Lynch · SREcon25 Americas
• 46mIntro: Scheduling SIGDa Ma, Shivram Srivastava · KubeCon + CloudNativeCon Europe 2019
• 42mReliably shipping containers in a resource rich world using TitanDiptanu Gon Choudhary · DockerCon SF 2015
• 34mPatterns for building Highly Available Cluster SchedulersDiptanu Gon Choudhury · ScaleConf Colombia 2018