Gang Scheduling for Llama
summary
Llama 4 uses a MoE design with >2 trillion parameters but only activates ~17B for Maverick, forcing distributed inference. Twine implements gang scheduling to place tightly coordinated containers with collocation on the same backend network and MSB boundary, enforce shared fate (gang-level restarts and autoscale by whole gangs), and provide discovery/routing via SMC. Context parallelism splits 1M–10M contexts across many hosts to cut prefill latency; expert parallelism yields ~2× decode throughput. Twine’s gang model is deployed in production for Llama 4.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:52Why distributed inference matters
- 3:20Llama 4 models and MoE
- 5:54System bottlenecks in LLM
- 8:45Disaggregate prefill and decode
- 10:33Context parallelism for long inputs
- 11:38Expert parallelism for decode
- 13:44Combining parallelism methods
- 14:26Gang scheduling for inference
- 16:31Collocation and failure domains
- 18:20Shared fate and gang lifecycle
- 20:07Discovery and routing for inference
- 21:12Deploying with gang scheduling
- 22:16Complex gang scheduling example
- 23:01Conclusion and future challenges
related talks
39mBeyond Model Sharding: Atomic Scheduling and Disaggregated LLM Serving With ...
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
32mDynamic Scheduling for Large Language Model Serving
32mScaling Generative AI Inference with llm-d
22mCustom Pipeline Parallelism for Frontier-Scale Model Serving
23mNew Pattern for Sailing Multi-host LLM Inference
33mllm-d: Kubernetes Native Distributed Inferencing
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d