Systems @Scale 2024
2024-06-12
-- online --
[Systems @Scale Spring 2021, Systems @Scale Summer 2021, Systems @Scale Fall 2021, Systems @Scale Winter 2021, Systems @Scale Summer 2022, Systems @Scale Winter 2022, Systems @Scale Summer 2023, Systems @Scale 2024]
who it's for
infrastructure engineers, ml platform engineers, cluster operators, ai ops engineers, site reliability engineers, storage engineers, performance engineers
topics
DGX H100 SuperPOD reference design; Infiniband leaf-spine fabric; NVMe caching storage fabrics; Slurm cluster scheduling; liquid cooling via CDUs; distributed training at 10k+ GPUs; topology-aware scheduler design; checkpointing and adaptive IO scheduling; serverless training orchestration; autoscaling for LLM inference; sparse-dense MoE training pipelines; DeepSpeed and Megatron stacks
| title | author | length |
|---|---|---|
| Building at Scale with H100: Eos as a DGX SuperPOD Reference Model for Large Data Center Builds | Julie Bernauer | 19m |
Eos is a DGX SuperPOD reference design built from DGX H100 servers organized into self‑contained pods that scale to ~4,000 GPUs and achieved a top‑500 rank… | ||
| GenAI Training In Production: Software, Hardware & Network Considerations | KR Kishore, Adi Gangidi, Jenya Lee | 22m |
Meta builds two 24K clusters (one RoCE-optimized, one InfiniBand) and relies on PyTorch to move research to production, mapping tensor/pipeline/data… | ||
| Training LlaMa - A Storage Perspective | Sumit Gupta, Robin Battey | 18m |
GenAI checkpointing creates spikes that saturate storage in under 10 ms, with observed checkpoint growths of 60x–70x and an eight‑order‑magnitude range… | ||
| Maintaining Large Scale AI Capacity | Benjamin Leonhardi, Saranyan A Vigraham | 19m |
Meta describes moving from one research cluster to nearly a hundred AI superclusters and targeting 600,000 GPUs while running 1,000,000 operations/day. A… | ||
| Keynote | Surupa Biswas | 18m |
Claims that over 100,000 LLM projects exist and that clusters reach 24,000 GPUs set the scale then quantify failure dynamics as roughly 1.3 failures per hour… | ||
| MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs | Haibin Lin | 21m |
Production system trains LLMs across >10,000 GPUs using a distributed profiler plus device-level torch profiling, fine-grained intra-node trunking (8 trunks)… | ||
| Training Arctic | Jeff Rasley, Hyungtae Kim | 22m |
Arctic is a 480B hybrid sparse-dense MoE (10B dense + 3.6B expert params, 128 experts) trained in under three months across three stages on 3.5 trillion… | ||
| AI Training Orchestration Evolution with Serverless Building Blocks | Maneet Bansal, Upasana Dixit, Shawn Swyx Wang | 21m |
Meta describes replacing a monolithic workflow engine with composable serverless pieces that cut ML training friction: a decoupled Action Service launches CPU… | ||
| Evolving Cluster Management | Shankar Selvam, Cedric Goh | 15m |
Meta replaces ad hoc per-task tweaks with a new stateful control service called the Twine job control plane that accepts high-level intents from Conveyor and… | ||
| Scalable Solutions for Running Large Language Models | Jiaxin Cao | 14m |
Adaptive autoscaler chooses between collocation (single replica handles prefill+decode) and segmentation (separate prepare and decode replicas) using measured… | ||
| nothing matches | ||