AI Infra @Scale 2024
who it's for
model engineers, ml platform engineers, hardware engineers, PyTorch developers, infrastructure engineers, site reliability engineers, research scientists
topics
Llama 3 405B training; 128k tokenizer workflows; synthetic-data rejection sampling; GPU fleet scaling to 600000 GPUs; MTIA V2 custom silicon; torch Titan 3D parallelism; adapter fine-tuning on consumer GPUs; mixture-of-experts at 3000 GPUs; low-bit quantization for inference; 10x faster checkpointing techniques; endpoint deployment cost math; networking and cooling for hyperscale clusters
| title | author | length |
|---|---|---|
| Bringing Llama 3 to Life | Joe Spisak, Delia David, Kaushik Veeraraghavan, Ye (Charlotte) Qi | 48m |
Meta releases an open-weight Llama 3 family including a 405B / 45B model trained on about 15 trillion tokens using 16k H100 GPUs, with the 45B comparing well… | ||
| Keynote | Aparna Ramani | 17m |
A tour of AI infrastructure stresses concrete numbers and engineering problems: Meta scales to 600,000 GPUs in its fleet, Llama 3.1 trains on 15 trillion… | ||
| How Pytorch Powers Training Inference | Wanchao Liang, Kimish Patel, Evan Smothers | 23m |
Shows how PyTorch stacks combine concrete techniques and numbers: torch Titan composes 3D parallelism, activation recomputation and TorchCompile to run model… | ||
| Hardware & Co-Design | Joel Coburn, Junqiang Lan, Jack Montgomery | 25m |
MTIA V2 is a TSMC 5nm, 90 W chip running at 1.35 GHz with 177 TFLOPS (FP16/bf16) and a 2.35 billion gate design, built around a very large on‑chip SRAM and… | ||
| Open Innovation: Unlocking AI's Potential | Hagay Lupesko | 17m |
Argues that open innovation drives faster progress in models and systems, citing an actual Mixture-of-Experts model — dbrx mixture-of-experts trained with… | ||
| Opening Remarks | Damien Sereni | 04m |
Quick overview of the program and concrete takeaways: a recent release Llama 3 405B is explained end to end with specifics on the training data, optimization… | ||
| Industry Panel | Chris Lattner, Michael Suo | 24m |
Panelists argue concrete shifts: PyTorch democratized AI by making training ubiquitous while model deployment is a dark art that still blocks smaller teams… | ||
| nothing matches | ||