Co-Designing Communication for AI Accelerators
summary
Meta describes a custom accelerator MTIA 300 with on-chip message engines and near memory compute that offload communications into HBM and avoid PE and host stalls. Measured overlap of compute+collectives reaches about 98% vs 70–80% on competitors, large-message throughput hits ~1.2 TB/s across 16 nodes using six Broadcom RDMA NICs per side, and small-message latency drops to ~6 µs. NIC banks are reconfigurable (default 10 scale-up + 2 scale-out) to shift between training and inference.
AI-written summary · about
chapters (10)
- 0:00Co-designing AI accelerators
- 1:49Co-design for efficient AI serving
- 3:14Overlapping compute and communication
- 5:21MTIA 300 hardware innovations
- 7:49Message offloading acceleration
- 10:59Latency-sensitive inference workloads
- 12:24Network flexibility for scaling
- 14:28Hardware roadmap future
- 15:53Co-design for full AI stack
- 16:36PyTorch-native systems
related talks
21mMTIA: Meta's First Generation of AI Accelerators
03mAccelerate with purpose
58mKeynote: Enabling the AI Revolution: Next-Generation Memory Solutions for AI Computing(…)
01h 55mNeural Networks and Accelerator Co-Design
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
19mAI/ML Networking Challenges: The Fast and the Finicky!
25mHardware & Co-Design
12mDemo: NVIDIA's use of Catapult HLS for Building ML Inference Accelerators