Co-Designing Communication for AI Accelerators
summary
MTIA 300 integrates message engines and near-memory compute to offload communication from compute cores, running RDMA via Broadcom NIC banks (up to 1.2 TB/s aggregate, typically 1 TB/s enabled). Benchmarks show about 98% simultaneous compute+comm efficiency versus ~70–80% on H100s, and single-message latency drops below 6 µs when PEs inject RDMA packets. Default network mix is 10 NICs for scale-up and 2 for scale-out, and the team targets roughly two silicon generations per year.
AI-written summary · about
chapters (12)
- 0:00Core design for AI accelerators
- 1:54Efficient AI serving
- 3:18Host-driven bottlenecks
- 4:22MTIA300 innovations
- 7:09Message offloading
- 9:58Overlapping compute and comms
- 10:40Latency for small messages
- 12:05Scale-up vs scale-out trade-offs
- 13:50Adapting network config
- 14:31Hardware roadmap
- 15:55Core design future
- 17:19Closing
related talks
19mAI/ML Networking Challenges: The Fast and the Finicky!
21mMTIA: Meta's First Generation of AI Accelerators
58mKeynote: Enabling the AI Revolution: Next-Generation Memory Solutions for AI Computing(…)
25mHardware & Co-Design
47mDesigning High-Performance Scalable Middleware for HPC, AI, & Data Science in Exascale Systems & ...
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
03mAccelerate with purpose
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights