Leveraging PyTorch for Generative AI in Distributed Edge Clouds
summary
PyTorch Edge and SQTorch enable exporting PyTorch 2.x models to compact on-device runtimes and use compiler-backed backends for Metal and NNAPI while combining INT8 quantization, pruning and knowledge distillation to shrink models. Models partition across many low-power devices with pipeline parallelization and a central coordinator; an NDIM case study reports memory per device falling significantly and throughput scaling nearly linearly as devices join. Benchmarks claim up to an order of magnitude drop in latency and memory; EC2 INF2 shows ~3x compute, ~4x memory, ~4x throughput and ~10x lower latency.
AI-written summary · about
chapters (14)
- 0:00PyTorch for Edge AI
- 1:05Generative AI and Edge Computing
- 4:42LLM deployment challenges
- 6:49PyTorch Edge and SQT Torch
- 8:12Distributed model partitioning
- 9:19Edge inference optimization
- 10:26Performance benchmarks and hardware
- 12:16Distributed LLM inference case study
- 13:17Edge AI deployment best practices
- 14:44Open Source Spotlight: YOMO
- 16:32AI infrastructure simulation with zFlow
- 17:57Future of Generative AI and Edge
- 19:46Infinite Edge AI updates
- 23:46Key takeaways
related talks
20mPyTorch Edge: Developer Journey for Deploying AI Models Onto Edge Devices
22mIntroducing ExecuTorch from PyTorch Edge: On-Device AI...
33mExecutorch: Enabling and Accelerating PyTorch Models on Edge Devices
12mLightning Talk: Orchestrating Machine Learning on Edge Devices...
16mLightning Talk: Efficient Inference at the Edge: Performance You Need at the Lowest...
23mBringing ExecuTorch To the Next Frontiers of Edge AI
38mMPI Meets Machine Learning: Unlocking PyTorch distributed for scaling AI workloads
24mTraining large scale models using PyTorch