Deploying GenAI for Audio Generation on Mobile CPUs With ExecuTorch
summary
ExecuTorch plus Clyde AI runs Stable Audio on Arm CPUs generating 11s stereo 44 kHz clips with local inference under 10s (about 7s on one core, <4s on four cores on SM2-enabled silicon) by using 8-bit PTQ for the diffusion steps and FP16 for the autoencoder to preserve quality. Runtime dynamic quantization adds under 10% overhead. Same code builds for mobile and Arm cloud instances, and SM2 vector instructions provide the large performance uplift demonstrated.
AI-written summary · about
chapters (11)
- 0:00GenAI on mobile CPUs
- 1:06Performance testing before mobile
- 2:53Clyde AI ARM library
- 5:23Audio generation use cases
- 7:07Local vs cloud deployment
- 9:56Audio generation pipeline
- 12:03Dynamic quantization for diffusion
- 14:30Performance on mobile and cloud
- 16:22SME2 AI acceleration
- 19:38SME2 on ARM CPUs
- 21:26Key takeaways
related talks
23mBringing ExecuTorch To the Next Frontiers of Edge AI
13mRunning ExecuTorch Applications With Silicon Accelera...
12mAccelerating On-Device ML Inference With ExecuTorch and Arm SME2
14mFlexible Deployment of PyTorch Models on MCU-Class...
22mIntroducing ExecuTorch from PyTorch Edge: On-Device AI...
15mBuilding Gen-AI experiences using LLM and LVM
13mAI inference for the smallest processors
27mAccelerating Generative AI