Create Generative AI experiences using Phi
summary
Shows deploying Phi family models with ONNX Runtime GenAI API and Olive, running Phi3-Mini on DirectML, CUDA and CPU with measured throughput: ~250 tokens/s on an RTX 4090, ~60–75 tokens/s on an A10/fast DML setup, ~15–20 tokens/s on low‑VRAM GPUs via CPU, and ~10 tokens/s on a Galaxy S21. Demonstrates Olive optimization workflows producing a ~2 GB base model plus ~25 MB adapters, hot‑swapped in an app, and finetuning via Azure Container Apps taking ~40–90 minutes.
AI-written summary · about
chapters (11)
- 0:00Welcome
- 1:36ONNX ecosystem trilogy
- 5:57ONNX Runtime performance
- 8:29ONNX Runtime Generative API
- 10:14Olive optimizes models
- 14:56Demo: Phi locally with AI Toolkit
- 19:32Demo: Generative API
- 23:51Fine-tuning scenarios
- 28:07Demo: Fine-tuning with AI Toolkit
- 36:22Demo: Hot-swapping adapters
- 39:13Wrap-up and Q&A
related talks
23mFoundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive
45mDeliver AI-powered experiences across cloud and edge with Windows
46mAI made easier: How the ONNX Runtime and Olive toolchain will help you Q&A
52mSimplifying the adoption of generative AI for enterprises
49mWriting GPU-Ready AI Models in Pure Java with Babylon
27mAccelerating Generative AI
31mOminiX - Unified Acceleration Framework for Both LLM and SD GenAl Models on the Edge
29mEfficient AI with Python: running AI models on CPUs