Characterizing AI inferencing stacks
summary
Project simulates stacks by running Granite Vision 3.2 (2B) on an Nvidia Ampere GPU in a Jetson AGX Orin with real 9.4, swapping only the runtime to compare Olama vs VLM. A client repeatedly sends random cat images to a server container while Prometheus exporters collect power, CPU, GPU memory, latency and throughput; Grafana visualizes averages and SD. Results show VLM delivers lower latency and higher throughput but higher CPU and power use, while Olama uses less memory and energy. Configuration file controls model, runtime, GPU/CPU and context size.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:04Why AI inference stacks matter
- 2:07Project setup
- 3:29Client-server interaction
- 5:16Visualizing performance
- 6:42Interpreting trade-offs
- 7:24Future improvements
- 8:05Summary and repo
- 8:25Q&A
related talks
06mThe Building Blocks of Agentic Al
20mGAMING: Benchmarking Sustainable LLM Deployment on Edge Devices
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
34mScaling AI Like a Pro-PepsiCo’s LLM Deployment Strategy on K...
43mLLMs in the Wild - Streaming, RAG, and Real Time GenAI at Scale
39mA New Stack for the Gen AI Era
35mGenAI Stack for Developers
17mUnpacking the Llama Stack: Architecting Next-Gen AI Applications