HW, SW, Performance and Costs for Llama-2 70b and Mixtral 8x7b LLM Inference with Low...
summary
Benchmarks use hamelsmu’s repo and containerized runtimes (NVIDIA Container Toolkit + CDI) across CTranslate2, MLC, Hugging Face Transformers, Triton, TGI and po‑infer, showing ~26–32 tokens/sec for 7B on single GPUs and 23 tokens/sec for Llama‑2 70B (CTranslate2 on 4x A100s). Hardware tested includes Nvidia L4 and A100 variants, 70B needs ~156 GB VRAM, on‑prem server quotes ~$65–$72k (~$7/hr) and cloud capacity is often unavailable or 50% pricier. Containers work; multi‑GPU setup and NVLink matter for speed.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:13Mission and client requirements
- 7:54Hardware choices AMD vs Nvidia
- 12:11Nvidia GPUs and containers
- 16:08Benchmarking inference speeds
- 22:28Llama-2 7B performance
- 29:07Hardware for Llama-2 70B
- 38:14Cloud deployment challenges
- 43:15Cost comparison cloud vs on-prem
- 47:54Key takeaways
related talks
29mLeveraging open-source LLMs for production
48mBringing Llama 3 to Life
35mLocal LLMs on low end hardware - a practical perspective
16mHow fast are LLM inference engines anyway?
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
31mBuilding LLaMA: Meta's Director of GenAI
31mDeploying Large Language Models with Ease