Running LLMs locally: Practical LLM Performance on DGX Spark
summary
Reproducible local benchmark of serving open models on an NVIDIA DGX Spark with 128GB Grace Blackwell memory uses vLLM inside Docker and an automated harness that timestamps first-token latency and logs GPU metrics. Key results: 1.5B model = 61.73 tokens/sec, 14B NVFP4 = 20.19 tokens/sec vs 14B base = 8.40 tokens/sec, and 14B NVFP4 is 3.4× faster to first token. Conclusion: quantization format and memory bandwidth drive real-world throughput and responsiveness.
AI-written summary · about
chapters (7)
- 0:00Local LLM performance intro
- 1:27Cloud-centric AI challenges
- 2:32Jetson Spark stack
- 3:58Benchmarking harness design
- 5:22Throughput with vLLM on Jetson Spark
- 7:28Time to first token metrics
- 9:23Key takeaways and use cases
related talks
28mServing a Frontier Open Model with Ray & vLLM
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
13mHow Open-Source vLLM Topped the Artificial Analysis Leaderboard
22mWhich GPU for Local LLMs?
01h 18mOptimizing LLM Training and Inference Performance on GPUs
19mNo GPU Left Behind: Scaling Online LLM Training With...
31mThe Evolution of Multi-GPU Inference in vLLM
30mRun Your Own AI Cluster on a DGX Spark: Kubernetes, GPUs, and DRA