Demo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM
summary
Demonstration shows how TensorRT-LLM converts Gemma checkpoints (JAX/Torch to safe tensors in ~2 minutes), builds optimized engines (~1 minute) and runs inference with measurable gains. Claims include two lines of code to start, FP8 post-training quantization and custom KV cache and multihead attention kernels. Benchmarks move the latency/throughput frontier moving A100→H100 gives higher throughput for same latency, and engines deploy with Triton for efficient batching and multi‑GPU tensor parallelism.
AI-written summary · about
chapters (3)
related talks
01h 40mFrom model weights to API endpoint with TensorRT LLM
01h 21mTLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM
09mUsing and fine-tuning Gemma 3
39mRunning On-Device AI With Qualcomm AI Engine Using LiteRT & ExecuTorch
46mFrom ML to LLM: on-device AI in the browser
52mSimplifying the adoption of generative AI for enterprises
08mHacks to Make LLM Training Faster
09mLightning Talk: Vllm-triton-backend: How To Get State-of-the-art Performance on...