mTuner: Accelerating Parameter-Efficient Fine-Tuning on Multi-GPU Servers with...
summary
mTuner implements an Elastic Tensor abstraction that exposes four tunable operations — gather, discard, execute, checkpoint — and a one‑time schedule search to decide when to cache or release weights and activations. It runs on multi‑GPU PCIe and NVLink servers, targets LLaMA models from 7B–70B, and reports throughput improvements up to 51.2% on PCIe and 24.8% on NVLink (28.3% and 14.5% average), with small search overhead and an open GitHub release.
AI-written summary · about
chapters (12)
- 0:00mTuner introduction
- 2:01Fine-tuning LLM challenges
- 3:44Memory in efficient fine-tuning
- 4:47Memory inefficiencies observed
- 6:12Flexible memory management
- 7:38Elastic tensor abstraction
- 9:22Elastic tensor optimizes FSDP
- 10:26Communication efficiency with relaxation
- 11:29Adaptive data accumulation
- 12:33Evaluation and performance
- 13:58Overhead and limitations
- 14:44Key takeaways
related talks
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...
10mAuto-tuning vllm
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
22mMulti-level Machine Learning-Guided Autotuning for Efficient Code Generation on a Deep(…)
26mVLLM-Helion: SOTA LLM Performance by Advanced Autotuning and Fine-grained Dispatching
01h 40mFrom model weights to API endpoint with TensorRT LLM
16mHydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters
32mTaming the Machine: Basics of ML Models Training and Inference Optimization