Multi Adapter Hosting LoRA Adapter
summary
LoRA fine-tuning keeps the base model frozen and trains only two low‑rank matrices A and B so adapters add few parameters yet match full fine‑tuning on many tasks. Practical numbers: a 7B baseline that needs ~70GB can run with adapters at ~16GB and drop to ~9–5GB with quantization. Adapters load and unload so one model serves many tasks — measured swap latencies ~10ms CPU→GPU and ~64ms disk→GPU and adapters train in ~20 minutes for about $13 on an 8B model.
AI-written summary · about
chapters (8)
- 0:00LoRA adapters overview
- 1:02Deep learning trends
- 4:57Parameter efficient fine-tuning
- 11:46Multi-adapter inference challenges
- 13:10SageMaker for LoRA hosting
- 17:57Performance metrics
- 18:17Live demo: LoRA hosting
- 23:56Summary of LoRA benefits
related talks
32mLoRAX: Serve 1000s of Fine-Tuned LLMs on a Single GPU
14mToppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference
15mdLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
32mFine-tune LLMs from Kaggle Models using (Q)LoRA
27mStageML: Partial Evaluation for Multi-Tenant MoE-LoRA Inference
29mLeveraging open-source LLMs for production