NanoFlow: Towards Optimal Large Language Model Serving Throughput
summary
NanoFlow splits request batches into nano-batches and duplicates per-batch operations to overlap compute, memory (KV cache), and network work within a single GPU, using profiling plus mixed integer linear programming to auto-search nano-batch count/size, ordering, and GPU resource partitioning (10–30 minute searches). Implementation runs on A100/H100, evaluates on LLaMA-2-70B, Mixtral 8×7B, LLaMA-3-8B, reports a 1.91× throughput boost and reaches 50–78% of optimal (≈2000 tps for LLaMA-3-70B on A100), with latency tradeoffs at low load.
AI-written summary · about
chapters (10)
- 0:00NanoFlow and LLM serving challenges
- 1:24Transformer pipeline bottlenecks
- 3:52Compute-bound LLM serving
- 5:38Throughput for compute-bound workloads
- 7:23Nano-batching to overlap operations
- 9:12Nano-batch scheduling design
- 11:44GPU resource partitioning in NanoFlow
- 13:09Throughput and latency evaluation
- 14:12NanoFlow architecture and future work
- 14:57Conclusion and open-source
related talks
22mPerformance Optimizations at 100K+ Scale
15mResource Multiplexing in Tuning and Serving Large Language Models
16mRevisiting Pipeline Parallelism for LLM Serving
26mEfficiently serving LLMs at scale.
17mCharacterization of Large Language Model Development in the Datacenter
16mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
19mSystem for Scaling Large Language Models
24mEfficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM