Keynote: Scaling Open Source Large Language Models
Talkat Cassandra Summit 2023 · on Wednesday, 2023-12-13· at 13:55· Grand Ballroom BC (Concourse Exhibit Level)·
summary
Together builds the Together Inference API to serve 100+ open source LLMs and shows it works but not without surprises. Model startup time is dominated by downloads for 70B = 140 GB checkpoints across thousands of GPUs, so aggressive caching and prepush are required. Bin packing forces evicting many small 7B instances to free whole 8‑GPU machines for large models. End‑to‑end timeouts need per‑hop tuning, and continuous batching is essential to reach usable capacity. Also highlights tools like FlashAttention and Helm for benchmarking.
AI-written summary · about
chapters (10)
- 0:00Open source LLM future
- 1:02Why open source leads in AI
- 2:25Control and transparency in LLMs
- 3:29Privacy in open source models
- 4:14Scaling challenges with LLMs
- 4:56Optimizing model startup and caching
- 6:39Bin packing tradeoffs
- 7:42Handling timeouts in inference
- 8:46Role of continuous batching
- 9:30Summary and LLM vision
related talks
10mKeynote: Scaling Open Source Large Language Models
11mKeynote: Building a Large Model Inference Platform for Heterogeneous Chinese Chips Based on VLLM
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
12mKeynote: AI Infrastructure
15mKeynote: From Models to Market: What's the Missing Link in Scaling Open Source Models on...
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
19mSystem for Scaling Large Language Models
18mKeynote