Keynote: Scaling Open Source Large Language Models
summary
Together runs an inference stack (exposed as the Together Inference API) and leverages research like FlashAttention while scaling 100+ open models. Startup latency is often dominated by model downloads — a 70B = 140GB checkpoint copied to many GPUs — so aggressive caching and placement is required. Bin‑packing policies trade off robustness versus efficiency, long end‑to‑end timeouts must be tuned across the whole path, and continuous batching proves essential to reach usable capacity.
AI-written summary · about
chapters (10)
- 0:00Open source LLM future
- 1:02Why open source leads in AI
- 2:25Control and transparency in LLMs
- 3:29Privacy in open source models
- 4:14Scaling challenges with LLMs
- 4:56Optimizing model startup and caching
- 6:39Bin packing tradeoffs
- 7:42Handling timeouts in inference
- 8:46Role of continuous batching
- 9:30Summary and LLM vision
related talks
10mKeynote: Scaling Open Source Large Language Models
12mKeynote: AI Infrastructure
15mKeynote: From Models to Market: What's the Missing Link in Scaling Open Source Models on...
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
11mKeynote: Building a Large Model Inference Platform for Heterogeneous Chinese Chips Based on VLLM
27mKeynote: Open Source in the Age of AI
19mSystem for Scaling Large Language Models