Federated llm-d: Elevating Distributed Inference Beyond Clus...
summary
Federated llm-d stretches LLMD across clusters using Cilium Cluster Mesh for cross‑cluster networking, Nova as the multi‑cluster orchestrator, body‑based routing and inference pool/import CRs to forward requests, and vLLM plus the variant autoscaler for serving. A local demo runs two kind clusters (each 4 NVIDIA + 4 AMD GPUs), a load tester sends 100 requests with a 50/50 split demo, and scheduling overhead measures under ~10 ms. Recommends starting with loosely coupled federation; add Cilium Hubble/Tetragon for observability and security.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:05Distributed inference and LLMD
- 3:36LLMD architecture
- 5:23Federating LLMD clusters
- 7:10Multi-cluster orchestration with Nova
- 8:55Federated LLMD blueprint
- 9:59Live demo of federated setup
- 14:14Takeaways and future directions
- 16:44Q&A
related talks
33mllm-d: Kubernetes Native Distributed Inferencing
42mStop Treating LLMs Like REST APIs
04mProject Lightning Talk: Empowering Federated Learning with Multi-Cluster Management for Privacy and Efficiency
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
26mBuilding a Multi-Cluster Privately Hosted LLM Serving Platform on Ku...
32mScaling Generative AI Inference with llm-d
15mReimagine the Multi-Cluster
10mLightning Talk: Connecting Klusters on the Edge with Deep Dive into Cilium Cluster Mesh