OpenTela: Unifying Decentralized Computing Resources for Heterogeneous LLM Serving (Oper…
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
A user-space overlay implements CRDT-based gossip and NAT-punching/relay transport to federate Slurm, Kubernetes and bare-metal serving engines without root or cluster changes. The system provides a persistent endpoint, heterogeneity-aware scheduling, and requirement-aware routing; it runs for 22 months, 13M requests, 15B tokens, 142 models across ~200 GPUs and reports zero user-facing API errors during volatility experiments. Operators release code and anonymized traces for reproducible sovereign LLM serving.
AI-written summary · about
chapters (11)
- 0:00Sovereign AI and OpenTela
- 1:45Fragmented AI inference challenges
- 3:51Decentralized GPU availability
- 5:58Requirements for unified compute
- 8:06Limitations of single serving engines
- 9:30OpenTela model serving access
- 10:56OpenTela architecture overview
- 13:27OpenTela production stats
- 14:53Elasticity and fault tolerance
- 15:58Decentralized system lessons
- 18:07OpenTela summary and future
related talks
40mBuilding a Modern Infrastructure Stack
32mDynamic Scheduling for Large Language Model Serving
40mSupercharging Cloud Orchestration with Temporal Open Source
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
38mOS-Climate & SIG-SRE: Benefiting from the Operate First Community Cloud
24mSponsored Session: PyTorch Distributed and Fault Tolerance
30mRunning Sovereign Cloud Stack Cluster for International Telecommunication Union: Lessons Learned
21mSponsored Session: Fault-Tolerant Training: How We Build Rel...