Adapt to Unified and Pluggable Cluster Management Platform at LinkedIn
summary
LinkedIn replaces parts of its homegrown stack with Kubernetes integrated into RAIN to manage tens of thousands of hosts per cluster across multiple datacenters (including Azure). A 1,600-node experiment running Orca post‑commit jobs shows Kubernetes cuts queue‑to‑run latency significantly. Production work includes JupyterHub on k8s with kubelets obtaining LinkedIn TLS certificates, debugging an IPv4‑forwarding automation bug and a pre‑1.14 init‑container env var issue, enabling distributed TensorFlow training and TF‑Serving, and building a delegation token controller + Locker CRI so pods access Kerberos HDFS securely.
AI-written summary · about
chapters (32)
- 0:00Introduction
- 0:25Agenda
- 1:12About LinkedIn
- 1:34Our Mission
- 2:08Our Scale
- 2:29Our Cost Management Platform
- 3:18Fabric Architecture
- 4:07Deployment Control Plane
- 4:31Locker Container
- 5:06Batch Jobs
- 5:37Challenges
- 6:24Kubernetes Performance
- 7:05Kubernetes extensibility
- 7:38Pluggable scheduler architecture
- 8:50First use case
- 9:53Jupiter Hub
- 12:01Kubernetes Journey
- 12:15Notebook Pod Crash
- 13:15Debugging
- 13:32Particle Networking
- 13:56Pod Network
- 17:16Pod Network Crash
- 18:05Jupiter Hub Pod
- 19:01Pod Preset
- 21:00Batch Batch
- 22:05Model Serving
- 23:16Use Cases
- 23:39Stack vs Kubernetes
- 24:38Bringing both worlds together
- 24:52Kerberos
- 26:49Unified Topology
- 28:10Wrap Up
related talks
29mAdapt to Unified and Pluggable Cluster Management Platform at LinkedIn
40mFrom Metal To Apps: LinkedIn’s Kubernetes-based Compute Platform
53mImplementing Kubernetes on with confidence
18mKeynote: Adapting Kubernetes for Machine Learning Workflows
14mKeynote: Kubernetes as a Foundational Layer of Infrastructure
32mKubernetes at Reddit: Tales from Production
20mKeynote: Kubernetes at GitHub
23mMore Power, Less Pain: Building an Internal Platform with CNCF Tools