Case Study: Kubernetes at Comcast
summary
Kubernetes runs a Cloud DVR stack deployed to 8 distinct physical locations (three in production) backing storage and a database that handles about 300 gigabits a second egress and roughly a million writes/sec; automation provisions an immutable undercloud and has 1,000 physical servers deployed automatically. IPVS-based load balancing replaces the slow userspace proxy, logs use container stdout shipped by a custom agent after hecka issues, Docker race conditions and a kernel IRQ bug cause the biggest operational pain, and weekly deploys to production succeed.
AI-written summary · about
chapters (31)
- 0:00Introduction
- 1:35Cloud DVR
- 2:34Architecture
- 3:15Existing Deployment Processes
- 4:53HighLevel Requirements
- 5:49Kubernetes
- 8:20Deploying Kubernetes
- 9:25Goals
- 11:53Outcomes
- 13:10Load Balancing
- 13:50How it Works
- 14:23Kubernetes Load Balancer
- 15:46Kubernetes Limitations
- 16:28Logging and Monitoring
- 17:20Deliver Immediate Value
- 18:05HighLevel Goals
- 18:25Logs
- 19:30Logging Service
- 19:48Monitoring Service
- 20:32Monitoring Tool
- 21:21Problems with Kubernetes
- 23:24Problems with Docker
- 26:48Kernel bug
- 27:05CPU bug
- 27:56Deployment
- 28:32Team
- 29:37Technology Choices
- 31:04Stumbling Blocks
- 31:58DevOps is Hard
- 34:00Future of VIPER
- 34:37Questions
related talks
19mPains and gains of Kubernetes for Developers, story from experience
05mMaking the Impossible Possible with Kubernetes
41mKubernetes the Very Hard Way
31mAn Adventure in Multi Cloud with Kubernetes and Distributed Data
39mBuilding Distributed Systems with Kubernetes
21mLessons from the Field: Running Kubernetes at the Wellsite
01h 40mInside Kubernetes Architecture Fundamentals
07mData Gravity and Kubernetes: Managing Large-Scale Data Ingest With Minimal Latency