A full-scenario colocation of workloads based on Kubernetes
Talkat KubeCon + CloudNativeCon + Open Source Summit China 2021 · on Thursday, 2021-12-09· at 12:10· Kubecon + CloudNativeCon 演讲厅, 虚拟会议 (Virtual)·
summary
Describes a Kubernetes-native system for colocating online services and offline jobs by leveraging techniques such as resource prediction, resource isolation, interference detection (including kernel-level metrics via eBPF), and offline eviction. Covers support for containerized and non-containerized online services and offline workloads across Kubernetes and Hadoop ecosystems, and presents large-scale deployment results from Tencent (40,000+ machines, ~2,000,000 cores, ~15% utilization improvement).
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:21Why colocation
- 1:11Make the colocation better
- 1:35Colocation on K8s — Caelus
- 2:29Principles on kubernetes
- 3:47Full-scenario colocation
- 5:05Resource prediction
- 7:15Prediction algorithms
- 8:14Cgroup hierarchy
- 9:54Resource isolation
- 11:10Interference detection
- 13:24Resource load detection
- 14:27RT detection
- 15:46Function detection
- 16:34Interference handling
- 17:46Improve resource utilization
- 18:00Run more offline jobs
- 19:07Results
related talks
06m⚡ Lightning Talk: Colocating Data Workloads and Web Services on Kubernetes to Improve Resource Utilization
37mA Hybrid Container Cloud With Kubernetes and Hadoop YARN
32mCoordinate Workloads Colocation: QoS-Oriented Scheduling Enhancement on K8s
19mKeynote: Building Bridges: Cloud Native and High Performance Computing
24mDynamic Big Data in the Cloud with Kubernetes
47mCloud Native and Sustainable, Reproducible Scientific Computing
36mMulti-Cloud Machine Learning Data and Workflow with Kubernetes
37mRunning Mixed Workloads on Kubernetes at the Institute for Health Metrics and Evaluation