DGL Operator: Distributed Graph Neural Network Training with DGL and K8s

on YouTube ↗31m

" summary

Describes DGL Operator, a Kubernetes-based controller for distributed graph neural network (GNN) training using the Deep Graph Library (DGL). Covers GNN and DGL background, native DGL distributed training and its production-scale challenges, then presents the DGL Operator design: containerizing DGL components, workflow automation, resource management, multiple graph-partitioning options, and planned future work.

AI-written summary · about

" related talks

• 26mDeep Graph Library: Deep Graph learning at scaleGeorge Karypis · AWS Machine Learning Summit 2021
• 14mP3: Distributed Deep Graph Learning at ScaleSwapnil Gandhi, Anand Padmanabha Iyer · 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI '21)
• 16mAccelerating Distributed Graph Learning by Using Collaborative In-Network...Kai Chen, Junxue Zhang, Zhaoyi Li, Jiawei Huang, Yijun Li, Jingling Liu, Hui Li, Xiaojun Zhu, Shengwen Zhou, Jing Shao, Xiaojuan Lu, Qichen Su, Jianxin Wang, Chee Wei Tan, Yong Cui · USENIX ATC '25
• 15mMGG: Accelerating Graph Neural Networks with Fine-Grained Intra-Kernel Communication...Boyuan Feng, Yufei Ding, Zheng Wang, Yuke Wang, Ang Li, Tong Geng, Kevin Barker · OSDI 2023
• 28mTraining Large Language Models on KubernetesRonen Dar · Kubernetes AI + HPC Day North America 2023
• 34mDemocratizing Machine Learning on Kubernetes [I]Lachlan Evenson, Joy Qiao · KubeCon + CloudNativeCon North America 2017
• 29mLarge Scale Distributed Deep Learning with Kubernetes OperatorsYuan Tang, Yong Tang · KubeCon + CloudNativeCon Europe 2019
• 14mLightning Talk: A GNN Based Framework for Kubernetes Security Agents: Threat & Vulnerability...Zeyno Dodd · Open AI + Data Forum @ Open Source Summit North America 2022