PyTorch on Windows

on YouTube ↗12m

" summary

PyTorch 1.7 on Windows adds distributed data parallel training with multi‑GPU support (demo runs on an Azure VM with two NVIDIA V100s and shows multiple Python processes and nvidia‑smi utilization). Initialization currently requires a remote file store, and the release enables the Gloo backend while NCCL and MPI lack Windows support; multi‑node is allowed but labeled unstable. Examples include a two‑layer model, checkpoint + barrier sync, and model‑parallel across GPUs, and the demos run successfully.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 0:39Overview
3. 1:04Why Care About Windows
4. 2:33State of PyTorch
5. 4:08Distributed Training
6. 5:31Demo

" related talks

• 07mPyTorch at MicrosoftSaurabh Tiwary · PyTorch Developer Conference 2019
• 24mTraining large scale models using PyTorchShagun Sodhani · PyData Global 2023
• 08mPyTorch Distributed: Towards Large Scale TrainingAnjali Sridhar · PyTorch Conference 2022
• 41mGPU Training Made Easy-Pre-train & fine-tune NLP modelsVamsi Sistla · Global Artificial Intelligence Virtual Conference Sep 2020
• 20mLearn You a PyTorch!Kendrick Tan · PyCon AU 2017
• 23mPyTorch on Google Cloud TPUsBrian McCann · PyTorch Developer Conference 2019
• 19mFine tune nanoGPT for Microsoft AzureKelli Belcher · Microsoft Build 2024
• 25mDistributed Training in PyTorch: Zero to HeroCorey Lowman · PyTorch Conference 2025