How to train a large diffusion model
summary
Built a Yandex image generator trained to predict compressed latents rather than pixels using VAE latents, conditioned on CLIP/T5-style text embeddings and trained on about 1 billion images (pretrain ~1 month, fine-tune on 1,000 high‑quality examples). Training uses ~32 denoising steps at inference, human preference ranking over ~500 prompts for RL tuning, and a visual language model to auto‑caption noisy web images. Major gotchas include infrastructure limits at ~2 GB/s and a catastrophic image defect caused by BFloat16 vs float16 bug.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:22Text encoders in diffusion models
- 5:52Latent diffusion with VAEs
- 7:19Billion-image dataset
- 10:53Data filtering and cleaning
- 12:19Three-stage training
- 14:48Human feedback alignment
- 16:32Large-scale training challenges
- 19:49Debugging training failures
- 20:53Training progress visualization
- 22:20Summary and availability
- 22:40Q&A
related talks
22mEfficient Pretraining of Text-to-Image Models
41mBuilding Generative Image & Video models at Scale
20mTraining a foundation image model and deploying it to millions of users
01h 17mDeep generative models for image...
10mDiffusion models
45mHey, Look, I'm an Artist Now © ML Image Generator
01h 21mGenerative AI with Diffusion Models
51mLarge Language Models for Real-World Applications - A Gentle Intro