Democratizing Diffusion Models with Diffusers
summary
An open-source Python library (Apache 2) that runs state‑of‑the‑art image and video diffusion models (Mochi1, Stable Diffusion 3/3.5, Flux) with four‑line inference examples, explicit device placement, and support for PyTorch and Flax backends. It ships native NF4 quantization (via bitsandbytes) and CPU offloading so models that otherwise need ~80 GB can run on a 16 GB Colab (Mochi1 and SD3.5 tested). Pipelines expose swap‑able components (text encoder, denoiser, scheduler, decoder), support model parallelism and text-to-video-zero (2D→video, ~16 frames), and logs mismatched configs while leaving evaluation of swapped components to users.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:23Talk and logistics
- 2:07Text-to-image diffusion rise
- 4:58How diffusion models work
- 6:48Beyond text-to-image
- 8:59Real-world applications
- 10:01Open and closed models
- 11:24Introducing Diffusers
- 12:27Generate images/videos in 4 lines
- 13:53Video generation models
- 14:56Additional Diffusers tasks
- 15:41Why native quantization
- 17:27Diffusion model components
- 19:12Model configuration
- 21:18Reusing pipeline components
- 22:21DiffusionPipeline explained
- 23:23Device placement and memory
- 24:26Memory with CPU offloading
- 25:08Minimal abstractions
- 25:52Diffusers for research
- 26:55Q&A
related talks
45mDiffusion to Prod: vLLM-Omni Unifies Multimodal Inference
19mYou Might Not Need 50 Diffusion Steps
01h 11mTutorial: Beyond Neural Search: Hands-on Building Cross-Modal/Multi-Moda...
50mAI'll Be Back: Exploring Diffusion Models for Image, Audio & Video
49mGenerative Diffusion Models: Development, Training and Releasing Kandinsky 2.1
20mRealtime Video AI with Diffusion Models
01h 16mOptimizing workloads with Transformers and Diffusers
30mHow to train a large diffusion model