Learning to Sketch with Differentiable Rendering
summary
Differentiable rendering trains a rasterizer to map curve control points to images and optimises those control points directly, implemented in PyTorch with PyTorch Lightning and demoed in JupyterLab. Bézier strokes use six numbers each; experiments run with up to 128 strokes and can drop to about five when using perceptual losses like Sobel or a CLIP perceptual loss. Training to fit one target image takes minutes on a CPU, adding a neural-net loss roughly doubles runtime, and outputs reliably capture the essence rather than photoreal detail.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:05Computers sketch badly
- 2:56How rendering lines works
- 5:49Why differentiable rendering matters
- 7:15Bezier curves for sketching
- 9:45Differentiable rendering in PyTorch
- 12:16Loss functions for sketches
- 14:03Training the sketch model
- 15:49CLIPasso results
- 18:58Improving sketch quality
- 20:22Handling curved strokes
- 21:24Training time hardware
- 22:06Art and animation apps
- 23:34Pressure and color in sketches
- 25:41Gradient descent convergence
- 26:463D differentiable rendering
- 27:29Open source tools
- 28:11Q&A
related talks
36mPyTorch: Fast Differentiable Dynamic Graphs in Python
31mDoug Eck, Research Scientist, Google
40mLearn to be a painter using Neural Style Painting
11mLightning Talk: But How Do You Autograd Through Your Simulator? Making a...
20mLightning Talk: Beyond Polygons: The Future of 3D Graphics with Implicit Models
49mExploring Neural Networks for Recognition of Hand-drawn Shapes
47mDeep Learning with PyTorch
19mSketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications