Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications
summary
Sketchforme composes sketched scenes from text using a two-stage pipeline: a transformer-based layout model trained on Visual Genome (≈102k relation instances) that outputs bounding boxes via a Gaussian mixture box sampler, and an aspect-ratio conditioned sketch-RNN that renders object strokes from Quick Draw. A Mechanical Turk study (51 raters) finds 36.5% of generated sketches pass as human, and a 38-person language learning study shows faster learning and 4.6/5 helpfulness while the system still struggles with occlusion and pose detail.
AI-written summary · about
chapters (9)
- 0:00Introduction to Sketchforme
- 1:35System workflow overview
- 2:43Scene composer model details
- 6:16Object sketcher model details
- 8:50Generated sketches examples
- 10:14Human perception evaluation
- 11:42Interactive applications demo
- 13:51Future work directions
- 14:54Conclusion and Q&A
related talks
07m#ProjectScribbler
12mOutline and Detail: A Semantic-Driven Framework for Layered 2D Character Generation
40mSketch-n-Sketch: Interactive SVG Programming with Direct Manipulation
22mDreamSketch: Early Stage 3D Design Explorations with Sketching and Generative Design
15mDeepSketch: A New Machine Learning-Based Reference Search Technique...
39mGenerative Machines
07mScreen2Words: Automatic Mobile UI Summarization with Multimodal Learning
19mA Mixed-Initiative Interface for Animating Static Pictures