VideoCraft: A Mixed Reality-Empowered Video Generation Workflow with Spatial Layer Editing for Co...
summary
VideoCraft combines a mixed-reality pre-stage with off-the-shelf video-to-video models (examples: Runway) and a new spatial layer mechanism to enable local edits of style, geometry, and dynamic states. A prototype records MR-decorated footage, constrains generation by blending virtual and real regions, and previews results; formative work with 5 designers and a controlled study with 12 participants shows improved spatial precision and faster, more predictable iterations. The system supports primitive and custom layer shapes, semantic-assisted masking, and geometry presets (e.g., room insertion, sunken seating), and the pipeline works end-to-end at typical V2V resolutions.
AI-written summary · about
chapters (6)
- 0:00Introduction to VideoCraft
- 2:34Mixed reality authoring workflow
- 4:38Introducing special layer mechanism
- 7:22Layer states and geometry editing
- 9:29Evaluation and user study results
- 10:11Conclusion and future work
related talks
39mDesigning for the Living Room: The Spatial Architecture of Mixed Reality
41mComputer Vision for Remote AR
03mVirtual Real Spaces: In-Home Gaming Experiences Bridging Reality and Virtuality
44mUX + Mixed Reality
47mKeynote: How VR, AR, MR and XR are changing the world
43mVR and MR Will Change Everything: How Can You Take Your
31mMerge Realities, Multiplay Wonder: Expert Guidance on Mixed Reality Development
01h 10mCreating Tools for Spatial Computing