Morphe: High-Fidelity Generative Video Streaming with Vision Foundation Model
summary
Morphe uses a vision foundation model with joint-trained visual tokenizers, variable-resolution spatiotemporal optimization, a super-resolution upscaler and similarity-based token selection plus pixel-residue coding to reconstruct high-detail video from tiny semantic tokens. A WebRTC prototype on NVIDIA Jetson Orin demonstrates real-time, end-to-end delivery and intelligent packet dropping for robustness. Evaluation shows 62.5% bandwidth savings versus H.265, graceful quality under packet loss and practical training cost (~185 GPU‑hours on 16 A100s).
AI-written summary · about
chapters (16)
- 0:00Morphe intro
- 1:42Generative models for streaming
- 3:28Neural codec limits
- 4:11Generative streaming challenges
- 5:56Morphe’s three-module design
- 7:00Visual generative codec details
- 9:05Resolution scaling accelerator
- 9:48Network streaming controller
- 11:12Training and dataset scale
- 11:55Real-time prototype
- 12:16Visual performance results
- 12:57Loss resilience under packet loss
- 13:38Frame rate stability
- 14:23Visual quality vs packet loss
- 14:44Real-time performance across platforms
- 15:43Live demo and conclusion
related talks
17mGemino: Practical and Robust Neural Compression for Video Conferencing
17mYuZu: Neural-Enhanced Volumetric Video Streaming
21mReal-Time Video Super Resolution for Live Streaming with WebGPU
15mReal-time video upscaling using CNNs and WebGPU
32mVideo Action Models
24mState of Open Video Generation Models
20mComputer Vision at the Edge with OpenVINO
22mMovie Gen Video: State-of-the-art Video Generation