Challenges in Generated Video Evaluation
summary
Recent generative video evaluation relies heavily on CLIP-based scores for frame and prompt alignment but CLIP often misses detailed prompt elements and poorly correlates with human judgments; there is no shared prompt set (only image-focused collections like Party Prompts and DrawBench), so papers use bespoke prompts. A mitigation fine-tunes models with ~57 ethnicities × ~120 professions, generating ~1,000k paraphrased prompts via GPT and synthetic images to improve diversity. Emerging tools include eval crafter benchmark and efforts to build video-centric, human-correlated metrics.
AI-written summary · about
chapters (7)
- 0:00Video evaluation challenges
- 2:33Human perception of visual quality
- 5:29Prompt collections and metrics limits
- 9:40CLIP score prompt alignment issues
- 12:12Fairness and bias in video models
- 15:46Training approaches for fairness
- 17:34Benchmarking tools and future directions
related talks
45mGenerative AI for Image and Video Production
10mA large, crowdsourced evaluation of gesture generation systems on common data: The GENEA ...
53mGenerative AI: Overview and use cases
31mChallenges and Considerations in Language Model Evaluation
17mThe State of Generative Media
30mIs Prompt Engineering Dead? How Auto-Optimization is Changing the Game
30mGenerative AI will enable crowdsourced interactivity
06mInterview with Spela Poklukar