Visual Quality Assurance Using Vision-Language Models
summary
Benchmarks include GlitchBench, a canvas UI set and Video Game QA Bench that split tasks into visual/unit UI regression, glitch detection and bug‑report generation; 16 VLMs (11 closed, 5 open) are evaluated. Off‑the‑shelf models reach about 75–80% glitch detection (GPT variants) but only ~50–54% for report generation. Fine‑tuning an 8B model (Video Game Bunny) on ~50k structured image→JSON examples outperforms a 34B baseline (~85% vs ~84%). Remaining gaps: small‑detail, spatial/counting errors and robust video temporal analysis.
AI-written summary · about
chapters (14)
- 0:00Visual quality assurance intro
- 1:28Visual bugs detection challenges
- 8:55Traditional testing challenges
- 10:43Vision-language models for QA
- 12:27Research directions in QA
- 13:10Benchmarks for visual QA
- 16:21Visual QA tasks with VLMs
- 23:03Glitch detection in gameplay
- 27:39Bug reports for visual issues
- 28:42Benchmarking 16 VLMs
- 31:56Improving VLM capabilities
- 34:43Fine-tuning Video Game Bunny
- 43:01Summary and future
- 44:03Q&A on false negatives
related talks
01h 08mMastering Visual AI with Vision-Language Models & Advanced Evaluation Techniques
18mBlind Quality Assessment for Real-Time Visual Communications
23mVision-Language Models for Fraud Detection
23mAVA: Towards Agentic Video Analytics with Vision Language Models
32mTeaching Machines to See:Vision-Language Models Made Efficient with SmolVLM
33mMeasuring Perceptual Video Quality at Scale
28mMachine Learning applications in accessibility
41mBridging Pixels and Words: The Transformer Vision Revolution