Objectionable Uses of Objective Quality Metrics
summary
Argues that objective metrics like VMAF PSNR and SSIM can be gamed and gives concrete attacks: tune psnr and similar encoder tweaks that improve scores while worsening perceived image, adversarial perturbations that exploit arithmetic pooling versus harmonic mean, and practical pitfalls like the ffmpeg compare filter inventing frames. Shows per-frame violin plots and BD‑rate shifts, and demos a distributed quality tool with 30x speedup for VMAF that verifies when metrics diverge from human judgments.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:20Agenda
- 3:20Peak SNR
- 5:52Common Metrics
- 6:19VMath
- 7:11Breaking VMath
- 8:05Violin Graphs
- 9:38Video Quality Metrics
- 14:48Good Hearts Law
- 15:16OnetoOne
- 16:30Takeaways
related talks
15mTowards Measuring Perceptual Video Quality & Why
33mMeasuring Perceptual Video Quality at Scale
12mMeasuring live video quality with minimal complexity, now available for everyone!
09mCMSD and the Temple of Doom: Escaping VQA's Legacy Traps
22mA taxonomy of video "quality," or: was Strobe right or wrong about quality?
09mUsing AI to get relevant KPIs for Audio Quality
16mPerceptual Evaluations: Evals for Aesthetics
14mEmpirical Privacy Metrics: The Bad, the Ugly… and the Good, Maybe?