Perceptual Evaluations: Evals for Aesthetics
summary
Argues that common metrics like FID break when images carry JPEG artifacts because models and training data inherit compressed internet artifacts, so quality scores diverge from human perception. Recommends training perceptual classifiers (continuous classifiers learned from pairwise human judgments) and folding brightness/color compression knowledge into evals. Describes KREA.ai building a small generative-media stack, hiring for aesthetics and hyperpersonalization research, and shows these learned perceptual metrics actually align better with human judgments.
AI-written summary · about
chapters (8)
- 0:00Start
- 0:15Introduction to Perceptual Evaluations
- 0:50The Problem with Current AI Evaluations
- 2:16Historical Context and Compression
- 5:14Limitations in AI and Human-centric Metrics
- 8:00Rethinking Evaluation and the Future of AI
- 12:44Evaluating Our Evaluations
- 13:32Krea's Role and Call to Action
related talks
32mTeach JS About Aesthetics with Machine Learning
11mThe Unbearable Lightness of (Agentic) Evaluations
26mObjectionable Uses of Objective Quality Metrics
01h 05mEvaluating Model Explainability Methods
09mQuantitative Evaluation of Machine Learning Explanations: A Human-Grounded Benchmark
01h 18mComputer Vision after the Victory of Data
18mBlind Quality Assessment for Real-Time Visual Communications
17mCalculating recommendations based on product images #bbuzz