20 days of compute vs 7 hours: rethinking what state-of-the-art means
summary
Benchmarking runs 26,000 battles at ~62 seconds per image and totals 20 days vs 7 hours of compute depending on model size and compression the cost moves from $5,000 vs $265 and energy from ~400 marathons to about four. Leaderboards disagree and most models lose ~40% of head to head battles. Pareto plots of quality vs latency expose 3–4 models tightly clustered in quality but up to 20× apart in efficiency, found by pruning quantization distillation and caching on compressed Flux two style models served at 1–5s.
AI-written summary · about
chapters (13)
- 0:00State of the art in AI models
- 1:19Public leaderboard limitations
- 3:26Aggregated leaderboard problems
- 5:51Statistical significance in evaluations
- 7:37Internal benchmark flaws
- 9:02Human bias in inspections
- 10:28Automated metric inconsistencies
- 12:12Hidden compute costs
- 13:38Efficiency vs quality in model selection
- 14:43Pareto fronts for comparison
- 16:05Practical benchmarking rethink
- 16:47Puna performance models
- 17:29Q&A on compression
related talks
01h 16mOptimizing workloads with Transformers and Diffusers
26mCanva's AI Leap: Scaling with Ray and Anyscale
20mFireworks AI: Go from Hype to High - Value AI
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
40mScaling generative AI workloads with efficient model choice
34mDeep Learning at Scale
10mPruna AI
44mArchitecture Effects When Interacting with GPU Technologies