Building AI Products That Actually Work
summary
Claim: many popular eval techniques don't predict real-world performance, and reliable AI products require live signals not just offline scores. Built Raindrop as a Sentry-for-AI to surface explicit and implicit signals (copy rates, refusals, regenerations) and use autogradable tests rather than LM judges. Use Trellis: discretize outputs, convert high-volume intents into semi-deterministic workflows, prioritize by negative sentiment × volume and achievable delta. Oleve-era apps hit viral scale (250M+ views) while keeping experiences repeatable and testable.
AI-written summary · about
chapters (5)
- 0:00Intro
- 0:30Who are you
- 1:45Building products that actually work
- 6:01Will AI products get easier
- 14:10Sid Bendre Oleve
related talks
27mBuilding AI applications: Successes and Lessons
30mIs This Actually Getting Better? A Three-Layer Approach to Measuring AI Products
14mHow We Turned AI Hype into Real Value
24mHow to Test AI Features Before Your Users Do
47mBuild Trustworthy AI Agents powered by Evals
27mBuilding Generative AI features
01h 26mShipping AI That Works: An Evaluation Framework for PMs
32mQuality engineering in the AI age: From testing AI to leveraging GenAI