How to build world-class AI products
summary
Notion AI describes running production AI for 100 million users, switching new models into production in under a day, and spending roughly 10% on prompting and 90% on evals. The team fine‑tunes models, manages hundreds of prompt versions, and uses Brain Trust for LLM‑as‑judge scoring, online scoring of live logs, and human‑in‑the‑loop labeling; retrieval is frozen during evals to avoid flakiness, and these practices drive measurable quality gains and faster model rollouts.
AI-written summary · about
chapters (17)
- 0:00Introduction and setup
- 1:46Building world-class AI products
- 8:06Notion AI journey and challenges
- 16:33Evaluation and observability importance
- 22:08Criteria for judging AI features
- 28:28Brain Trust workshop overview
- 31:17Why evals matter in AI development
- 34:31Core concepts of Brain Trust
- 42:01Activity one: Platform walkthrough
- 50:15Running evals in the UI
- 1:03:17Activity two: SDK-based evals
- 1:05:47Production logging and online scoring
- 1:15:04Activity three: Logging setup
- 1:24:01Human-in-the-loop workflows
- 1:26:55Activity four: User feedback
- 1:35:51Remote evals and advanced features
- 1:42:32Q&A and closing remarks
related talks
28mAI at Work
49mHow to be an AI First company
36mBeyond the Hype
33mEngineering AI Applications: A Conversation with AI Leaders
14m15 Months of building AI agents in 15 minutes
37mOn Building Production-Quality AI Systems
27mBuilding AI applications: Successes and Lessons
37mPanel Discussion: Developing with AI