Why building eval platforms is hard
summary
Claim: an eval platform is a systems project not a UI problem, requiring shared definitions of “good,” reliable data pipelines, labeling workflows, versioning, and trust across teams. Example build path moves from spreadsheet to bespoke UI to observability with production traces. Tools actually used include object storage for multimodal traces, an open‑source warehouse plus DuckDB, and a bespoke DSL (BTQL that no one liked). Surprising scale: traces contain 10–20 megabytes spans. Outcome: the platform works only when tracing, analytics, and governance are engineered together.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:18Evals for agent quality
- 4:46Hidden complexity of eval platforms
- 8:42Maturing eval platforms
- 12:40Experimentation and playgrounds
- 14:27Evals and observability flywheel
- 17:19Scaling eval platforms
- 20:08Agent trace data challenges
- 22:38Future eval platform features
- 24:04Q&A on multimodal support
related talks
09mPlatform Engineering for Data Science at Scale
20mFive hard earned lessons about Evals
15mImportance of Observability in Platform Engineering
40mSurvivability of Observability: Taking your DIY system to success
18mYour Toolchain is Production: The Case for Observability
33mEffective observability in microservice architectures
33mBeyond Monitoring: The Rise of Observability Platform
17mLLM Observability, Evaluation, Experimentation Platform