The Moving Target: Benchmarking Dynamic AI Systems End-to-End
summary
Doctolib is building a patient-facing agentic stack spanning triage, clinical interview, retrieval, memory, booking and image understanding and evaluates it end-to-end rather than per-component. Concrete artifacts include a Dr BERT embedder (3072‑d), planned sparse autoencoders (top‑256) and a vector DB for NHS/WHO sources; evaluation uses Health Bench (5,000 conversations, 256 physicians), relevance@5 (baseline ~80%), cascade error attribution, TauBench metrics, pass@K and a 4s time‑to‑first‑token latency target while progressing from stage four toward stage five.
AI-written summary · about
chapters (10)
- 0:00Benchmarking dynamic AI systems
- 1:32Doctolib product vision
- 3:16LLM health answers evaluation
- 5:45Adding retrieval and grounding
- 7:52Benchmarking retrieval and open models
- 9:38Multi-agent orchestration challenges
- 11:45Agentic capabilities with Tau-Bench
- 13:49Patient journey simulation
- 15:54Full care journey orchestration
- 17:17Current stage and future
related talks
43mTesting Agentic AI Applications: Beyond Traditional QA
15mPractical tactics to build reliable AI apps
28mImplementing AI-Driven Observability in Healthcare
55mSurviving the AI Testing Apocalypse
41mTaming Testing of AI apps
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
50mAI IN SRE: HYPE VS REALITY IN 2026