The Agentic Data Science Multiverse
summary
Built autonomous AI analysts on LLMs that run complete Python analysis pipelines and an LLM auditor that checks methodological validity, generating thousands of traces across three datasets (soccer refereeing, AI-assisted programming RCT, and TV news/political views). Analyses show substantial dispersion in estimates, p‑values, and conclusions driven by preprocessing, model specification, inference choices and by persona/model steerability; the auditor excludes many invalid runs but does not remove all bias. Visualized with a specification curve, the work argues for multiverse-style reporting and prompt disclosure.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:57Forking paths in data science
- 5:15LLM agents for analysis
- 9:09Datasets and hypotheses
- 11:39Visualizing specification curve
- 16:21Prompting personas
- 21:09Personas and p-values
- 27:17Effect comparisons across datasets
- 31:34Summary
- 36:36Statistical validity lessons
- 39:02Q&A
related talks
40mPannel Discussion | From Hype To Value: Real World Applications Of Generative AI And LLMs
30mWhat Can LLMs Do with Messy Data?
56mDehumanizing Agents: Why Explainability is Crucial in the LLM Era
44mOpen-Source and Science in the Era of Foundation Models
47mThomas in't Veld | AI Track | Compass AI & Data Summit 2025
26mAI Agents as Financial Co-Pilots | AI-Driven Strategy & Decision Making
38mLLMs for Self-serve Analytics - Lessons Learned
43mEval-Driven Development: Mastering Agentic Tracing and Expert-Aligned Judges With MLf...