Rethinking Dataset Discovery with DataScout
summary
DataScout uses LLMs plus embedding collections and search indices to map a user goal to actual datasets, precomputing descriptions and three embedding levels so online queries return interactive help in <seconds (naive strategy hit >12s). Features include reformulation suggestions grounded in the search space, relevance indicators that flag task-specific strengths/limitations, and column concept filters that remap semantic needs to actual columns; a 12-participant within-subjects study shows faster first-target discovery and fewer decision seconds per dataset.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 0:47Challenges in dataset discovery
- 1:53Current search system limitations
- 2:56User struggles with requirements
- 3:38DataScout features overview
- 5:00Column concept filtering
- 5:20Granularity and relevance indicators
- 6:04System architecture and latency
- 7:07User study methodology
- 7:48Study findings and insights
- 9:13Key takeaways and future work
- 9:56Audience Q&A
related talks
19mHow to look at your data
16mAutomated data discovery: Break free from hours of manual data hunting
28mSculpting Data for Machine Learning V02
57mVisualization Tools for Data Discovery
23m3-Step Path for Fixing Data Discoverability
43mIntroduction to Spark Datasets
34mSearching for My Next Chart
20mDesigning for Big Data