Managing Small Datasets Without Losing Your Mind
summary
Starts with 2,000 images and ends up managing ~200,000, reduces dataset update time from 5–7 days to about 2 days by using a clear stage pipeline instead of heavy Airflow/DagsHub setups or endless Roboflow/CVAT import‑exports. Concrete practices: keep new inputs in a staging folder, enforce a fixed train/val split, build a custom annotator with OpenCV, run models locally on GPUs via Vast AI, and version data with DVC; iterate until measurable metrics improve.
AI-written summary · about
chapters (16)
- 0:00Managing small datasets
- 3:59Big data tools limits
- 5:24Import-export workflow problems
- 6:29Stage 1: New data analysis
- 7:51Stage 2: Export and format
- 8:35Stage 3 Annotation and automation
- 11:02Stage 4: New data staging
- 12:47Custom dataset formats
- 13:50Train-validation splitting
- 15:13Stage 5: Local model training
- 16:16Stage 6: Model improvements
- 17:39Stage 7: Holdout test errors
- 18:22Stage 8: Cross-validation
- 19:28Stage 9: Data merging
- 20:50Results and benefits
- 21:54Universal approach
related talks
58mWorkshop: Empowering Collaboration: AI Developer Experience - Your Bridge from Mod...
28mEfficient Time-Series Forecasting with Thousands of Local Models on Databricks
35mData Processing Pipelines for Small Big Data
18mPutting the Fast in fast.ai - Machine Learning Models in 15 mins
31mThe role of CDEs in Generative AI
33mThe Role of Software Developers in Machine Learning
38mML Data Version Control and Reproducibility at Scale
31mAI Coding Agents Don't Need Bigger Models. They Need Better Tools.