Where Are My Tomatoes? Extracting Data from Graph Images with AI
summary
System reconstructs Kaplan–Meier-style plots by combining occlusion-aware instance segmentation (fine-tuned vision transformers) with classical CV cleanup (morphological ops, skeletonization, grouping) to produce pixel-level event points, then uses an LLM for axis/legend metadata to map pixels to real X/Y units. Thousands of synthetic KM graphs are generated for fine-tuning; the pipeline handles crossings and heavy occlusion, reduces noise, and returns reconstructed series suitable for analysis — not perfect per-curve accuracy but good enough for pharmaceutical paper comparisons and automated data recovery.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:27Kaplan-Meier graphs challenges
- 3:12Computer vision for graphs
- 4:35Extracting plots with segmentation
- 7:03LLMs for metadata extraction
- 8:07Medical research applications
- 9:10Fine-tuning models
- 10:35Summary & Q&A
related talks
19mmayAI? Exploring Graphical Tools for Model Visualization & Enh.
01h 00mData, visualization, and designing AI
37mUsing Exploratory Data Analysis to Discover Patterns
14mEthical AI for Biomedical Imaging
27mData Prep for Graphs
34mSearching for My Next Chart
43mVisualising Machine Learning Models
21mInterpretability For Everyone