Image search without image captions
summary
Wellcome Collection faces ~10,000 images a day (~350,000 a month) with sparse captions, so a pipeline maps image feature vectors (ResNet/VGG-style embeddings) into sentence embedding space (InferSent/word vectors) to let queries retrieve images directly. A working demo exists and helps catalogers, but it misfires on tight word clusters (e.g., birthday cake returns surgery/steak-like images). Transfer learning into the collection’s domain improves results but the system is not yet public-facing.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:56Image search without captions
- 5:42Image embeddings and word vectors
- 10:49Bridging image and text spaces
- 13:22Sentence embeddings and model choices
- 16:57Testing and evaluating the model
- 19:48Limitations and unexpected results
- 20:30Future plans
- 20:50Q&A
related talks
01h 26mBuilding an Image Search Engine with Generative Computer Vision Models
01h 28mBuilding a Semantic Search Engine
43mThe Search for a New Visual Search, Beyond Language
28mVisual Search
01h 30mBuilding a Search Engine
04mUsing Computer Vision to Understand Big Visual Data
22mHarnessing image and text data for product matching
01h 04mFrom Pixels to Words: Exploring Multi-Modal AI with Python