Visual Search
summary
Walks through building Allegro's visual search engine with concrete results and tools: ResNet embeddings extracted from the penultimate layer, perceptual DCT p‑hash for deduping, active learning to label 20k images, and triplet loss with hard‑negative mining. Reports measured jumps: ImageNet features ~2% accuracy, attribute model 33%, cleaned triplets 21% then 71% with hard negatives, and a final combined + pretraining setup reaching 81% accuracy on dresses. Uses Faiss for large‑scale nearest‑neighbor search across ~100M offers.
AI-written summary · about
chapters (25)
- 0:00Visual search challenges
- 1:12Traditional attribute limits
- 2:14Neural networks for image similarity
- 3:21Accuracy metrics for search
- 4:04Predicting attributes with neural nets
- 5:08Duplicate image detection
- 6:11Evaluating attribute search results
- 7:17Siamese Networks introduction
- 8:48Positive image pair challenges
- 9:30Creating positive pairs from offers
- 10:10Dataset cleaning with active learning
- 12:39Image type classification
- 13:41Filtering training pairs
- 14:23Triplet loss and hard negatives
- 16:31Hard negative mining
- 17:15Hard vs random negatives
- 18:21Triplet loss with attribute prediction
- 19:44Multitask learning with losses
- 20:49Fixing L2 normalization
- 21:29Pre-training on large data
- 22:33Random hard triplet datasets
- 23:16Pre-training on 400M images
- 24:00Final model performance demo
- 24:48Testing with doodles and sketches
- 25:09Q&A
related talks
33mMachine Learning in Visual Search
43mThe Search for a New Visual Search, Beyond Language
42mDeep Learning for Computer Vision
01h 26mBuilding an Image Search Engine with Generative Computer Vision Models
22mHarnessing image and text data for product matching
26mSimilarity search: Take your search engine to the next level
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
36mFinding needles in haystacks with Deep Neural Networks