Maximizing Multimodal: Exploring the search frontier of text-to-image models
summary
Claims concrete: Eezy swaps CNNs for CLIP/OpenCLIP embeddings to power reverse image search (launched May 2023) and reports visibly better results so obvious they skip A/B testing. Engineering uses Elastic and a 50-50 blend of BM25 token score and embedding similarity, but naive replacement harms downloads until behavioral signal is prioritized. Built features include a BLIP/Q‑former pipeline for magic metadata, a linear probe style detector trained with a few thousand labels, K‑means clustering of 50 centroids, and an exploratory vector “Babel” averaging tool.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:59CLIP and multimodal models
- 8:37Reverse image search with CLIP
- 11:07Hybrid scoring for text search
- 13:57Balancing AI and behavior in search
- 15:46Metadata with BLIP and CLIP
- 17:29Style detectors from CLIP
- 18:32Image clustering for categorization
- 19:15Recommendations with user embeddings
- 20:19Vector space with Babel Library
- 22:42Q&A
related talks
13mInside Multimodal Models
01h 08mGoing multi-modal
28mLarge Multimodal Models: Understanding the State of the Art in Compute
43mThe Search for a New Visual Search, Beyond Language
01h 26mBuilding an Image Search Engine with Generative Computer Vision Models
41mSemantic Multimodal Multilingual Similarity Engine
24mRediscover Your Data with a New Multimodal Search Capability
32mGenerative AI Text and Multimodal Embedding Models for Real World Use Cases