Going multi-modal
summary
Hands‑on coverage of model selection, embedding storage, fine‑tuning and production deployment with two concrete builds: a Cairo Geniza search system that indexes roughly 40,000 fragments using Nomix/CLIP‑style embeddings, ElasticSearch, a separate embedding service (Docker compose) and an Olama RAG assistant, and a flood forecasting pipeline that fuses satellite imagery with time‑series embeddings for flash‑flood alerts. Shows which components fail unless the image encoder is fine‑tuned, why a hybrid pipeline (OCR then LLM for JSON structuring) beats end‑to‑end VLM transcription, and what actually worked in production.
AI-written summary · about
chapters (9)
- 0:00Multimodal AI fundamentals
- 7:28CLIP and fusion mechanisms
- 13:55Cairo Geniza project challenges
- 22:21Embedding spaces for RAG
- 30:45Multimodal LLMs limits on scripts
- 36:23Hybrid OCR for structured docs
- 44:56Time series and climate models
- 53:11Open-source architecture walkthrough
- 1:01:09Q&A
related talks
26mAdvancing Multi-Modal Search Capabilities in Search Pipeline
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
01h 30mBuilding a Search Engine
01h 28mFrom RAG to Riches Building Enterprise Search Systems with LLMs
43mLarge Language Models for Devs: from zero to your first LLM application
01h 23mMultimodal Deep Learning in the Real World
31mTaking AI/ML to the next level: Text!
22mBuilding a Multimodal Search Engine