Building Multimodal AI Agents From Scratch
summary
Builds a multimodal agent that answers questions over mixed text+images and explains charts, implemented in Python using MongoDB as a vector database and short‑term memory store and a multimodal LM (Google’s Gemini 2.0) for reasoning. Documents become a screenshot‑per‑page corpus, embeddings come from a VLM‑style multimodal embedder to avoid the CLIP modality gap, vector search returns image references, agent fetches images and runs a REACT loop over tools. Starter GitHub repo and two notebooks run locally in ≈10 minutes and produce a working demo.
AI-written summary · about
chapters (10)
- 0:00Introduction to AI agents
- 3:01Agent versus other paradigms
- 7:09Core components of agents
- 11:22Tools and memory
- 12:52Agent interaction flow
- 16:05Introduction to multimodality
- 18:24Multimodal data challenges
- 24:02Data preparation pipeline
- 31:48Agent workflow and memory
- 34:12Hands-on workshop lab
related talks
01h 34mHands-on AI workshop: Multimodal agents
40mThe Marvelous Magic of Multimodal AI
01h 34mHands-on AI workshop: Graph RAG, Memory & Multimodal Agents
02h 08mIntroduction to AI Agents
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
24mMulti-Modal AI Agents Explained | Open-Vocabulary Alignment
28mBuild an AI Research Agent
30mLook ma, no hands! Multimodal AI-Agents in the browser