The Marvelous Magic of Multimodal AI
summary
Claims that multimodal models already generate full explainer videos end-to-end in about 30 seconds, producing script, music, animations and video from one prompt; demos use open-source Momo (Ai2) to do advanced VQA and a weather-map description that correctly ties a graphic to Tropical Storm Helen; underlying stack uses transformers, embeddings and a joint embedding space with CLIP + Glide diffusion to map text↔image and sample diverse outputs, and the examples shown work reliably for retrieval, VQA and image generation.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:55LLM vs LMM
- 1:08What is multimodal AI?
- 4:30Molmo by Ai2
- 9:26Data ≈ Representation
- 14:19Natural language is hard
- 22:04What about context?
- 28:55The inner workngs
- 35:21Text-to-images
- 40:04Outro
related talks
37mBuilding Multimodal AI Agents From Scratch
15mMulti-Modal AI Optimization: Combining Text, Vision, and Speech Efficiently
40mKeynote: Open-source Multimodal AI in the Wild
45mGenerative AI for Image and Video Production
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
13mInside Multimodal Models
26mAccelerating Explorations in Vision and Multimodal AI Using Pytorch...
17mAI/ML demo derby