Inside Multimodal Models
summary
Explains the actual architecture and hands-on results: attention + shared Transformer layers, encoder fusion and contrastive training that CLIP contrastive training uses, and generation from text with DALL-E with billions of parameters. Shows API-accessible demos: image captioning that offers concrete recipe fixes, image-to-text anomaly detection for floods, dashboard-to-code conversion, language translation from photos, and handwriting feedback. Cites adoption stats (over 50% of orgs, 70% of employees) and runnable examples.
AI-written summary · about
chapters (7)
- 0:00Welcome to PROMPT Engineering Conference 2024
- 0:41Understanding Multimodal Models
- 2:54The Technology Behind Multimodal Models
- 4:12Exploring CLIP and DALL-E
- 6:11Multimodality in Action: Demos
- 6:47Practical Applications of GPT-4
- 11:57Conclusion and Next Steps
related talks
28mLarge Multimodal Models: Understanding the State of the Art in Compute
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
01h 06mMixed-modal Language Modeling: Chameleon, Transfusion, and Mixture of Transformers
01h 04mFrom Pixels to Words: Exploring Multi-Modal AI with Python
42mDALL·E and CLIP from OpenAI: a new era in machine learning
37mA Practical Guide to Robust Multi modal Machine Learning in Education
01h 08mGoing multi-modal
40mKeynote: Open-source Multimodal AI in the Wild