Crossing the multimodal chasm
summary
Argues that crossing the multimodal chasm requires composing smaller models into a plumbing pipeline rather than one monolithic foundation model, and shows concrete tooling and patterns: explicit output constraints via LMQL and JSON mode, external actions via tool calling, and multimodal retrieval using Weaviate near_image and embeddings stored in a vector space. Offers code snippets and a phone-photo→identify microprocessor demo but no production metrics, focusing on integration patterns, orchestration tradeoffs, and when a pipelined approach actually produces reliable, pipeable outputs.
AI-written summary · about
chapters (6)
- 0:00The problem with AI apps
- 0:57Compound AI systems
- 2:45Predictable output strategies
- 4:42Tool calling and integration
- 6:07Multimodal retrieval
- 7:15Building integrated workflows
related talks
31mOrchestrating Multimodal RAG With Agentic Workflows
30mBridging Classic ML Pipelines with the World of LLMs
17mWhat is multimodal design? An interview with John Alderman and Christine Park
46mArchitecting Efficient Agents with Semantic Routing and Mixture-of-Tools
16mAny-to-Any: Building Native Multimodal Agents
19mFrom Demo to Deployment: Multimodal Conversational Agents
51mMultimodal interfaces: don't shout at me, I can't see you
37mBuilding Multimodal AI Agents From Scratch