What is multimodal RAG, and can we build a village with it?
summary
Builds a local multimodal RAG system that actually runs: a Unity front end talks to a Kotlin/Quarkus backend using LangChain4j, VV8 vector DB, llama 3.1 for generation and CLIP/CLAP for image and audio embeddings, with a small Python wrapper for audio. It creates a living Orc village in Mordor where NPCs move, remember conversations, answer questions from chunked lore, generate paintings and fetch matching images, but GPU swapping and noisy audio retrieval reveal limits. Key tricks are llama 3.1 + LangChain4j and image and audio CLIP/CLAP embeddings.
AI-written summary · about
chapters (6)
- 0:00Introduction and concepts
- 3:24Project stack and architecture
- 9:40Text-based agent logic
- 19:44Text-based retrieval system
- 27:35Image-based RAG integration
- 40:02Audio RAG and conclusion
related talks
41mWhat is multimodal RAG, and can we build a village with it?
43mWhat is multimodal RAG, and can we build a village with it?
40mWhat is multimodal RAG, and can we build a village with it?
48mCrafting your own RAG system: Leveraging 30+ LLMs for enhanced performance
48mCrafting your own RAG system: Leveraging 30+ LLMs for enhanced performance
31mOrchestrating Multimodal RAG With Agentic Workflows
42mTo RAG or not to RAG
41mFrom RAGs to riches: Build an AI document interrogation app in 30 mins