Production-Ready GenAI with Open Models for Java Teams
summary
Shows a Java Spring Boot demo that ingests three docs into an in‑memory embedding store, runs RAG with LangChain for J and Ollama, and evaluates answers with a 3‑question fidelity test that reports latency, precision, relevance and faithfulness. Demonstrates swapping models without code changes, quantized GGUF models on CPU, TTFT and tokens/sec tradeoffs, and an example MoE model NeMo‑Tron 3 Super 120B (120B params, 12B active). Cites a real case that cuts token cost by 90% via routing.
AI-written summary · about
chapters (17)
- 0:00Introduction to open models
- 1:03Why enterprises choose open
- 4:00Key open model terms
- 7:24Open source vs open weight
- 8:46Open model landscape
- 9:42Model selection and benchmarks
- 12:37JVM resource management
- 15:08LangChain4j integration
- 19:19Local inference with Ollama
- 20:10Live demo: local inference
- 21:51Tool calling and orchestration
- 26:56Building RAG pipelines
- 29:02Evaluating RAG quality
- 35:45Safety layers
- 38:07Production deployment options
- 47:24Model tools and runners
- 48:54Observability and conclusion
related talks
48mBuilding AI Powered Applications with Java
01h 03mBuilding AI powered Applications with Java
46mGenerative AI and Java
41mFrom Chat to RAG to MCP: Enhancing Java Applications with AI
51mIntelligent Applications with Spring AI
26mJava’s GenAI Era: Getting Spring AI and Neo4j Applications to Production
52mIntelligent Beans with Spring AI
33mOperationalizing GenAI Applications with DevOps Practices