Building Scalable RAG Pipelines with Java and Kubernetes

on YouTube ↗42m

" summary

Shows a microservice RAG stack built in Java that actually runs on Kubernetes: a stateless orchestrator calling a retriever backed by Weaviate and a generator served with QServe/Gemma on GKE. Claims ingestion = extract/chunk/embed/load, caching with Redis cuts vector DB latency by an order of magnitude, fallback to OpenSearch when vectors fail, and observability via Prometheus, Grafana and OpenTelemetry. Demo aims at scaling to a few thousands requests per second but shows GPU scale‑to‑zero and cold start issues.

AI-written summary · about

" chapters (39)

1. 0:00🚀 Build Stuff Conference – Get Your Tickets!
2. 0:18Introduction to Scalable RAG with Java and Kubernetes
3. 1:00What RAG Is: Retrieval, Augmentation, and Generation
4. 2:08Why RAG Matters: Hallucinations, Stale Knowledge, and Private Data
5. 3:31The Problem with Most RAG Tutorials and Demo Apps
6. 4:26Why a Monolith Breaks Down for Production RAG
7. 5:24A Microservice Architecture for RAG Pipelines
8. 6:18The Orchestrator, Retriever, and Generator Roles
9. 7:18End-to-End Request Flow Through the RAG System
10. 8:22Why Java for AI Applications
11. 9:04Spring AI vs. LangChain4j
12. 10:55The Retrieval Layer: Ingestion and Online Search
13. 12:00Ingestion Pipeline: Extract, Chunk, Embed, Load
14. 13:35Online Retrieval Flow and Similarity Search
15. 14:45Vector Database Options: Weaviate, Milvus, pgvector, Redis, and More
16. 15:53Spring AI Vector Store Integration
17. 16:43Caching Retrieved Results with Redis
18. 17:43Failure Handling and Fallback Search Strategies
19. 18:42The Generation Layer and Prompt Engineering
20. 19:34LLM Options: Hosted APIs vs. Self-Hosted Models
21. 20:40Prompt Templating with Spring AI
22. 21:28Streaming Responses with Flux and Reactive Java
23. 22:20Why Kubernetes Fits This Architecture
24. 23:06Scaling Stateless Orchestrators, Vector Stores, and GPU-Based LLMs
25. 24:25HPA, KEDA, and Service Mesh Considerations
26. 25:35Multi-Region Challenges for RAG Systems
27. 26:54Observability: Logs, Metrics, and Traces
28. 27:48Using OpenTelemetry for Distributed Tracing
29. 28:27How Traces Reveal Bottlenecks in Retrieval Latency
30. 29:09Dashboards with Grafana and Prometheus
31. 29:52Live Demo: RAG App Architecture and Components
32. 31:06Reviewing the Java/Spring Boot Project Structure
33. 31:54QServe and Gemma Deployment in Kubernetes
34. 32:42Demo Walkthrough: Orchestrator, Retriever, OpenSearch, Redis, and Weaviate
35. 33:40Demo Requests, Caching, and Scale-to-Zero Behavior
36. 35:16Prometheus Metrics Endpoint and Kubernetes Health Checks
37. 36:09Load Testing and Watching Metrics in Grafana
38. 37:18Tracking Retrieval Requests, Fallbacks, and Latency Trends
39. 38:15Final Thoughts and Q&A

" related talks

• 25mComposable RAG Pipelines With OPEA Highlighting GraphRAGArun Gupta, Rachel Roumeliotis · NODES 2024
• 46mBlow past benchmarks using vector search and RAG with ElasticsearchJeff Vestal, Jody Bailey, Kim Brylle · Microsoft Build 2024
• 27mGen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster EnvironmeChristopher J Nuland, Selvi Kadirvel · Cloud Native AI Day Europe 2024
• 19mBuilding Production-Ready RAG SystemsSrinivas Chennupati · Conf42 Platform Engineering 2025
• 02h 47mFrom naive to advanced RAG: the complete guideGuillaume Laforge, Cedrick Lunven · Devoxx Belgium 2024
• 15mAgentic RAG on KubeflowRohit Kumar, Santhosh T · Kubeflow Community Showcase 2026: GenAI and MLOps in Action
• 26mResource-Aware Scheduling for Production GenAI with RAG running on Multicluster Cloud KubernetesAnne Marie Holler, Dave Southwell · Cloud Native AI Day Europe 2024
• 19mScalable Data and AI Solutions with Cloud-Native ArchitectureMeethun Panda · Conf42 Cloud Native 2025