When Semantic Search Breaks in your RAG systems: RAM Walls, Silent Failures, and the Architecture Decisions
summary
Team ships semantic search, hits 100M vectors, and RAM cost triples overnight because HNSW indexes live in memory: 1536‑dim vectors at 4 bytes ≈ 1 TB for 100M rows. Practical fixes include dimension reduction (Metryoshka dolls), aggressive quantization delivering ~32x compression with 92–96% recall, or a DiskANN hybrid that keeps quantized thumbnails in RAM and full vectors on SSD. Beware that filtered queries (tenant filters) can silently return zero results; workarounds are per-tenant indexes, iterative scans, or hybrid BM25+vector.
AI-written summary · about
chapters (11)
- 0:00Semantic search failures in RAG
- 3:12How vectors work in semantic search
- 10:38Why vector indexing approximates
- 16:12RAM wall in vector search scaling
- 18:21Optimizing RAM usage techniques
- 23:44Demo: Vector compression with quantization
- 26:59Disk-based ANN for large vectors
- 29:23Silent failures in RAG production
- 31:31Fixing tenant-specific vector search
- 32:35Detecting data drift in RAG
- 34:45Vector databases vs extensions
related talks
32mRetrieval Augmentation and Semantic Search at Scale
48mUnlocking Next-gen Semantic Search With OpenSearch 3.x
44mScaling Semantic Search in PostgreSQL
34mThe Challenges of Making Vector Search Billion-scale
45mBehind the hype: managing billion-scale embeddings in Elasticsearch and OpenSearch
20mChallenges in Building RAG Systems (and Solutions Using OpenSearch)
33mTowards a deeper understanding of retrieval and vector databases
43mOpenSearch and the Path to a Trillion-scale Vector Database