One GPU, Four Retrieval Modes: Multi-Model Search Serving
summary
Shows how one server process runs dense, sparse, ColBERT and cross‑encoder retrieval together on a single GPU by exposing three primitives: encode, score, extract. Uses a shared request queue, token‑based batching and LRU hot‑swap of models to maximize GPU utilization, with BEIR benchmark results that quantify when each mode improves recall or just wastes compute. Includes deployment details (Kubernetes helm/terraform), examples on L4 GPUs, and practical tuning for latency vs throughput.
AI-written summary · about
chapters (7)
- 0:00Introduction to SAI
- 2:31Agentic inference with small models
- 7:30Evolution of search pipelines
- 13:14SAI architecture and design
- 27:07Production use cases and integrations
- 28:55Key takeaways and open source benefits
- 31:28Q&A session
related talks
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
26mEfficiently serving LLMs at scale.
36mAccelerating High-Performance Machine Learning at Scale i...
15mAlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
28mServing a Frontier Open Model with Ray & vLLM
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
19mThe Small Model Infrastructure Nobody Built (So We Did)
21mCost Effectively Deploy Thousands of Fine Tuned Gen AI Models Like...