Cortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via
summary
Cortex caches semantically similar LLM queries by storing a Semantic Element (query, response, and metadata: cost, latency, frequency, size, staleness) and indexing them with Seri: a coarse ANN (HNSW) filter + a lightweight small LLM judger (0.6B ranking model) and dynamic recalibration. Eviction uses LCFU scoring (frequency×cost×latency / size) and Markov prefetching; the judger co-locates with the main LLM via adaptive scheduling. Evaluation shows up to 3.6× throughput, ~95% hit rates on bursty web workloads, preserved accuracy versus non-cached baselines, and large cost reductions (reports up to 6×).
AI-written summary · about
chapters (11)
- 0:00Cortex and agentic workflows
- 1:04Agent loops and remote bottlenecks
- 3:12Caching potential for search queries
- 4:37Agent-aware caching requirements
- 7:51Cortex design beyond caching
- 10:38Metadata and retrieval pipeline
- 12:24Cache policies and prefetch
- 13:29Shared GPU architecture
- 14:11Performance benchmarks
- 15:59Cost efficiency and rate limits
- 16:21Key takeaways and future
related talks
36mOptimizing LLM Performance With Caching Strategies in OpenSearch
30mDesigning Fast & Cost-Efficient Agents: The Power of Caching
26mLMCache: Lower LLM Performance Costs in the Enterprise
24mCost and Performance Optimization of LLM Inferencing.
14mOptimizing RAG with Semantic Caching & LLM Memory
24mConversational Search with LLMs
17mCLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
09mIntegrating LLMs into Front-End Apps for Enhanced UX