Agentix: An Efficient Serving Engine for LLM Agents as General Programs
summary
Agentix intercepts LLM calls from programs, tags requests with program and thread IDs, and maintains a process table of program-level stats to drive decisions. It implements preemptive MLFQ scheduling and engine-aware load balancing that favors KV cache locality. Evaluated on chatbot, React-style agents, and Monte Carlo tree search, Agentix improves program throughput 4–15× versus vLLM at equal latency, uplifts multi-engine tail latency ~2× and average throughput ~1.4×, and cuts offline makespan up to 40%, demonstrating practical gains.
AI-written summary · about
chapters (10)
- 0:00Agentix and LLM agents
- 3:11Serving dynamic agent programs
- 5:39Agentix serving engine
- 7:04Agentix frontend and process table
- 8:53Load balancing and cache locality
- 10:14Preemptive scheduling for LLM calls
- 12:22Priority functions and metrics
- 13:25Evaluation setup and workloads
- 14:29Performance results
- 15:11Conclusion and Q&A
related talks
30mDesigning Fast & Cost-Efficient Agents: The Power of Caching
49mAI Agents: Collaboration in Real-Life Applications
26mContext Engineering: Practical Techniques for Improving Agent Quality Today
50mIntelligent Agents, the New Frontier of LLMs
45mA comprehensive showcase of AI Agents and Agentic LLMs
36mLLMs and AI agents demystified
29mAgentic AI: What It Is and Why It Matters
34mAI Agents: Transforming Autonomous Decision-Making