The Small Model Infrastructure Nobody Built (So We Did)
summary
A working inference engine called SIE is open sourced that actually runs small models with dynamic loading, hot-swap models on one GPU, and a memory-aware least-recently-used eviction policy so dozens of 1–4 GB models share GPUs instead of wasting one GPU per model. It reimplements forward passes and variable-length flash attention to avoid token-padding waste, exposes encode/score/extract primitives, ties routing and KEDA autoscaling to Prometheus metrics, and reports successful tests with Chroma Weaviate LanceDB and production taxonomy-classification use.
AI-written summary · about
chapters (9)
- 0:00Introduction and the gap in small model inference
- 0:53Moving from research to building inference infrastructure
- 2:54Introduction of the Superlinked inference engine
- 4:34The importance of context management for agents
- 7:03Misconceptions: Why more GPUs isn't the only answer
- 9:33The "Yin and Yang" of inference: Model support and infrastructure
- 10:43The challenge of supporting diverse model architectures
- 14:33Deep dive into infrastructure and scalability
- 16:10Conclusion and the open-source launch of SAI
related talks
12mPreview: ML Serving
34mKServe in Production: Scaling Generative Inference the Cloud Native Way
21mOptimizing Model Serving on Kubernetes With Model Streaming
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
24mKServe Next: Advancing Generative AI Model Serving
15mAI Model Inference on K8s
15mDesign Compact Deep Learning Models: Small is the New Big
24mCost and Performance Optimization of LLM Inferencing.