StreamBox: A Lightweight GPU SandBox for Serverless Inference Workflow
summary
StreamBox replaces heavy per-function CUDA contexts with concurrent GPU streams plus an auto-scaling memory pool that profiles models offline, does lazy allocation and eager recycling, and recompiles kernels to enable software-side GPU partitioning. It implements transparent intra-GPU zero-copy communication and splits transfers into blocks scheduled by an I/O daemon to realize PCIe bandwidth sharing. Evaluation on real workloads reports up to 82% GPU memory reduction and up to 6.7× throughput improvement versus state-of-the-art serverless inference systems.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:31Default sandbox limits
- 5:00GPU strings alternative
- 6:26Challenges with GPU strings
- 8:33StreamBox architecture
- 9:16Three key components
- 11:48Performance gains
- 14:00Summary and future
- 14:42Q&A
related talks
21mOptimizing Model Serving on Kubernetes With Model Streaming
26mEfficiently serving LLMs at scale.
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
08mLightning Talk: GPU.x - Dynamic GPU Fraction and Sharing on ML Mainstream Framework
32mAnyscale's Ray Data: Revolutionizing Batch Inference
36mAccelerating High-Performance Machine Learning at Scale i...
20mGPU Cloud Deployment Without Leaving Your IDE