Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient...
summary
Torpor implements late binding with model swapping to keep models in host memory and move them to a shared GPU pool on demand, using asynchronous API redirection, GPU runtime sharing, pipelined layer execution and a centralized GPU memory manager to cut swap and allocation overheads to near zero. An interference‑aware scheduler uses GPU interconnect topology and model bandwidth profiles to meet tail SLOs. Prototype on V100 nodes serves hundreds of functions on 4 GPUs with native-like latency, and a pilot saves ~70% user and ~65% platform GPU costs.
AI-written summary · about
chapters (12)
- 0:00GPU-enabled serverless inference
- 1:02Traditional GPU serverless challenges
- 3:09Ideal GPU inference properties
- 3:53Late binding core design
- 5:19Late binding implementation challenges
- 5:39Torpor architecture overview
- 6:42API redirection and model swapping
- 8:29GPU memory management
- 9:33SLO and efficiency policies
- 11:19Torpor evaluation and benefits
- 13:30End-to-end results and deployment
- 14:54Torpor’s advantages summary
related talks
15mStreamBox: A Lightweight GPU SandBox for Serverless Inference Workflow
05mContainerization Beyond CPUs – A Kubernetes-based Serverless Pl...
20mTetris: Memory-efficient Serverless Inference through Tensor Sharing
17mHydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
57mBuilding GPU-accelerated multi-agent apps
21mHarmonizing Efficiency and Practicability: Optimizing Resource Utilization in...
22mUnFaaSener: Latency and Cost Aware Offloading of Functions from Serverless Platforms