Tetris: Memory-efficient Serverless Inference through Tensor Sharing
summary
Tetris claims an order‑of‑magnitude drop in memory footprint for serverless inference by actually sharing immutable model tensors across co‑located containers and scheduling instances to maximize tensor similarity. It implements an agent that memory‑maps tensors into a tensor store per server, a reclaimer that frees unreferenced maps, and a greedy scaling engine that combines batching and threading. Evaluation over 21 models shows up to 93% memory reduction and up to 30× function density without latency penalty.
AI-written summary · about
chapters (4)
related talks
26mEfficiently serving LLMs at scale.
19mThe Small Model Infrastructure Nobody Built (So We Did)
12mPreview: ML Serving
23mThe Race to the Bottom: Low Latency in the age of the Transformer
16mServerlessLLM: Low-Latency Serverless Inference for Large Language Models
25mMoving Inference to Triton Servers
30mTetrisAI: a new way to play Tetris
39mBuilding model testing infrastructure and scaling AI predictions