Ladder: Enabling Efficient Low-Precision Deep Learning Computing through...
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
Ladder is a compiler that represents evolving low‑bit formats as a first‑class tType and exposes four concrete tensor transforms (slice/map/convert/pad) to decouple storage and compute so layouts and types change per memory layer. It integrates with PyTorch/ONNX, emits TVM/LLVM/ISA code and a bit‑blast operator library, and claims concrete wins: 2× average speedup on W4/A16 workloads, up to 8× on extreme mixed‑precision cases, about 3× on AMD GPUs, a 10% win on A100 over a prior compiler, and end‑to‑end model compile times under one hour.
AI-written summary · about
chapters (9)
- 0:00Trends in deep learning
- 1:31Challenges in low precision
- 3:10Analyzing hardware mismatches
- 5:30Proposed insights and solution
- 6:53The Ladder framework design
- 10:05Implementation and scheduling
- 11:22System integration and tools
- 12:10Evaluation and results
- 15:23Conclusion
related talks
08mHacks to Make LLM Training Faster
32mTaming the Machine: Basics of ML Models Training and Inference Optimization
11mQiMeng-Xpiler: Transcompiling Tensor Programs for Deep Learning Systems with a...
12mLightning Talk: Accelerating Inference on CPU with Torch.Compile
23mThe Race to the Bottom: Low Latency in the age of the Transformer
01h 00mThe Shape of Things to Come
24mOptimizing CPU LLM Inference in PyTorch: Lessons From VLLM
28mFunctional Sparse Tensor Compilation