THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression
Talkat 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24) · 2024-04-16..04-18·
summary
Proposes Tensor Homomorphic Compression that makes compressed gradients directly summable (homomorphic property) so parameter servers or programmable switches perform only integer adds. Prototype uses quantization with a randomized Hadamard transform, nonuniform indices + lookup tables, and a Tofino2-based in-network aggregation. Evaluation shows 4–8× compression ratios, PS decompression previously raised PS time by ~23.5%, and THC reaches target accuracy ~1.40–1.47× faster with INA and ~1.28–1.33× faster with a software PS.
AI-written summary · about
chapters (11)
- 0:00THC introduction and motivation
- 1:20Distributed deep learning challenges
- 3:27Limitations of gradient compression
- 6:12Homomorphic compression benefits
- 7:16THC algorithm design
- 11:13THC prototype and in-network aggregation
- 12:57Experimental setup and metrics
- 13:40Time to accuracy with THC
- 15:24Additional results and impact
- 15:44Conclusion and future work
- 16:05Q&A
related talks
28mNext-generation frameworks for Large-scale Machine Learning
11mHow Do We Compress Giant AI Models to Run at the Edge?
01h 13mTutorial: Federated Optimization, Part II
30mCompactifAI: Quantum-Inspired AI Model Compression
17mData-Parallel Differentiation by Optic Composition
36mHigh-Performance Distributed Tensorflow
01h 14mDistributed and concurrent optimization for machine learning
23mBatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning