FPGAs for Highest Performance Inference
summary
FPGAs claim lower end-to-end latency by avoiding GPU-style batching and operate well at batch=1, letting pipelines start producing outputs immediately. Reported device quotes: U200/U250 list ~18/33 TOPs while T4/P40/P4 quote higher TOPs but drop in usable throughput under memory and power limits. Example results: GoogleNet inference at batch=1 shows higher images/sec on U200/U250; a speech-to-text net runs about twice as fast on FPGA versus GPU, and a YY Inc. deployment achieves 10x energy improvement and 3.3x performance improvement after consolidation, enabled by pruning and compression.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker intro
- 1:54Why use FPGAs for inference
- 2:56Training vs inference in neural nets
- 5:01Rate of AI innovation problem
- 7:51FPGAs for low-latency performance
- 12:08Pruning and compression in FPGAs
- 14:13Benchmarking FPGAs vs GPUs
- 17:43Whole app acceleration with FPGAs
- 20:56Case study: YY Inc. success
- 21:59FPGA advantages summary
- 22:26Q&A on price performance
related talks
01h 52mNeural Network Accelerator Design
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
01h 54mDNNs on FPGAs
30mAccelerating Neural Networks for Vision Systems via FPGAs
09mAccelerating compute with software defined hardware (FPGA's)
44mArchitecture Effects When Interacting with GPU Technologies
14mFine-tuning giant neural networks on commodity hardware with automatic pipeline...
13mFENIX: Enabling In-Network DNN Inference with FPGA-Enhanced Programmable Switches