On challenges and considerations for real time AI processing
summary
The project builds a real-time video inpainting app that removes objects from live streams using a detection/segmentation front end tuned to beer bottles as a demo and a decoupled spatial–temporal Transformer inpainting model. Engineering choices include 8-bit quantization, TensorRT and RTM instant segmentation + TensorRT for ~10 ms segmentation, GPU encode/decode, float16 with torch.compile, and dataset-driven masked-frame training. The deployed pipeline runs at 30 FPS under 200 ms latency and proves practical for edge-like real-time use.
AI-written summary · about
chapters (11)
- 0:00Introduction to real-time AI
- 1:51What is real-time AI
- 3:39Edge vs cloud processing
- 4:47Real-time video inpainting
- 7:34Balancing speed and performance
- 10:04Benchmarking challenges
- 12:57Handling streaming inputs
- 15:06Resource constraints in AI
- 17:54Final model architecture
- 20:50Development process and lessons
- 21:37Q&A session
related talks
47mBest Practices for Real-time Intelligent Video Analytics
21mLeveraging Python for Real-Time Image processing
19mReal-Time AI: Low Latency Solutions for Interactive Applications
18mDeploying ML solutions with low latency in Python
17mAI/ML demo derby
22mThe media processing pipelines behind AI
53mVideo Analytics for mere mortals with NVIDIA DeepStream SDK
20mRealtime Video AI with Diffusion Models