Operating Voice in Marusya: Problems We Faced and Why GPUs Matter
summary
Discussion of running large-scale voice processing in Marusya using GPUs, focusing on hardware and driver reliability problems (cards failing, disappearing, or returning errors) and operational strategies for handling hundreds of GPUs in production. Presents concrete issues encountered with GPUs and practical approaches for detecting, diagnosing, and responding when simple fixes like driver reinstalls no longer work.
AI-written summary · about
chapters (8)
- 0:00Marusya introduction
- 3:22Voice recognition pipeline
- 6:57GPU crash mysteries
- 12:17Memory issues diagnosis
- 15:47Kernel and driver fixes
- 18:42Overheating and cooling
- 22:14GPU in containers
- 27:38Q&A
related talks
16mHolmes: Localizing Irregularities in LLM Training with Mega-scale GPU Clusters
22mKeep HPC Running - an SRE's Guide to Supporting GPUs on Kubernetes
21mEnhancing Runtime Reliability in LLM Training via Fine-Grained Observability
38mInference in Progress… Please Monitor Responsibly
23mFrom Hallucinations To Hardware: Diagnosing LLM Failures
01h 00mSystems @Scale Summer Q&A Week 4
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
23mObservability for LLMs: Understanding What’s Happening Under the Hood