AI Platform Q&A
summary
Uber’s generative AI platform plugs in multiple detectors — self check GPT, Nemo Gotta Raya framework and a module in Nama index — and reports mixed results on hallucination detection rather than a silver bullet. Multi‑tenancy uses containers per model for host isolation, a unified hierarchical config model for safe overrides, and quota/demand control to avoid noisy neighbors. Large models split into modules to handle huge embedding tables and different batching needs. Snapshot transitions follow capacity shift then traffic shift to preserve consistency, and job‑level chargeback shows resource usage and ROI.
AI-written summary · about
chapters (13)
- 0:00Q&A
- 0:24Project learnings and metrics
- 1:05Detecting hallucinations
- 2:09Multi-tenancy for models
- 3:53Optimizing AI workloads
- 5:19Hardware for inference
- 6:43Security in AI platforms
- 8:29Real-time traffic handling
- 9:56Splitting models
- 12:07Pipeline frameworks
- 13:13Optimization priorities
- 14:39Cloud FinOps
- 17:35Closing remarks
related talks
30mAI Application Architecture: Building for Scale and Efficiency
17mKeynote: Machine Learning & A.I. At Uber
26mUber’s Intelligent Insights Assistant
15mChallenges in moving Gen AI apps to production
25mMonitoring AI with AI
35mUber's Journey in Building Multi-Modal Foundation Models
01h 03mIt's All About AI
48mNavigating the Divide: Open vs. Proprietary AI