How Intuit Streamlined AI/ML Inference Workflows on K8s
summary
Numaflow is a Kubernetes-native, serverless platform that runs real-time inference at Intuit scale — about 4 million models and 60 billion predictions per day — by providing built-in Kafka/Pulsar/SQS/HTTP connectors, forcing ML code into a single UDF per vertex (Python/Java/Go/Rust/TypeScript), and supporting GPU node selectors. It autos-scales per vertex using autoscaling on consumer lag and pending messages, removes boilerplate consumers, and shows a working image-caption demo using an LLM from Hugging Face and a production pipeline with vertices scaling independently (one at 23 pods).
AI-written summary · about
chapters (14)
- 0:00Intuit’s AI/ML workflows
- 1:45Scale and technology
- 3:09Inference on streaming data
- 4:13ML teams’ challenges
- 5:39How Numaflow solves these challenges
- 7:03Streaming inference architecture
- 8:48Live demo: image recognition
- 11:20Pipelines with custom resources
- 13:07User-defined functions
- 14:10Complex multi-source pipelines
- 15:13JSON payloads in pipelines
- 16:57Anomaly detection pipeline
- 17:39Scaling pipeline vertices
- 18:00Community uses and OSS
related talks
24mInference on Streaming Data at Scale at Intuit
28mAccelerating AI at Scale with Data
23mAnalyzing Operational Data at Scale Using ML at Intuit
35mAdvanced Model Inferencing Leveraging KNative, Istio & Kubeflow Serving
17mDevOps for Private AI: A Scalable Platform Using Kubeflow
14mKubeflow: Where ML Meets Kubernetes
54mEconomics and Best Practices of Running AI/ML Workloads on Kubernetes
27mKubeflow for enabling AI Powered Drug Discovery and development...