Data Engineering and AI
summary
Panel explains concrete choices and failure modes: teams use Kubeflow for model orchestration and Argo/Airflow for workflows, prefer Python stacks (PyTorch, TensorFlow) because serving often forces conversion to TensorFlow for speed, and PyTorch 1.0 finally adds serving. Labeling still needs human-in-the-loop (Figure Eight) and startups store huge embeddings in RocksDB-like systems. Testing becomes probabilistic unit tests rather than asserts; one cited case cuts a 15-node Spark cluster to a single node.
AI-written summary · about
chapters (8)
- 0:00Introduction and panel setup
- 5:39Scala vs Python in ML
- 15:17ML software best practices
- 22:51PyTorch vs TensorFlow
- 34:40Tooling and infrastructure
- 41:30Building vs buying ML solutions
- 49:11Data engineering for AI
- 52:20Closing remarks
related talks
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
42mDeep learning beyond the learning
26mEngineering Excellence: Platforms and Architectures for Real-Time AI Analytics
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
42mKubeflow From the End User’s Perspective: The Good, The Bad, and The Ugly
17mDevOps for Private AI: A Scalable Platform Using Kubeflow
26mKubernetes and MLOps for Scalable and Reproducible Generative AI