Building a Coud Native Data Pipeline for Your Real-Time ML & AI Apps
Talkat Data on Kubernetes Day NA 2025 · on Monday, 2025-11-10· at 16:45· Building B | Level 2 | B211-212·
summary
Practical blueprint for a cloud‑native real‑time ML pipeline: use Apache Pulsar for low‑latency ingestion and CDC feeds, stream into a streaming SQL engine like RisingWave (or Bytewax) that exposes materialized views for sub‑second analytics, and augment prompts via RAG with a vector DB before calling an LLM. Claims: end‑to‑end real‑time ingestion, SQL‑driven continuous queries, integrations with Kafka/Redpanda, and support for exactly‑once processing; tradeoffs include harder debugging and the need for strong observability.
AI-written summary · about
chapters (8)
- 0:00Cloud-native AI pipelines
- 4:17Event-driven AI data flow
- 10:59Data pipelines and ingestion
- 17:01Data transformation and ETL
- 18:26Kubernetes data pipelines
- 21:38Rising Wave analytics
- 25:35AI use cases with streaming
- 27:40Resources and closing
related talks
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
08mReal-Time Data Systems: Speed, Scale, and Resilience
19mBuild Real time, AI powered Apps
19mScalable Data and AI Solutions with Cloud-Native Architecture
30mFive Challenges in Real-time Stream Processing and Five Solutions
38mReal Time ML Pipelines in Multi-Tenant Environments
41mDeep Dive into Building Streaming Applications with Apache Pulsar
14mFast Data on-Ramp with Apache Pulsar on K8