Simplify Stream Data Processing with Declarative, Scalable K8s Native Technologies
Talkat Data on Kubernetes Day NA 2024 · on Tuesday, 2024-11-12· at 09:45· Salt Palace | Level 2 | 250 A·
summary
NumaFlow is a Kubernetes-native stream platform that runs as a CRD-based pipeline of vertices and edges, connects to Kafka for sources and sinks, and executes UDFs in Go/Python/Java/Rust; demo runs on a local K3s cluster, performs one-minute windowed flat-map and reduce to produce per-category counts and revenue, and shows live charts. It implements persistent inter-buffers to avoid data loss, supports cycle and side inputs, and auto-scale to zero while remaining language-agnostic with CRD-based pipeline semantics.
AI-written summary · about
chapters (11)
- 0:00Numa flow introduction
- 1:24Stream processing challenges
- 4:56How Numa flow simplifies
- 6:41Numa flow in one line
- 7:03Demo setup e-commerce
- 11:35Live demo pipeline walk
- 16:51Pipeline spec UDF details
- 19:42Advanced pipeline patterns
- 21:09Key features community
- 22:12Call to action resources
- 22:34Q&A
related talks
24mInference on Streaming Data at Scale at Intuit
29mMy First Event-Driven Serverless App with Numaflow
21mStreaming DataFrames: A New Way to Process Streaming Data
30mCloudflow: Spark, Flink, and Akka Working together on Kubernetes
36mHail Hydrate! From Stream to Lake
16mLife is But a (Data) Stream: Building Quality Data Pipelines
39mAdvanced Stream Processing on the Edge
47mBuilding Modern Data Streaming Apps with Open Source