Streaming DataFrames: A New Way to Process Streaming Data
summary
Library called Quix Streams presents a Python-first way to process Kafka streams without a JVM, using Docker + Kubernetes for containers and Kafka changelog topics for state so compute scales horizontally. Demo builds a crash-detection microservice: ingesting ~20 million numeric values per minute use case, computing an accelerometer magnitude, applying a 2s hopping window that emits every 250ms, and publishing alerts — all running locally in VS Code and debuggable in one runtime.
AI-written summary · about
chapters (20)
- 0:00intro
- 0:20preamble
- 0:30hello, nice to meet you!
- 0:41racing background
- 2:08agenda
- 2:31why strean processing?
- 4:19stream processing architecture
- 5:12kafka producer and consumer apis
- 6:07stream processing frameworks
- 9:15stream processing library example
- 9:35debugging
- 9:54jar files...
- 10:11udfs are nasty
- 10:26our approach to stream processing
- 11:10goals
- 11:33is there a third way?
- 12:08quix streams - sdf
- 12:45goal
- 13:16demo
- 20:09try quix
related talks
43mSimplifying Real-Time ML Pipelines with Quix Streams: An Open Source Python Library for ML Engineers
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
39mStreaming doesn’t have to be hard
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
37mIntroducing Kafka Streams, new stream processing library ...
03mInterview
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
01h 00mIn Depth Kafka Streams