Streaming DataFrames: A New Way to Process Streaming Data
summary
An open source Python library Quix Streams exposes a Pandas DataFrame API for streaming, handling checkpointing, state, and PVC-backed storage so stateful pipelines run from an IDE. It connects to Kafka/Redpanda, supports joins and rolling windows, and claims to scale to examples like 30 million values per minute per car. Live demo builds a real-time clickstream heatmap using a 5-minute hopping window with one-second step, produces correct incremental counts, and installs via pip.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:47Why stream processing
- 6:26Stream processing options
- 10:00Quick Streams overview
- 11:31Live demo start
- 14:30Building the pipeline
- 17:21Real-time aggregation
- 18:26Deployment and state
- 19:09Closing and resources
related talks
27mReal-time Event Streaming with Python
40mPyData Stack: Pure Python open source data platforms
29mUnified batch and stream processing in python
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
36mKafka Streaming, the Pythonic Way
32mEnabling real-time insights through stream processing in Python
39mHow Python Drives Modern Data Engineering
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]