Simplifying Real-Time ML Pipelines with Quix Streams: An Open Source Python Library for ML Engineers
summary
Shows a concrete real‑time ML pipeline built with the open‑source Quix Streams Python library: phones act as producers into Kafka topics, a Collab notebook runs a consumer that uses Hugging Face transformers for sentiment and a pickled model for crash/shake detection, enriches messages with pandas, and publishes alerts. Demo uses an Android Quix Tracker app, processes events live and emits alerts; end‑to‑end works in the demo with observed lab latency around 20–30 seconds.
AI-written summary · about
chapters (8)
- 0:00Introduction and demo warning
- 3:55What is streaming technology
- 15:15Quix Streams library overview
- 19:10Live demo setup
- 24:15Publishing data to topics
- 27:40Sentiment analysis with Transformers
- 34:29Real-time crash detection demo
- 41:22Closing thoughts and call to action
related talks
21mStreaming DataFrames: A New Way to Process Streaming Data
43mDemocratizing ML Pipelines using StreamPipes
41mAdding Generative AI to Real Time Streaming Pipelines
24mUnder Pressure: Applying ML in Real Time
45mLive build: How to harness streaming data in real time
52mMachine Learning Data Pipelines
30mFive Challenges in Real-time Stream Processing and Five Solutions
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT