Building A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
summary
Builds a real-time Python chat pipeline that measures message sentiment and filters abusive language using pre-trained model artifacts, connecting front end via websockets to Kafka and Kubernetes for horizontal scaling. Uses Kafka partitions and replicas for durability, a Quix pop-and-sub template to scaffold services, and a Hugging Face pipeline that loads model once in memory to avoid per-request API overhead. Demonstrates stateful rolling-room averages, a Twitter streaming connector, and a live demo that handles spikes and persisted messages (30 million values per car per minute cited as a motivating scale).
AI-written summary · about
chapters (12)
- 0:00Introduction to streaming data
- 1:03Why streaming matters in real-time
- 3:09Three pillars of scalable pipelines
- 7:25Pub/sub architecture explained
- 9:08Demo setup and chat app
- 11:15Building a Python pub/sub service
- 13:05Simple real-time text processing
- 14:54Integrating ML sentiment analysis
- 19:41Stateful processing for room sentiment
- 22:13Twitter integration for Bitcoin sentiment
- 24:49Efficiency and next steps
- 25:29Q&A and resources
related talks
44mShaping the Future of Real-time Data Pipeline
27mReal-time Event Streaming with Python
45mLive build: How to harness streaming data in real time
01h 47mRadanalytics.io streaming applications workshop
42mCloud Native Apache Pulsar Development 202 with Python
01h 25mBuild and Deploy an End-to-end Streaming NLP Insight System
43mSimplifying Real-Time ML Pipelines with Quix Streams: An Open Source Python Library for ML Engineers
40mBuilding Real-Time Applications: Cyclist Crash Detection