Reactive data processing in Python
summary
Pathway provides a Python reactive framework that keeps batch-style code but applies incremental computation so models and predictions update as new data arrives; demo shows a real-time Twitter sentiment app with iterative geolocation and external textblob sentiment calls processing thousands of tweets per second (replay sped ×50) and working live. Example pipelines include an MNIST streaming classifier whose error falls toward ~5%, Pathway is enterprise-tested and now in open Beta.
AI-written summary · about
chapters (10)
- 0:00Reactive data processing intro
- 3:19Streaming data consistency
- 6:30Batch processing failures
- 8:16Reactive ML with Pathway
- 13:50Incremental computation in Pathway
- 17:22Pathway tables
- 19:54Pipelines with Transformers
- 21:17Twitter sentiment demo
- 25:55Reactive design benefits
- 26:36Q&A on Pathway algorithms
related talks
44mShaping the Future of Real-time Data Pipeline
01h 26mData Driven Action: a primer on Data Science
34mData Products, From 0 to data science pipeline
17mR & Python playing nice, in Production
52m100% Big Data. 0% Hadoop. 0% Java.
26mBuilding A Python Service Using Open Source Components That Connect, Transform & Deliver Data In RT
29mDagster: a Framework for Data Processing Applications
40mPython based predictive analytics