Extracting Insights from Big Data using Transactional ML with Kafka and Python
summary
Defines Transactional ML as running AutoML on live Kafka offsets to train and update models continuously from joined JSON streams, using a Python low‑code library and a Kafka connector called Viper hosted on GitHub. Architecture uses offsets to roll back tranches, joins dependent and independent streams, and pushes models back to topics for real‑time fraud detection in 15–30s. Visualization streams over websockets; tools run on Linux/Mac/Windows and are free.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:46About me
- 1:34Transactional Machine Learning
- 2:07Agenda
- 3:16Business of Data
- 4:54Big Data
- 8:30Data Stream Challenges
- 9:58Conventional vs Transactional ML
- 11:07Predictive Analytics
- 13:18Transactional ML
- 15:19Key Features
- 16:45Transactional ML vs Conventional ML
- 17:40Evolution of Transactional ML
- 22:39Kafka
- 24:28Kafka Connector
- 25:11Python
- 26:00TML
- 26:37Visualization
- 27:21Kafka TML Architecture
- 28:25Kafka TML Technologies
- 29:16Fraud Detection Architecture
- 30:12Summary
related talks
52mMachine Learning Data Pipelines
16mAnalysing Real time Activity Streams for Security
14mPerformance Optimization in High-Volume Transaction Systems
24mUnder Pressure: Applying ML in Real Time
27mBeyond Logs: Real Time Anomaly Detection with Machine Learning
16mOperationalizing AI in Finance: Real-Time Analytics
36mActive-Active Machine Learning Infrastructures For Training And Prediction
37mThe Role of Machine Learning in Today’s Payment Ecosystems