Streams to the River: Powering an Online XGBoost Classifier with Kafka
summary
Built an Adaptive XGBoost that learns from a Kafka stream using a fixed-size buffer, an EDWIN drift detector, and three ensemble strategies (push, replace, update) implemented with River and exported via MLflow. Deployed inside Helicon MLOps as a pipeline that joins sensor and delayed feedback with a 12-hour window, computes F1 online, checkpoints model state, and writes inferences back to Kafka. Evaluated on an industrial anomaly task where a single cloud model ranks best and per-plant instances rank third.
AI-written summary · about
chapters (18)
- 0:00Online machine learning intro
- 4:03What is online learning
- 7:11Concept drift explained
- 9:43When to use online learning
- 10:05Online vs offline learning
- 16:58Adaptive XGBoost intro
- 19:09Handling concept drift
- 23:13Testing adaptive XGBoost
- 25:45Deploying with Helicon
- 26:05Data sources setup
- 27:12Streaming pipelines
- 28:17Model lifecycle in Helicon
- 31:35Monitoring data integrity
- 32:19Exporting the model
- 34:29Model inference in streams
- 35:56Building the pipeline
- 37:41Monitoring the solution
- 38:25Drawbacks and future goals
related talks
23mMoving from Offline to Online Machine Learning with River
31mMoving from Offline to Online Machine Learning with River
30mMoving from Offline to Online Machine Learning with River
42mHow to Leverage the Apache Kafka Ecosystem to...
40mHow to leverage the Apache Kafka Ecosystem to productionize Machine Learning
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
20mShip your Machine Learning Application #bbuzz