AMIDST Toolbox: Scalable probabil. machine learning w/ Flink
summary
AMIDST toolbox is a Java open‑source library integrated with Flink and AWS that runs distributed Bayesian parameter learning using variational message passing and a Flink-backed data wrapper. It scales to very large deployments—demonstrated on synthetic data simulating 42 million clients across 2/8/16‑node clusters—and improves a financial default model from about 0.81 AUC (logistic regression) to near 0.99 AUC, with daily/monthly model updates completed within operational time windows.
AI-written summary · about
chapters (18)
- 0:00Introduction and background
- 1:07Use case with BCC financial group
- 2:54AMIDST toolbox overview
- 4:24Key features of AMIDST
- 5:52Financial health prediction model
- 8:18Code structure and data loading
- 9:47Bayesian vs traditional learning
- 11:15Distributed learning with Flink
- 12:01Daily model updates
- 13:28Results and production deployment
- 14:14Scalability analysis setup
- 15:18Performance results and speed-up
- 16:44Integration with other software
- 17:29Concept drift detection use case
- 20:48Correlation with unemployment rate
- 21:31Toolbox resources and licensing
- 22:15Future plans and community call
- 23:39Q&A session
related talks
29mBuilding Resilient AI Platforms: Financial Fraud Detection
36mFlink-JPMML: An open source Flink library for Streaming ML Model Serving
42mApache Flink deep-dive #bbuzz
37mBuild a Flink AI Ecosystem
33mComputing recommendations at extreme scale with Apache Flink
21mIntroducing FlinkML
18mCluster Serving: Distributed Model Inference using Apache Flink in Analytics Zoo
38mAI made easy with Flink + AIFlow