Automated data analyzer using Machine Learning
summary
Movile Wavy processes ~230 million daily billing attempts across high-performance distributed systems and needed automated monitoring and problem detection for platform, product and operator issues. The talk explains how they built an automated data analyzer using time-series analysis and a decision-tree model implemented in Apache Spark, including data extraction from Amazon Redshift and the feature engineering used to optimize model performance.
AI-written summary · about
chapters (13)
- 0:00Project introduction
- 3:02Platform overview
- 7:39Manual analysis challenges
- 10:48Machine Learning solution
- 17:16Model training in Spark
- 21:35Regression experiments
- 23:18Decision tree in Spark
- 28:53Deploying trained model
- 33:12HotE system architecture
- 36:22Lessons learned
- 41:04Results
- 46:01Project impact and simplicity
- 47:05Q&A
related talks
45mAutomated data analyzer using Machine Learning
37mBig Data Analytics, Machine Learning, Real Time Processing
30mBetter data quality with ML & visualization
52mDevil in the Haystack
36mMonitoring High-Performance Platforms With Machine Learning
28mBeyond the Balance Sheet: Transactional Data as a Window to Company Health
26mTelCo Network Analytics & Customer Experience Management on top of Data Lake
16mAnalysing Real time Activity Streams for Security