Architecting Sub-Second Analytics Pipelines
summary
Claims a sub‑second pipeline that hits under 10 ms latency using concrete components: Kafka ingestion (≈2M events/s, ~10 ms) feeding Flink submillisecond processing, a real‑time feature store and optimized model serving with automated retraining. Reports: realtime analytics gives 35% faster insights and 42% efficiency gains; personalization lifts sales ~18% and AOV ~12%; fraud systems cut losses ~27% (~$15M/yr for a midsize bank). Addresses practical hurdles—15% out‑of‑order events, solved with sliding windows and watermarking—and shows the architecture deployed and working.
AI-written summary · about
chapters (11)
- 0:00Introduction and Speaker Background
- 0:38Overview of Real-Time Machine Learning
- 1:04Data Explosion Challenge
- 2:57Batch vs. Streaming Processing
- 7:21Real-Time Fraud Detection in Financial Services
- 10:18Real-Time Personalization in E-Commerce
- 13:53Predictive Maintenance in Manufacturing
- 16:11Technical Challenges in Real-Time ML
- 22:04Choosing the Right Streaming Framework
- 25:03Implementing Real-Time Architecture
- 25:58Key Takeaways and Conclusion
related talks
08mReal-Time Data Systems: Speed, Scale, and Resilience
13mBuilding Financial Fortresses: Real-Time Fraud Detection
47mReal-time Fraud Detection Challenges and Solutions
51mReal-time Fraud Detection: Challenges and Solutions
13mBuild Real-Time Analytics Platforms for Financial Crime Detection
18mReal-time Experiment Analytics at Pinterest with Apache Flink
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
13mScaling AI Systems for High-Volume Transaction Processing