Emergence of Real-Time Analytics: Real-time Analysis of...
summary
Capital One builds a real-time pipeline that ingests customer activity events via Apache NiFi into a Kafka-backed durable buffer, processes them with Apache Flink (event-time semantics) on AWS, enriches and flattens records, applies a sliding 2s window that advances every second for aggregations, fires alerts to SNS, indexes results into Elasticsearch for Kibana dashboards, and writes HDFS for batch. Flink replaces batch-only ETL, cuts time-to-insight from days to seconds, and supports fraud detection, duplicate-transaction blocking, and on-the-fly credit-line increases.
AI-written summary · about
chapters (15)
- 0:00Introduction to real-time analytics
- 1:02Capital One’s tech-driven banking
- 4:37Traditional batch analytics model
- 7:50Limitations of batch processing
- 9:53What is fast data?
- 14:07Real-time analytics explained
- 15:10Real-time fraud prevention use case
- 21:31Drivers for real-time shift
- 28:13Evolution of big data technologies
- 31:45Apache Flink overview
- 38:06Business case and architecture
- 43:24Implemented features in Flink
- 48:50General real-time processing pattern
- 51:40Analytics spectrum: batch vs real-time
- 52:42Key takeaways and conclusion
related talks
26mReal time Platform for Second Look Business Use Case Using Spark & Kafka
41mBeyond the Hype: When Apache Flink Solves Real Problems
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
08mReal-Time Data Systems: Speed, Scale, and Resilience
47mExpert Panel: Current & Future State of the Art in Real-Time Data Infrastructure
18mReal-time Experiment Analytics at Pinterest with Apache Flink
09mLet's Get Real: Acting on Data in Real Time
50mThe Real-Time Analytics Stack