Supercharging Operations and Analytics: Using Spring XD to Support Analytics and CEP
summary
Capital One moves from nightly ETL to a streaming pipeline built on Spring Extreme Data (XD), custom Java libraries and Jackson JSON: applications emit annotated POJOs to remote RabbitMQ, XD taps and processors transform and fan out to Splunk, Kafka (secured with stunnel), Mongo and HDFS; data scientists get topics for ad hoc use. The stack handles surprising volume — about 450 GB/day on early apps into a Hadoop data lake now at 2 petabytes — client-side encryption marks PCI fields, provisioning uses Chef/Terraform/Jenkins, and rolling dashboards in Splunk/Tableau/Kibana drive real-time ops.
AI-written summary · about
chapters (36)
- 0:00Introduction
- 1:36About Pivotal
- 2:35Agenda
- 4:13Quotes
- 5:18Example
- 6:58Challenge
- 7:37Journey
- 8:22Data Capture
- 10:02Data Capture Requests
- 11:14Challenges
- 12:44Incremental Improvement
- 13:32Splunk
- 14:05Big Data Tools
- 17:12Spring XD Qualities
- 18:38Spring XD Internals
- 20:22Spring XD Streams
- 23:18Our Architecture
- 29:44RabbitMQ Console View
- 31:20Replica Sets
- 32:21Authentication
- 35:51Data Flow
- 37:05Three Patterns
- 41:27Data Registry
- 44:40Container Groups
- 45:29Building a Stream
- 46:08Kafka Brokers
- 46:35Stream Deploy
- 48:53Client Optimizations
- 50:24Central Rabbit
- 51:15Data Framework Service
- 52:27Stream Entry Point
- 53:29Simple Annotations
- 53:51Encrypted Annotations
- 55:05Multiple Emitters
- 56:46At Audited
- 57:19Config Example
related talks
01h 29mDevelop powerful Big Data Applications easily with Spring XD
01h 21mSpring XD for Real-time Hadoop Workload Analysis
01h 12mImplementing the Lambda Architecture with Spring XD
01h 24mSpring XD - A Guided Tour
30mDelivering the Power of Data Using Spring Cloud Data Flow - Gilbert Lau, Wayne Lund
45mModern Data Architecture
53mBig Data: from mammoth to elephant
30mArchitecting Applications With Multiple Open-Source Big Data Technologies