Spring XD for Real-time Hadoop Workload Analysis
summary
A real system is built: Spring XD ingestion pulls Yarn and MapReduce log4j entries via a rabbitmq log4j appender, a Groovy processor normalizes messages and an HDFS sink writes partitioned files; HAWQ (SQL on Hadoop) plus PXF exposes external tables for interactive queries that find failed attempts, hot hosts, and bad counters. Deployment targets a public 1000-node Analytics Workbench cluster; GemFire XD is proposed for low‑latency dashboards. The demo shows streams, rolling files, SQL queries and job kills working end-to-end.
AI-written summary · about
chapters (14)
- 0:00Introduction and agenda
- 1:56Problem statement and use case
- 7:16Analytics Workbench environment
- 12:17Reference architecture components
- 15:25Spring XD deep dive
- 21:27Hadoop ecosystem overview
- 27:27Architecture implementation
- 34:59Stream definition and demo setup
- 46:55Live demo of log ingestion
- 51:14SQL-based log analysis use cases
- 1:03:36Scalability and high availability
- 1:05:53Future enhancements
- 1:09:25Big Data Lake architecture vision
- 1:13:22Q&A session
related talks
01h 20mSupercharging Operations and Analytics: Using Spring XD to Support Analytics and CEP
01h 14mHow to build Big Data Pipelines for Hadoop using OSS
52mSolving Analytical Problems using Apache Spark
30mDelivering the Power of Data Using Spring Cloud Data Flow - Gilbert Lau, Wayne Lund
38mHow to run Real Time processing on Big Data
19mSupercharging Kafka Applications with Real-time Contextual Insights
36mInteractive Session: Running Flink SQL jobs in production
28mBest of Stream + Batch: When Streaming Analytics Isn’t