How to build Big Data Pipelines for Hadoop using OSS
summary
Shows a concrete approach: use HDFS + MapReduce with Pig, Hive, Cascading and Spring for Apache Hadoop RC1 to declare jobs, auto-start a Hive server, run Pig scripts, and submit jars programmatically. Demonstrates a working pipeline that tails logs, stages files to HDFS, triggers scheduled MapReduce via Spring Scheduler and SpEL-produced output paths, and writes results to MongoDB/Redis. Spring Batch coordinates chained Hadoop/Pig/Hive steps and STS visualizes the workflow.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 1:45What is Big Data
- 2:40Data Trends
- 5:51Big Data Pipeline
- 12:25Big Data Platform
- 15:26MapReduce
- 16:37MapReduce in practice
- 23:10Spring for Apache Hadoop
- 24:00Spring Framework
- 24:38What does it take
- 26:00Spring for a Patch
- 29:16Pig
- 30:47Hive
- 31:35Spring
- 32:15GDBC Template
- 33:07Filesystem Shell
- 34:14Scripts
- 35:49Scheduling
- 39:17MongoDB
- 40:56Enterprise Application Integration
- 45:21Component Overview
- 46:37Example
- 52:52Endpoints
- 53:36Batch workflows
- 53:48Spring Batch
- 56:06Hadoop Workforce
- 57:57Spring Batch Workflow
- 59:29Show Tasks
related talks
01h 21mIntroduction to Spring for Apache Hadoop
42mBuilding Robust Streaming Data Pipelines with Apache Spark
47mSolving Real & Big (Data) Problems Using Hadoop
01h 21mSpring XD for Real-time Hadoop Workload Analysis
44mAutomating a Big Data Platform with Jenkins
18mHadoop Ecosystem
21mSerious Network Analysis Using Hadoop & Neo4j
50mWalking Through Spring Cloud Data Flow