Simplifying Big Data Management with Apache Falcon
summary
Explains how Apache Falcon models cluster, dataset and process as XML entities to automate pipelines on Hadoop, using Kafka for ingestion and YARN/HDFS for execution and storage; converts declarations into scheduled workflows, handles replication between data centers, enforces retention and archival policies, and triggers reruns on partition-level change detection. Claims reduce reprocessing by keeping intermediate states; cites InMobi scale: ~25 TB ingested daily and ~200–250k Hadoop jobs per day, with examples of pipeline gating and lineage.
AI-written summary · about
chapters (8)
- 0:00Start
- 20:53Dependency and Schema Management
- 33:28Core Features
- 36:43Retry Logic and Delay Handling
- 38:02Where Exactly Is It Deployed
- 42:36Pre-Processing Step and the Post Processing Step
- 50:43Case Study
- 54:31Cluster Definition
related talks
45mModern Data Architecture
47mSolving Real & Big (Data) Problems Using Hadoop
53mBig Data: from mammoth to elephant
24mHow To Remove Data Siloes: Tips & Considerations
29mBig Data Analysis
47mBuilding Modern Data Streaming Apps with Open Source
01h 35mDeep Dive into the Big Data Landscape
42mBuilding Robust Streaming Data Pipelines with Apache Spark