Let your Big Data Processing take flight with Apache Falcon
summary
At InMobi the team handles 10 billion events per day across 200 pipelines and builds an orchestration and lifecycle system called Apache Falcon that they open source and promote to a top-level Apache project. Falcon declaratively defines datasets, processes and dependencies, schedules and retries jobs (with exponential backoff), manages retention, replication and snapshots across multiple data centers, and surfaces lineage and alerts. It runs with Hadoop/HDFS, Hive, Oozie and Spark integrations and proves effective at reducing operational complexity and handling late-arriving data.
AI-written summary · about
chapters (10)
- 0:00Start
- 1:34The Simple Data Pipeline
- 4:25Problem Pattern
- 6:36The Three Problem Areas at Falcon Tries To Solve
- 8:09Late Data Arrival
- 11:58Configurable Resource Consumption
- 13:05Slm Monitoring
- 13:55Data Governance
- 22:27Lifecycle Data Management Framework
- 23:02Do We Need To Have a Dedicated Server
related talks
38mLessons Learned from the Migration to Apache Airflow
16mBridging Data Science and Data Infrastructure with Apache Airflow
37mBuilding and operating large-scale streaming applications at King
22mOrchestrate your data pipelines using Apache Airflow
42mCloud based low cost low maintenance scalable data platform
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
22mEffective Migration, Balancing Cost, and Performance
36mElegant data pipelining with Apache Airflow