Apache Hive Replication V3

on YouTube ↗42m

" summary

Hive Replication V3 introduces policy-driven automation by turning REPL DUMP/LOAD into scheduled-query based orchestration, uses a staging area to avoid cross-cluster Kerberos issues, and adds checkpointed incremental dumps so a failed dump resumes where it left off. It records run metrics in sys.replication_metric, synchronizes producer/consumer runs with _finished_dump and _finished_load markers, batches partition adds by default to 10,000 per HMS call, collapses many DistCp jobs into one for warehouse-wide copies, and is in upstream master (targeted release 4.0) and in Cloudera builds.

AI-written summary · about

" chapters (18)

1. 0:00Introduction
2. 0:41Who am I
3. 1:03Agenda
4. 1:59Hive Replication
5. 5:20Export Import
6. 11:34Line of Sight
7. 14:37How Replication Works
8. 16:31Schedule Query
9. 17:29Challenge
10. 22:11Replication Metric
11. 24:14Metadata Replication
12. 26:13Performance Stability
13. 26:32Partition Metadata
14. 29:09Single Top Copy Task
15. 31:22Resilience Debugging
16. 32:32Ripple
17. 33:54Map jobs
18. 36:01Questions

" related talks

• 19mProcessing High Volume Updates in HiveOwen O'Malley · Berlin Buzzwords 2012
• 35mHCatalog, Table Management for HadoopAlan Gates · Berlin Buzzwords 2012
• 59mHadoop Essentials for the SQL Server ProfessionalBill Preachuk · 24 Hours of PASS: Summit Preview 2015
• 36mFlink SQL: Unified Streaming and batch for hive integrationRui Li, Jingsong Lee · Flink Forward Global Virtual 2020
• 43mChange data capture in production with Apache FlinkDavid Morin, Yann Pauly · Flink Forward Europe 2019
• 01h 03mThe Future of Apache Hive and Hadoop 2.0: Bigger, Faster, StrongerCarter Shanklin · PASS Business Analytics Conference 2013
• 35mProduction-Ready Flink and Hive Integration - what story you can tell now?Bowen Li · Flink Forward Virtual 2020
• 34mApache Iceberg.Dipankar Mazumdar · Scale By the Bay 2023