Processing High Volume Updates in Hive

on YouTube ↗19m

" summary

A production solution handles 1 TB/day with 15% updates by keeping a base file per bucket and storing edits as separate HDFS files, then using a custom record reader that merges sorted base and edit streams at read time to synthesize current rows. It requires sorting on the primary key, consistent bucketing, compaction scheduled off-peak, and a cut‑off timestamp for repeatable reads. Prototype scales in daily tests but needs compaction tooling and fixes for Hive I/O quirks before wider release.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 5:27Hive Table Layout
3. 9:47Stitching Buckets Together
4. 11:00Limitations
5. 12:02Additional Challenges from Hive

" related talks

• 20mMin and Max Aggregations with Updates in Real TimeMinakshi Korad · Berlin Buzzwords 2022
• 48mContinuous Optimization for distributed BigData analysisKai Sasaki · HighLoad++ Moscow 2018
• 31mBuilding large scale, transactional data lakes with Apache HudiNishith Agarwal · Big Data & AI Toronto 2020
• 43mChange data capture in production with Apache FlinkDavid Morin, Yann Pauly · Flink Forward Europe 2019
• 50mApache Hudi : The Data Lake PlatformVinoth Chandar · ApacheCon@Home 2021
• 42mApache Hive Replication V3Pravin Kumar Sinha · ApacheCon@Home 2021
• 01h 03mThe Future of Apache Hive and Hadoop 2.0: Bigger, Faster, StrongerCarter Shanklin · PASS Business Analytics Conference 2013
• 23mHadoop & Cloud Storage: Object Store Integration in ProductionRajesh Balamohan · The Fifth Elephant 2016