HCatalog, Table Management for Hadoop

on YouTube ↗35m

" summary

HCatalog exposes the Hive metastore to Pig and MapReduce so jobs use a shared table abstraction instead of hard-coded HDFS paths and formats. It supplies HCatLoader and HCat input/output formats, supports partition pruning, per-partition storage formats (text, SequenceFile, RCFile, ORC), and schema evolution where missing columns return null. It adds JMS notifications for new partitions, initial HBase backing, a planned REST endpoints API, Pig 0.11 integration, and ~2% input overhead.

AI-written summary · about

" chapters (7)

1. 0:00Start
2. 5:28Tool Comparison
3. 7:18Hadoop Ecosystem
4. 9:49Tools With HCatalog
5. 13:47Working with HCatalog in MapReduce
6. 26:19HCatalog Project
7. 29:20Questions

" related talks

• 59mHadoop Essentials for the SQL Server ProfessionalBill Preachuk · 24 Hours of PASS: Summit Preview 2015
• 34mApache Iceberg.Dipankar Mazumdar · Scale By the Bay 2023
• 01h 03mThe Future of Apache Hive and Hadoop 2.0: Bigger, Faster, StrongerCarter Shanklin · PASS Business Analytics Conference 2013
• 30mBuilding a metadata ecosystem using the Hive MetastoreNeelesh Salian · Berlin Buzzwords 2021
• 15mThe Future of Hadoop: What Happened & What's Possible?Doug Cutting · Strata Conference New York + Hadoop World 2013
• 23mHadoop & Cloud Storage: Object Store Integration in ProductionRajesh Balamohan · The Fifth Elephant 2016
• 01h 14mHow to build Big Data Pipelines for Hadoop using OSSCostin Leau · SpringOne2GX 2012
• 49mBuilding a 100‑Petabyte Analytics WarehouseAleksandr Mazurov · HighLoad++ Moscow 2018