Hadoop & Cloud Storage: Object Store Integration in Production
summary
Hadoop now integrates an improved s3a connector that adds multiple auth methods, server‑side encryption and caching; engineering changes fix expensive seek/backward reads and reduce split generation overhead so Hive can cache footers locally. Benchmarks on ~200GB TPC‑DS data running on m4.4xlarge nodes show queries all complete and yield an average 2.5x query speedup and about a 3x metastore calls reduction after parallelizing and batching metadata and tuning split/thread settings. Practical fixes and multipart/upload tweaks are deployed and working.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:26Why Hadoop in the Cloud?
- 1:52Hadoop Cloud Storage Utilization Evolution
- 4:51What is the Problem?
- 6:28Goal and Approach
- 7:10Hadoop-compatible File System Architecture
- 8:09Cloud Storage Connectors
- 9:45Authentication
- 11:30Encryption
- 11:45Performance Improvements
- 13:50Hive Access Patterns
- 15:30Hive - Analyze Column Statistics Improvements
- 16:31Performance Considerations when Running Hive Queries
- 21:38Hive-Test Bench Benchmark Results - LLAP
- 21:55Best Practices
related talks
59mHadoop Essentials for the SQL Server Professional
29mAccelerate your cloud architectures with the right storage options
41mThe New Generation Of Data Stores
01h 08mApache Hudi: A Database Layer Over Cloud Storage for Fast Mutations & Queries
51mBIG DATA STORAGE & PROCESSING: Introduction to Hadoop and Its Ecosystem
50mApache Hudi : The Data Lake Platform
01h 01mOK S3
50mSQL 2022 and S3 – Dell’s overview of consuming object data via Microsoft SQL