Hive on Spark #bbuzz
summary
Ports Hive to run on Spark as an alternative execution engine, keeping existing Hive operators and SQL dialect while replacing MapReduce jobs with Spark transformations; typical Hive queries that used to spawn about 10 MapReduce jobs collapse into far fewer stages and often a single Spark job. Uses Spark on YARN with long‑lived executors and dynamic allocation, tested on an 8‑node cluster (32 cores, 64 GB RAM, 10 Gbps) against TPC‑DS at 320 GB and 4 TB, showing consistent speedups over Hive on MapReduce and mixed results versus Tez but overall meaningful gains.
AI-written summary · about
chapters (9)
- 0:00Introduction and speaker background
- 1:33Hive Spark project overview
- 8:26Technical deep dive into Hive on Spark
- 12:40Eliminating redundant map reduce phases
- 16:53Optimizing shuffle phases in Spark
- 20:19Process lifecycle improvements with Spark
- 22:03User experience and installation
- 27:23Performance benchmarks and tuning
- 32:48Q&A and project availability
related talks
01h 03mThe Future of Apache Hive and Hadoop 2.0: Bigger, Faster, Stronger
15mOptimizing Query Performance in Big Data Systems
55mSpark Architecture
52mSolving Analytical Problems using Apache Spark
30mBeyond Experimental: Spark on Kubernetes
53mLord of the Spark
19mApache Beam pipelines at 100TB+ scale using Apache Spark
42mApache Spark on planet scale