Optimizing Query Performance in Big Data Systems
summary
Compares Hadoop MapReduce (rule‑based, batch) to Spark (DAG, Catalyst optimizer) and shows Spark cutting iterative query time by using in‑memory caching to avoid recomputing intermediate stages. Recommends partition pruning to limit scans, broadcast joins to avoid shuffling terabyte‑scale tables, and bucketing in Hive to align join keys. Advises cost‑based planning (Presto/Spark) and predicate pushdown to push filters to storage so CPU, memory and network costs drop while latency improves.
AI-written summary · about
chapters (12)
- 0:00Introduction and speaker intro
- 0:28What are Big Data Systems
- 1:30Need for query optimization
- 2:53Key challenges in query processing
- 4:37Techniques for query optimization
- 6:24Cost-based optimization in distributed systems
- 7:47Hadoop vs Spark optimization
- 8:51Practical query optimization examples
- 11:20In-memory caching in Spark
- 12:27Best practices for query optimization
- 13:08Conclusion and future trends
- 14:37Q&A and contact information
related talks
41mOptimising Apache Spark and SQL for Improved Performance
45mFrom Big Data to Big Information
55mSpark Architecture
43mOptimizing Spark applications from simple to complex with examples
47mWhat's Ahead for Big Data?
29mBig Data Analysis
52mSolving Analytical Problems using Apache Spark
39mBig Data Systems Performance: The Little Shop of Horrors