Improving interactive querying experience on Spark SQL
summary
Pinterest runs over 700 PB on S3 and ~150,000 daily compute jobs, so interactive Spark SQL needs low latency and isolation. The solution uses Apache Livy in interactive mode, a DB-API Python client (BigPy) to submit statements, and a local pool of lightweight Spark sessions to run DDLs (avoiding ~2 minute YARN allocation). Syntax checking uses Spark’s parser to return errors <2s. Added: Dr. Elephant error classifier, OpenTSDB-backed memory metrics, per-instance load balancing and JSON event logs for debugging.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:54Agenda
- 2:41Data Scale at Pinterest
- 5:48Scheduled vs Interactive Querying
- 7:53Interactive Querying with Spark SQL
- 9:52Design Choices
- 13:06Seamless query submission
- 13:56Faster metadata queries
- 16:07Faster syntax checking
- 17:00Error handling recommendations
- 18:55Resource Utilization Visibility
- 19:58Livy Operational Improvements
related talks
36mIn-memory data pipeline and warehouse at scale
21mAvoiding Spark Pitfalls at Scale
52mSolving Analytical Problems using Apache Spark
11mScaling R with Spark
30mOptimize Spark Jobs Using Its UI
36mNo more struggles with Apache Spark (PySpark) workloads in production
49mAzure Synapse Analytics
35mInteractive data integration with Glue