Automated Translation of Functional Big Data Queries to SQL
summary
Translates Spark RDD-style functional queries to SQL using counterexample-guided inductive synthesis plus a novel column-wise decomposition that extracts dependency-free query sketches (DFQs), synthesizes per-column SQL, then merges them. Prototype RDD2SQL handles 100 GitHub-sourced RDD programs, solves 98% solved of benchmarks within one hour (most under seven minutes), yields average 2× speedups and up to 7×, and outperforms prior CIS-based synthesis baselines on number solved and time.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:23Functional Big Data queries
- 1:26Spark vs SQL performance
- 3:17Column-wise decomposition
- 6:49Synthesizing SQL from queries
- 9:15RDD2SQL evaluation
- 11:22Summary
- 12:05Q&A
related talks
14mUDF to SQL Translation through Compositional Lazy Inductive Synthesis
36mData pipelines engineering made simple with Scala
44mNonrelational SQL
30mSimplify Big Data & AI on Spark and Ray with MLSQL
35mQuerying the queries: SQL Metaprogramming in Python
41mHow types can turn a SQL interpreter into a SQL compiler
47mApache Spark Lakehouse Pushdowns with Transpilation
48mCompositional, Expressive, Performant Database Queries: Choose 3