Optimal Predicate Pushdown Synthesis
summary
Systematically moves filters through stateful fold-based UDFs by synthesizing the strongest pre-filter Q and weakest residual P' that preserve semantics via a bisimulation invariant. The tool Pusharoo enumerates Q from strongest to weakest, checks feasibility with unrealizability certificates, and repairs candidates using root-cause guided predicate repair over a finite predicate universe built from UDF atoms. Evaluated on 150 real pandas and Spark pipelines it finds optimal pushdown decompositions for 100% of benchmarks with median synthesis 1.6s and average 2.4× (up to 100×) speedups.
AI-written summary · about
chapters (11)
- 0:00Predicate pushdown synthesis intro
- 1:06UDFs challenges in pipelines
- 2:52Breaking down UDFs for pushdown
- 4:18Optimal pushdown splits criteria
- 5:01Verifying equivalence via bisimulation
- 7:27Constraint solvers limitations
- 8:09Structured search in synthesis
- 9:56Early unrealizability checks
- 11:21Predicate repair via root cause
- 12:28Evaluation results impact
- 13:10Q&A
related talks
21mFrom boolean towards semantic ...
33mPugs, Poe's and pipelines; An engineering perspective on big-data streams...
13mDPSynth: From Research to Production—Engineering Differentially Private Synthetic...
29mSliding Look-Back Window Assisted Data Chunk Rewriting for Improving Deduplication...
30mSMART DATA PIPELINES FOR AI
34mThe Synthetic Data Flywheel
06mUnlocking graph analytics in DuckDB with SQL/PGQ
42mComposable Parallel Processing in Apache Spark and Weld