Implementing Piped Processing Language in OpenSearch Via Apache Calcite
summary
OpenSearch integrates Apache Calcite into a new core engine v3 to compile PPL into Calcite RelNodes, reuse v2 scan logic, and generate physical plans with codegen (Janino) and a custom script engine for Calcite expressions. About 300 optimization rules land (predicate pushdown, column pruning, aggregation rewrite) and many operators are pushed down to DSL so cluster-side parallel search is leveraged. Benchmarks show ~30% speedup on Big Five and ~3x on ClickBench; new features include subsearch, join, lookup and extended UDF support.
AI-written summary · about
chapters (5)
- 0:00OpenSearch query languages
- 7:12V3 engine architecture
- 24:47V3 engine optimizations
- 37:37New features benchmarks
- 39:04Future roadmap
related talks
39mBuilding modern SQL query optimizers with Apache Calcite
47mTaking a look under the hood of Apache Flink’s relational APIs
35mGetting Started with the OpenSearch Core Codebase
16mSponsored Keynote: OpenSearch at Atlassian Scale
39mA deep dive into Flink SQL
31mIPASIR-UP: User Propagators for CDCL
35mDesigning Domain-Specific Languages: A Comparison of Three Approaches by Example
57mHigh-Performance Logic Programming on Servers, Clusters, and GPUs