Turning a Relational RDBMS Table into a Spark Datasource
summary
A Spark DataSource implementation for Oracle exposes relational tables to Spark with parallel direct access, generating logical splits (single, rows, block, partition, custom) and rewriting SQL per split so each task queries a bounded range. It pushes down predicates and projects columns to reduce data moved, implements partition pruning (demo shrinks five splits to one for a salary range), adds JDBC-type conversion optimizations and connection caching, supports Kerberos/SSL, and currently leaves joins to Spark; alpha code works in the Oracle Big Data Lite VM.
AI-written summary · about
chapters (12)
- 0:00Introduction and agenda
- 1:41Why turn RDBMS tables into Spark data
- 7:22Apache Spark overview
- 10:51Optimizations for RDBMS access
- 13:24Partitioning strategies explained
- 17:44Query rewriting and type conversion
- 22:01Demo setup and environment
- 24:37Predicate pushdown demonstration
- 27:58Partition pruning demonstration
- 30:38Join operations with RDBMS data
- 34:02Current status and future plans
- 38:54Q&A session
related talks
23mOracle Autonomous Data Warehouse and data lakes
38mOn the path to data Nirvana: Supporting OLTP and OLAP on Hadoop
24mOracle's vision for simplifying data-driven apps and analytics
28mAutonomous Data Warehouse and the Data Lake
30mAnalyze connections and find patterns in data with graphs
50mChallenges that everyone struggles while productionizing Apache Spark workloads
20mDatabase developer productivity tips and tricks
28mReliable data integration from applications to analytics