Creating Standard Patterns to Load Data into Yugabyte with Apache Airflow
summary
Shows a concrete path: use existing Airflow MySQL->S3 operator, then rely on Yugabyte’s Postgres-compatible Yugabyte surface and Airflow’s decoupled providers to replace ad‑hoc scripts with a custom Airflow operator that standardizes MySQL→S3→Yugabyte loads. Demo boots Astronomer’s CLI, wires hooks/operators, and runs DAGs; Airflow 2.0’s provider model and the plugin operator handle multi‑table loops and retries so the pattern is reusable, maintainable and proven to work in the demo.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:50We are big fans of distributed databases
- 2:15Airflow loves the data ecosystem
- 2:50Pipeline overview
- 3:53Airflow ecosystem
- 5:27Airflow providers
- 6:37Express data pipelines as code
- 7:30Scenario
- 8:03Data Engineer
- 8:18Connecting to MySQL
- 8:53Extracting from MySQL
- 9:35Loading to you by cloud
- 10:09Postgres compatibility
- 11:05Demo
related talks
01h 59mIntroduction to orchestration ETL processes with Apache Airflow
26mGo with the Flow: Automating Your Workflows with Airflow
01h 18mIntroduction to orchestration ETL processes with Apache Airflow
35mBecome a Conductor with Apache Airflow
22mOrchestrate your data pipelines using Apache Airflow
18mData Modernization Using Confluent and Yugabyte
38mAirflow DAGs Made Easy: Introducing the Astro SDK
34mData Ingestion in Python using Apache Airflow