Extract Transform Load using mETL
summary
mETL is a Python ETL package built for Central European University that loads CSV/JSON/XML into MySQL or PostgreSQL, runs field-by-field transformations in Python, and chains record-level manipulators, filters, expanders and aggregators. The project claims 9 source types, 11 target types and ~30 built-in transforms, is configurable via Yaml files, can auto-generate configs, run directory walks, and create migrations to move or diff databases — used in production for multiple clients and open source on GitHub.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:11About the speaker
- 3:18What is mETL
- 5:47mETL features
- 7:15mETL components
- 10:18Source and target config
- 14:08First mETL example
- 17:27Headers and data types
- 19:55Load to database
- 20:58Field transformations
- 23:51File format conversion
- 25:18mETL for reporting
- 27:48Dynamic Excel sheets
- 29:14Advanced mapping
- 31:26mETL manipulation
- 33:59Modifiers and filters
- 37:15Why mETL is easy
- 38:40Aggregators and expanders
- 41:12Nested data handling
- 46:19API data processing
- 49:39Batch processing
- 50:23mETL migrations
- 55:02Compare migrations
- 58:19Database migrations
- 1:00:04Joining data sources
related talks
16mETL vs ELT: Why the T has shifted
47mMulti Database Colaboration and Extract - Transform - Load with HTSQL
55mThe ELT pattern in action with DBT and SQLMesh
15mNext Gen ETL Generators
28mData Plumbing 101 - ETL Pipelines for Everyday Projects
24mMetadata driven framework for data discovery and data quality
57mCase Study, Converting the CATME application from MySQL to Postgres
36mPython and ETL Schemas: The essential pivot for Artificial Intelligence