Metorikku: spark ETL made super simple

on YouTube ↗09m

" summary

Metorikku is an open-source library that makes Apache Spark ETL development simpler by letting engineers define complex ETL jobs with simple configuration files and SQL statements. It also provides a built-in infrastructure for creating unit tests, and was applied at Yotpo to broaden Spark access across R&D.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 1:38What is metorikku?
3. 2:35Spark SQL
4. 7:18Metorikku testing

" related talks

• 24mVisual Flow, open-source ETL tool powered by Apache SparkDmitri Pavlov, Alexander Shevtsov · ApacheCon@Home 2021
• 38mEnabling Big Data and Machine Learning for the Masses...Tim Kral · Scale By The Bay 2018
• 15mNext Gen ETL GeneratorsMiljenko Vukovic · DSC DACH 25
• 23mRethinking scalable machine learning with Spark ecosystemAdi Polak · Scale By the Bay 2021
• 14mLogging Apache Spark - How we made it easySimona Meriam · DataEngBytes 2021
• 16mDemocratizing Data Pipelines: Introducing the AI-Powered ETL AssistantHaya Axelrod Stern · hayaData 2024
• 31mKotlin for Apache Spark: Why the world needs one more API for SparkPasha Finkelshtein · Data Love 2021
• 51mMetaConfig driven FeatureStorePiyush Kumar · Data Con LA 2019