Enabling Big Data and Machine Learning for the Masses...
summary
Medium replaces mixin-heavy Scala Spark jobs with a dependency-injected platform that treats extraction and loading as services and leaves product teams to write only transformations. It uses Spark, Protobuf, S3, Redshift, Dynamo, MySQL, SQS and an AppBuilder/Guice-style injector to inject plugins, supports about 35 point-to-point source/destination combinations, and ships mocking data extractor and asserting data loader test hooks so teams get end-to-end tests without real databases. Result: clearer config, more tests, three active product teams and even a non‑developer shipping a pipeline.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:35What Do We Do @Medium?
- 2:14Where Does Big Data and ML Fit?
- 7:00Configuring an ETL Job, Continued
- 7:50Writing an ETL Job: Problems
- 12:02Writing a NEW ETL Job
- 14:56Behind the Scenes
- 16:26Reimagine Spark Config: Polymorphism
- 19:30Configuring a NEW ETL Job, Continued
- 20:42Extensible Platform Through Plugin Architecture
- 21:53Data Loader Service with Dependency Injection
- 22:24Data Loader Plugins with Dependency Injection
- 23:29Data Extractor Service with Dependency Injection
- 24:03Testing: Revisit Anatomy of ETL
- 25:18Mocks for Testing: Data Extraction
- 26:29Mocks for Testing: Data Loading
- 26:52Testing Framework
- 29:55Take Aways
related talks
39mMachine Learning with Scala on Spark
01h 48mBig Data made easy with a Spark (3.0)
31mProductizing Machine Learning over Big Data with AWS tools
23mRethinking scalable machine learning with Spark ecosystem
30mSMART DATA PIPELINES FOR AI
34mBig Data, Small Code ... #bbuzz
35mPySpark : Combining Machine Learning & Big Data
45mBuilding a Feature Platform to Scale Machine Learning