Scala DSL for ML Training Set Stratification

on YouTube ↗21m

" summary

A Scala DSL for user training-set stratification runs on Apache Spark and expresses targets as counts or percentages, expands rules like country.each, and uses a linear-programming backend via the third-party Scala library optimist. It formulates constraints, supports reference distributions and an allowed error margin, and typically solves real examples in seconds (one demo reduces 1.6M rows to 1.26M to meet a 70/30 target with a small error in about 27s).

AI-written summary · about

" chapters (10)

1. 0:00Introduction and motivation
2. 3:46ML pipeline and Scala benefits
3. 5:33Why stratification is needed
4. 7:42DSL building blocks explained
5. 10:17Target counts and percentages
6. 12:26Error margin and reference distributions
7. 13:49Linear programming approach
8. 15:36Live demo in Zeppelin notebook
9. 17:39Complex queries and performance
10. 18:24Q&A and closing

" related talks

• 39mMachine Learning with Scala on SparkJose Quesada · Scala Days Berlin 2016
• 42mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 50mApache Spark vs Machine LearningPatryk Pilarski · Confitura 2019
• 44mApache Spark and ML workflowsJuliet Hougland · Jazoon Zurich Autumn 2017
• 33mScala DSLs and Probabilistic ProgrammingJoe Wingbermuehle · Strange Loop 2018
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler · GOTO Chicago 2016
• 39mA full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and consJose Quesada · PyData Berlin 2016