A Taste of Random Decision Forests on Apache Spark
summary
Demonstrates building decision trees and random forests with Spark MLlib using Scala code that creates labeledPoint vectors, caches training RDDs, and runs grid searches in parallel. Uses the CoverType dataset of 580,000 examples and shows baseline decision tree accuracy ~70% vs random guessing ~38%, hyperparameter tuning (impurity, depth, bins) raising accuracy to ~91% and representation fixes to ~94%, and random forests pushing accuracy to ~96% by voting over many trees.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 3:31Feature Vectors in the Iris Data Set
- 6:33Good Pet Data Set
- 7:40Possible Decision Trees
- 10:00Interpreting Models
- 15:51Building a Decision Tree in MLlib
- 19:40Evaluating a Decision Tree
- 22:56Better Than Random Guessing?
- 26:54Decisions Should Make Lower Impurity Subsets
- 29:12Tuning Hyperparameters
- 38:20How to Create a Crowd?
- 39:06Trees See Subsets of Examples
- 39:51Or Subsets of Features
- 41:06Diversity of Opinion
- 41:39Random Decision Forests
related talks
50mApache Spark vs Machine Learning
39mMachine Learning with Scala on Spark
38mMachine Learning with Clojure and Apache Spark
48mMachine Learning in Azure Databricks
17mMachine Learning at the Speed of Light with Spark
27mMachine Learning at Scale
44mData Science at Scale with Apache Spark
20mDeep distributed decision trees on Apache Spark