Large Scale Graph Mining with Spark: What I learned from mapping 15 million websites
summary
Built a web graph from a sampled set of September 2017 Common Crawl WARCs, parsed links, and ran Spark GraphFrames label propagation at scale. Starts with ~15 million nodes, after cleaning yields ~4,700 communities with the largest ~3,500 domains and many one- or two-node clusters. LPA converges in about five iterations on these data; implementation is one line in GraphFrames and produced interpretable clusters (bedbug spam ring, Star Wars collectors, online learning sites) without text features.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:52Understanding graphs
- 4:52Unsupervised learning
- 8:29Community detection
- 10:47Graph assumptions
- 14:17Label propagation algorithm
- 17:12Implementation with Spark
- 20:40Results and findings
- 26:18Conclusion and resources
related talks
43mBuilding Billion Node Graphs for Machine Learning
39mThorny path to the Large Scale Graph Processing
39mThorny path to the Large Scale Graph Processing
21mBuilding a Graph of all US businesses using Spark technologies
27mData @scale Seattle - Yucheng Low
41mUsing Clojure+Spark to Find All the Topics on the Interwebs
18mEhtsham Elahi, Senior Research Engineer, Netflix @ MLconf SEA
01h 41mGraph Intelligence: Enhance Reasoning and Retrieval Using Graph Analytics