Building Billion Node Graphs for Machine Learning
summary
Builds production-ready graph pipelines using Spark dataframes for cleaning and graph generation and FastRP embeddings for node vectors. Raw URL-level graphs reach over 2+ billion nodes and 6 billion edges for a few days of traffic; domain graphs are ~200M nodes and 400M edges. Initial Spark pipeline takes about three days to produce embeddings; a custom in-memory message-passing engine (partitioned workers, ingester, controller) fits billion-node embeddings into terabytes of RAM and promises up to 100x faster prototype runs, successfully unblocking researchers.
AI-written summary · about
chapters (31)
- 0:00Introduction
- 1:15Rorys background
- 2:14AD Tech
- 4:29Ad Targeting
- 5:31The AI Lab
- 13:13Building a graph
- 13:37Design constraints
- 16:01Tools
- 17:22Bring it all together
- 17:54Data cleaning
- 18:58Performance
- 20:25Notebooks
- 21:51Spark
- 22:38Overengineering
- 23:39Framework
- 24:01Notebooks Spark
- 26:40Optimus Cirrus
- 27:24Research Papers
- 27:37Implementing Research Papers
- 28:37Whats Wrong with Version 1
- 30:49How Much Space
- 32:35Memory Requirements
- 33:52Loading Data Computer Fast
- 35:22Distributed Systems
- 36:44Worker Engines
- 37:31Silver Message
- 37:59Message Passing
- 38:36Current Status
- 39:52Conclusions
- 40:48Build what you need
- 41:48Power and storage
related talks
39mThorny path to the Large Scale Graph Processing
39mThorny path to the Large Scale Graph Processing
13mMarius: Learning Massive Graph Embeddings on a Single Machine
28mLarge Scale Graph Mining with Spark: What I learned from mapping 15 million websites
01h 41mGraph Intelligence: Enhance Reasoning and Retrieval Using Graph Analytics
21mBuilding a Graph of all US businesses using Spark technologies
15mGraph feature engineering with Neo4j and Amazon SageMaker
16mDeep Stream Dynamic Graph Analytics with Grapharis