Controlling Memory Footprint of Stateful Streaming Graph Processing
summary
Two memory-saving iterative models are implemented inside graph code: the Selective Stateful Iterative Model keeps intermediate state for a tunable subset (choose top in-degree vertices) and the Minimal Stateful Iterative Model discards most state by using the distributive update property to compute changes directly. Evaluation on seven algorithms and five graphs shows tracking 20% of vertices cuts memory up to 70% while preserving ~80% of graph code’s speed, and the minimal model raises memory only ~30% while giving up to 8.2× speed over stateless re-computation.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:24Graph Analytics
- 0:35Streaming Graph Processing
- 1:25Stateful Iterative Processing Model
- 2:13Streaming Graph Systems: GraphBolt & DZIG
- 4:24Memory-Efficient Stateful Iterative Models
- 6:13Selective Stateful Iterative Model: Challenges
- 6:37Selectively Tracking Intermediate State
- 7:42Selective Incremental Processing
- 9:22Distributive Update Property • Computation distributed into sub-computations on subsets of inputs
- 10:20Minimal Stateful Iterative Model
- 11:40Experimental Setup
- 14:41Other Experiments
- 14:54Conclusion
related talks
16mIn support of workload-aware streaming state management
51mMemory Masterclass: Make Your AI Agents Remember What They Do!
09mEfficient Large Graph Processing with Chunk-Based Graph Representation Model
53mData Deduplication For Redundant Graph Structures Files
49mIncremental Computation
20mGraphWalker: An I/O-Efficient and Resource-Friendly Graph Analytic System...
27mWhat Does Dynamic Optimality Mean in External Memory?
06h 54mTopical Edition on Databases - Track 1