Efficient Large Graph Processing with Chunk-Based Graph Representation Model

on YouTube ↗09m

" summary

ChunkGraph implements a chunk-based graph representation on NVMe SSDs, classifying vertices into tiny, medium and super categories and using a four-layer hierarchical vertex storage (index, small chunks, layered chunks, huge pages) to cut wasted I/O and TLB overhead. It adds chunk layout optimization (degree-aware reordering plus complementary-vertex filling) to reduce fragmented chunks significantly. On a dual‑socket 24‑core server with 1008 GB RAM and several real web graphs, it runs multiple graph algorithms several times faster than an out‑of‑core baseline and a page‑cache based system while lowering I/O and sync overhead.

AI-written summary · about

" related talks

• 32mLarge-Scale Graph Processing on Emerging Storage DevicesChangho Choi, Nima Elyasi, Anand Sivasubramaniam · FAST '19
• 27mData @scale Seattle - Yucheng LowYucheng Low · Data @Scale Seattle 2015
• 50mBig Data on Smallish MachinesGuy Blelloch · Parallel and Distributed Algorithms for Inference and Optimization (Simons Institute, Oct. 21–24, 2013)
• 20mGraphWalker: An I/O-Efficient and Resource-Friendly Graph Analytic System...Rui Wang, Yongkun Li, Yinlong Xu, Hong Xie, John C. S. Lui · USENIX ATC '20
• 39mThorny path to the Large Scale Graph ProcessingAleksey Zinovev · HighLoad++ 2014
• 39mThorny path to the Large Scale Graph ProcessingAleksey Zinovev · HighLoad++ 2014
• 37mGraph Computing at ScaleMatthias Broecheler · Strange Loop 2013
• 50mHigh-Performance Graph Database Built on CouchbaseDmitriy Levanov · HighLoad++ Siberia 2018