Specializing Data Access in a Distributed File System (Generative Pearl)
summary
DDLoader embeds HDFS partitioning and distribution into generated programs using LMS, fusing data-access and computation so code knows exact block paths at compile time. It emits generated C that uses mmap (or read) and exposes MPI intrinsics for parallel runs. Results show up to 12.56x single‑thread speedup vs Hadoop Java and about 39x for 64 processes; limitations: no automatic parallelization, requires explicit MPI use and recompilation if topology changes.
AI-written summary · about
chapters (8)
- 0:00Specialized data access
- 3:00Distributed file system challenges
- 6:51HDFS file layout
- 10:24Staging and LMS
- 13:32Parallel computation with MPI
- 16:02Hadoop performance
- 17:46Flare integration future
- 18:31Q&A
related talks
01h 06mBest practices when accessing Big Data or any other data!
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...
02h 05mBig Data and the Data Lake from the Analyst Perspective
41mParallel and distributed big joins in H2O
29mBig Data Analysis
35mScaling Parallel Algorithms to Massive Datasets using Multi-SSD Machines
31mHadoop Vectored IO: your data just got faster!
45mModern Data Architecture