Training AI Models on Massive Datasets
summary
Framework DASF combines Dask lazy loading, RAPIDS GPU acceleration and direct-GPU IO (Kvikio / GPU Direct Storage) to train models on seismic SEGY data converted to Zarr/MDIO chunks. Chunking strategy preserves the time axis and uses border overlap to avoid recomputation and excess transfers. Applied to Santos-basin datasets (~200 GB) and smaller public cubes (1.3 GB), pipelines run on 16–32 GB workstations and show about 10× speedups versus baseline, enabling feature-extraction and UNet/XGBoost workflows without materializing terabytes of intermediates.
AI-written summary · about
chapters (9)
- 0:00Training AI on massive datasets
- 2:09Seismic data analysis meaning
- 6:23Preprocessing irregular seismic formats
- 10:45Data dependencies in AI pipelines
- 14:16Chunk-based feature extraction challenges
- 18:15GPUs and storage acceleration
- 20:48Performance improvements results
- 23:21DAS framework introduction
- 25:50Summary and open discussion
related talks
41mAccelerating Data Science with RAPIDS
01h 22mHigh-Performance Data Science at Scale with RAPIDS, Dask, and GPUs
34mRAPIDS: How to accelerate your data science pipeline by orders of magnitude
25mAccelerating fuzzy document deduplication to improve LLM training w/ RAPIDS & Dask
39mAccelerating Data Science with RAPIDS
01h 08mGPU Data Science with Rapids
18mUnleash your Private Datasets to Power up AI
38mRAPIDS- Open Source GPU Data Science