Data Loading for Data Engineers
summary
Describes concrete changes: treat ordering as a first-class artifact with an explicit permutation table (row ids + split ids) instead of implicit per-worker RNG, offers clumped shuffle as a compromise, and measures throughput so teams know whether IO, CPU transforms, or model are the bottleneck. Notes real numbers: cloud setups need 100+ parallel IO requests per worker and modern GPUs consume on the order of megabytes per second per GPU (hundreds of thousands tokens/sec), and reports Lance DB prototypes and caching options that already speed distributed loading.
AI-written summary · about
chapters (9)
- 0:00Data loading challenges
- 4:57Why data loading isn't simple
- 7:03Permutations in pipelines
- 15:14Transformation stages and bottlenecks
- 17:22Performance myths and GPU efficiency
- 23:01Measuring throughput
- 27:25Evaluation metrics
- 30:15Caching strategies
- 32:03Lance DB’s data loading
related talks
34mData Engineering Architecture
43mBeyond Shuffling: Scaling Apache Spark
58mUnderstanding Big Data for Software Engineers
29mBuilding Large Scale ETL Pipelines with Dask
47mThe Loader is a Lie: How to Elegantly Load a Million or More Data Points
02mHow to Manage the Changing Data Landscape
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed Caching
27mTales at Scale: Analytics at 1000 QPS and Beyond