Data Preprocessing for ML at Production Scale
summary
Describes a production pipeline that replaces Python list dict hacks with a streaming, composable DataPipe system and an Arrow-backed DataFrame to scale to terabytes and petabytes, claiming big speedups in Meta training workloads. TorchData supplies composable data pipes, archive and S3 readers, shuffle and zip by key and easy backend swaps. TorchArrow exposes a DataFrame API, Arrow memory layout, Velox CPU backend, graph tracing and one-call tensor conversion for nested arrays. A disaggregated DPP service reads mini‑batches, preproc and serves tensors to trainers for real deployments using the Criteo style 14‑column dataset example.
AI-written summary · about
chapters (4)
- 0:00Start
- 9:02Critero Dataset and Transformation
- 10:33Example TorchArrow Program
- 12:25Dis-aggregated data ingestion architecture
related talks
30mSMART DATA PIPELINES FOR AI
34mLeveraging Apache Arrow for ML Workflows
29mStructuring Data for Scalable AI Applications
05mDump the Ducktape: Unblocking data pipelines at scale in healthcare using
37mTaming the Data Tsunami
17mProductionizing Data Science Workloads
43mDemocratizing ML Pipelines using StreamPipes
23mDaft - The Distributed Python Dataframe for Complex Data