HyCache: Hybrid Caching for Accelerating DNN Input Preprocessing Pipelines
summary
Hybrid Cache automatically profiles ~1% of a dataset, models fetch/decode/augment costs, and uses an ILP solver to pick per-stage, tier-aware placements that allow partial caching across RAM and disk while enforcing exclusive tier placement (no duplicate copies). It estimates working-memory needs, populates memory and disk caches, and emits a pipeline that ingests from cached stages. Across diverse image and voice pipelines it reports end-to-end pipeline speedups ranging from 1.11× to 10.1×.
AI-written summary · about
chapters (8)
- 0:00DNN preprocessing pipelines
- 2:50Preprocessing stalls impact
- 4:34Caching approach limitations
- 6:40HyCache design principles
- 8:08HyCache automated framework
- 10:55HyCache implementation
- 12:24Performance results
- 13:31Summary
related talks
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
17mApproximate Caching for Efficiently Serving Text-to-Image Diffusion Models
18mPreparation Meets Opportunity: Enhancing Data Preprocessing for ML Training With Seneca
19mCachew: Machine Learning Input Data Processing as a Service
18mTierTrain: Proactive Memory Tiering for CPU-Based DNN Training
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed Caching
16mGL-Cache: Group-level learning for efficient and high-performance caching
23mHeterogeneous Pipelines as the Unlock for ML at Scale