CrystalPerf: Learning to Characterize the Performance of Dataflow Computation...
summary
CrystalPerf learns from source code to infer per-operator resource breakdowns (CPU, network, disk) and combines that with non‑intrusive sampling profiler traces to estimate resource time and predict job runtime under configuration changes. It trains classifiers on ~5,000 labeled functions, is implemented as a CLI tool CrystalPerf, evaluates on Spark, Flink and TensorFlow, and predicts performance within about 14% deviation. It requires no instrumentation and flags concrete bottlenecks (an IO bottleneck fixed by enabling NCCL in a case study).
AI-written summary · about
chapters (11)
- 0:00Start
- 3:06Resource Problems
- 4:21Overview: Finding Resource-Time Relationship
- 6:12Outline
- 6:37Resource Classifiers
- 8:08Model Training
- 8:37Why Classifiers Work?
- 9:32Execution Profile
- 10:28Performance Prediction
- 11:34Evaluation: Prediction Accuracy
- 12:29Case Study: Identifying Bottleneck
related talks
14mKPerfIR: Towards a Open and Compiler-centric Ecosystem for GPU Kernel Performance...
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
17mIs advance knowledge of flow sizes a plausible assumption?
40mOptimizing AI/ML Workloads
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
35mResource Monitoring and Optimization with Metaflow
22mCampo: Cost-Aware Performance Optimization for Mixed-Precision Neural Network...
32mA Practical Guide To Benchmarking AI and GPU Workloads in Kubernetes