Performant Training for All Workstreams
summary
Latitude measures end-to-end wall-clock throughput rather than GPU utilization and instruments Ray Data + Ray actors to break training into load, transfer, and train stages. A run tracker and live dashboard push per-stage throughput and queue-depth to BigQuery and show one-click profiles; lightweight background profiling of 2–3 actors for a few steps surfaces kernels. Across ~2,000 weekly experiments with ~40–50 engineers this locates shifting bottlenecks and surfaces throughput not utilization, yields 2–5x gains on some workloads and uses an async actor profiler for production-safe profiling.
AI-written summary · about
chapters (15)
- 0:00Introduction and team intro
- 1:11Latitude AI overview
- 3:15Data pipeline and model workloads
- 4:40Ray Data in ML stack
- 6:45Challenges with baseline comparisons
- 7:51Choosing throughput over utilization
- 8:58Identifying bottleneck stages
- 11:05Run tracker and throughput monitoring
- 13:18Async actor profiler benefits
- 15:46Tracking bottlenecks over time
- 16:29Metric store and fleet dashboards
- 18:37Nightly regression dashboards
- 19:02Agent-accessible optimization loops
- 20:03Key takeaways summary
- 21:14Q&A on metric storage
related talks
01h 00mSystems @Scale Summer Q&A Week 4
40mPerformance Testing Done Right
01h 03mMonitoring Performance on Massive Games.
36mProgrammatically Performant
39mProfiling and Benchmarking 101
48mFantastic Performance and Where to Find It
29mA New Kind of Analytics: Actionable Performance Analysis
27mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors