Mobile performance testing at Slack
summary
Built a CI pipeline that validates every code change for performance by running pre-merge Slack alert checks from Jenkins on device farms, using Firebase Test Lab, Espresso tests and OkReplay to hermetically replay network traffic. Tests collect an n-sized sample per build and compare dev vs master with the Mann‑Whitney U test to turn p‑values into confidence; thresholds and sample size are tuned to hit a noise target (~0.5%). The system runs for ~1.5 years, catches regressions, and still fights flakiness and external outages.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:30Emojis
- 2:10Adhoc approach to performance
- 3:55Visualizing performance
- 4:52Slack alerts
- 5:26Native approach
- 7:28Data analysis
- 7:50Statistics
- 8:23MannWhitney U test
- 9:34Statistical approach
- 12:38Noise
- 13:51Backend
- 15:14Performance Test Job
- 16:48Device Lab
- 17:53Loss of control
- 19:57Device
- 20:47Client
- 22:12Preserve prod
- 23:12Execution
- 23:30Stability
- 24:32Merge blocking
- 25:05The backend
- 25:39Performance data
- 26:39Python default argument issue
- 27:36Python backend
- 27:55Trends
- 29:20Slack
- 30:10Historical trend
- 31:22Debugging
related talks
44mE2E2U: Slack’s journey to developer-driven end-to-end testing
30mImprove app performance and stability with Firebase
44mRock-Solid UI and Instrumentation Testing
40mAndroid performance monitoring with automation
22mThese 5 Firebase tools will help you test and run your app with confidence
43mUI testing is so easy - said no developer ever
30mHow We Moved from Android Studio to Slack
30mMeasure What Matters with Jetpack Benchmark