Testing Salesforce at Cloud Scale
summary
Salesforce claims a working CI that runs hundreds of thousands of tests by parallelizing what would be ~3000 serial hours across ~2000 VMs plus ~1500 Selenium boxes and occasional EC2 m2.2xlarge instances using VMware link clones and EBS AMIs. Full change lists sometimes run ~100000 tests producing tens of failures so the system does partial runs to isolate commits. Jenkins augments the homegrown runner with satellite builds that boot a local checkout in ~3 minutes and experiments with PhantomJS headless UI tests and HBase analytics to tame flaky tests.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 3:45Autobuilds
- 5:21Change Lists
- 6:41Test Flaps
- 9:23Cloud Migration
- 12:03Test Architecture
- 13:56VMware vs EC2
- 17:07VM provisioning
- 21:58Selenium
- 27:41Jenkins
- 28:48Satellite builds
- 30:52Running tests in isolation
- 31:41Code coverage reports
- 32:51Jenkins as a service
- 37:50Patents
- 38:21Platform Optimization
related talks
19mSelenium Security Scanner
43mInfinite Space for Your Tests
40mScaling Selenium to infinity using AWS Lambda
35mMassively Continuous Integration: From 3 days to 30 minutes
24mBetter Testing Through Statistics
51mAutomatic testing of a heavily integrated system - a case-study
24mSupercharging CI: Test Suite Optimization for Speed, Stability, and Savings
32mBuilding Continuous Integration and Feedback