Faster, Cheaper, Leaner: Horizontally Scaling a CI Pipeline
summary
Migration from Jenkins on EC2 to CircleCI cuts idle VM waste and halves monthly infra spend (from about €2.5k) by swapping VMs for containers, using timing-based splitting from JUnit XML to balance shards, running cucumber UI tests at 40x concurrency, and moving heavy prep to single jobs. Warm builds drop from ~45 core-hours / 90 GB RAM and ~15 minutes per shard through shared caches keyed by the dependency lockfile (dependency cache key), flaky tests are fixed by randomized seeds and bisecting, and live debugging via SSH into builds speeds root-cause fixes.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:393scale Timeline
- 2:13Open Source projects need CI
- 3:51Build Time
- 4:50Random test failures
- 5:46Build Queues
- 6:35Jenkins maintenance
- 8:15AWS Costs
- 9:45Concerns
- 11:25Split by timings
- 12:33Rerun from failed
- 13:22Debug Cl failures
- 17:50Dirty State
- 18:42Reliance on other tests
- 19:38Tips how ensure test reliability
- 20:55Steps to debug test order dependencies
- 23:06External dependencies
related talks
31mCI/CD Agility and Controlling Pipeline Sprawl
41mContinuous Integration Tips & Tricks
01h 02mContinuous Integration: Tips & Tricks
35mOptimizing Your CI: Lessons Learned from a Successful Jenkins Rebuild
30mContinuous Integration; Tips & Tricks
27mThe Need for Speed (in your CI/CD setup)
35mJenkins Scaling and Organization for an Efficient CI
32mPractical Tips & Tricks for CI/CD Success