Testing the Untestable
summary
Heroku’s Ruby buildpack starts with zero tests and grows into a 4,573‑line, edge‑case ridden system; the cure is black‑box integration tests that actually clone repos, create Heroku apps, deploy them and run commands. A test harness called Hatchet drives git push, VCR records external API traffic, and an eight‑line gem re_retry retries flaky deploys. Parallel runners drop 45s per test suites to 12 minutes for 44 cases, buildpack performance improves ~40%, and complex tools like Puma auto‑tune get verifiable behavior.
AI-written summary · about
chapters (30)
- 0:00Introduction
- 5:52Heroku
- 9:55Blackbox Testing
- 15:19Threaded
- 17:22Troubleshooting
- 17:57Retry
- 18:28Positive Happy Path
- 18:55Bundler Update
- 19:38Deploy
- 20:22Development Tactics
- 20:43Combating Non Determinism
- 21:19Speeding up the iteration cycle
- 21:37The actual tests themselves can take a little
- 22:13Parallel test runner
- 22:36Build pack speed
- 22:54Refactoring
- 23:27Bit testing
- 23:37Unit tests
- 24:31Faster test
- 24:50Code triage
- 25:35Network dependency
- 26:47Web Mock
- 27:53Puma
- 28:39Puma Autotune
- 29:30Puma Test
- 29:55Logs
- 31:10Integration Tests
- 31:28User Testing
- 32:28Heroku Tests
- 32:49Heroku Up and Running
