Selective Testing + Q&A
summary
Addresses test-selection for CI by computing the exact set of tests that could be affected by a change rather than running bazel test ... everywhere. Starts with reverse rdeps via bazel query but covers failures for modified BUILD files, .bzl changes, deleted files, recursive globs and external workspaces. Final solution snapshots query output (target protos) on old and new revisions, compares them via rule proto hashing (a Merkle-style tree) to detect changed targets, and runs that inside a cached, parallel query service for ~2 minute selection latency.
AI-written summary · about
chapters (33)
- 0:00Introduction and speaker setup
- 4:57Defining the selection problem in CI
- 7:05Limitations of running all tests with Bazel
- 9:11Input and problem definition for target selection
- 10:16Controlling example with the slashfu package
- 11:19Using Bazel query for dependency analysis
- 12:21Complexities and gotchas of Bazel query
- 13:25Initial solution with rdeps and test functions
- 14:10Problem one: Modifying build files
- 15:56Fixing build file changes with siblings and filter
- 16:39Problem two: Modifying Starlark files
- 17:22Introducing Sky Query for Starlark files
- 20:09Problem three: Deleting files
- 20:31Handling deleted files with two queries
- 21:33Problem four: Adding or removing build files
- 22:13Glob function limitations and file system traversal
- 23:17Combining fixes into a universal query
- 23:41Reconsidering the approach with target hashing
- 25:06Introducing the build miracle tree concept
- 26:10Hashing targets and their dependencies
- 27:15Computing changed targets between revisions
- 28:21Trade-offs of the hashing approach
- 29:05Handling external workspaces and limitations
- 30:28Disgusting hack for external workspace rules
- 31:11Optimizing target selection with a Bazel query service
- 32:13Benefits of the Bazel query service
- 33:18Operating the Bazel query service
- 34:21Challenges with Git and system stability
- 35:02Being incorrect for profit in test selection
- 35:22Limitations of running only affected tests
- 36:04Batching and policy-based test pruning
- 37:08Predictive testing and machine learning
- 38:12Closing remarks and Q&A introduction
related talks
18mDesigning a language Agnostic CI using Bazel queries
48mCI With Bazel (Why Selenium is Using It & You Should Too)
35mBisection: It's Not Just for Git
33mFewer, Smarter, Faster: Scaling Testing @Spotify
45mUsing AI, statistics, and set theory to know what to test
08mWhy We Should Care About Test Execution Output in Safety-Critical Industries
21mDebugging and fixing build non-determinism with Bazel
11mIntroducing Remote Bazel