Bug fixed, now what?
summary
An open-source GitHub bot generates fail-to-pass tests by combining LLM + search-based testing (it seeds the genetic tool Penguin with LLM-generated inputs), then executes candidates on pre-PR and post-PR code and only posts tests that fail before and pass after. On benchmarks it produces such tests for 151 of 426 PRs (~30%), and in deployment it helped produce eight confirmed useful tests (six Python, two JavaScript) across three Mozilla repositories. Requires a Docker test runner and an LLM API key.
AI-written summary · about
chapters (9)
- 0:00Bug fixes introduction
- 3:05Regression bugs after fixes
- 5:16Fail-to-pass regression tests
- 7:28High-quality test generation
- 9:36Live demo automated tests
- 11:03Empirical evaluation feedback
- 13:50Tool architecture backend
- 16:43How to try the tool
- 17:14Q&A tool limitations
related talks
47mCan You Trust Your ChatBot: Techniques for Testing LLM Responses
15mLLMs and Test Automation: A New Era of Self Healing
38mMaking LLMs reliable - A practical framework
15mAutomating menial engineering tasks with LLMs
47mValidating LLMs – A True Story
41mHow continuous testing keeps your LLM on track
43mMutation Testing with PIT
41mMutation testing in practice