Mutation Testing in Python
summary
Built a Python mutation tester called Cosmic Ray that mutates AST nodes, installs mutated modules via importlib finders/loaders, and runs tests in isolated processes using multiprocessing and an actor-style queue. It supports pytest plugin and unittest runners, parallelizes across cores, and uses baselines plus git-diff to avoid full runs. Results classify mutants as killed, incompetent (crashes or timed-out loops) or survived; equivalent mutants remain a hard, practical-to-detect problem and runtime can be very long.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 3:27What is mutation testing?
- 4:38Basic algorithm
- 5:12What does mutation testing tell us?
- 7:31Goal #1: Coverage analysis
- 8:29Goal #2: Detect unnecessary code
- 10:03Examples of mutations
- 12:56It takes a loooooooong time
- 15:51Incompetence detection
- 17:12Equivalent mutants
- 19:54Cosmic Ray operates on packages
- 22:25Test system plugins
- 25:10Operators
- 26:15Example operator: Reverse unary subtraction
- 27:02Module management: overview
- 29:03Module management: Finder
- 30:47Module management: Loader
- 31:33multiprocessing
- 35:30asyncio
- 36:03Operational overview
related talks
57mMutation Testing in Python with Cosmic Ray
59mTesting in Python
30mKill All Mutants! (Intro to Mutation Testing)
56mLightning Talks
27mDon't Make Me Laugh, My Coverage
28m"WHAT IS THIS MESS?" - Writing tests for pre-existing code bases
26mMashing up py.test, coverage.py and ast.py to take TDD to a new level
36mHacking the Python AST