mutmut

Mutation testing is technique for evaluating how good your tests are. The idea is that a special tool (or a very bored human) breaks the code by slightly changing (“mutating”) something in it and then running all the tests, and the tests must fail. If they don’t fail, you either have dead code or the tests aren’t good enough.

There are some pre-requisites for when it mkes sense to do that:

1. The test coverage is 100%. If it’s not, you already know what you should write tests for.
2. The tests are fast.
3. The tests are reliable.
4. The codebase is small.

mutmut is a tool for mutation testing of Python code. You simply point it to the directory with the code, directory with the tests, and what command to use to run the tests, and it will do the rest. When it’s done, you can generate an HTML report with the list of diffs of mutations for each file that weren’t detected by the tests.

I think it’s an “advanced” testing technique. I use it not that often and only on small projects that need to be reliable. There are quite a few “false positives” (things that you can’t and shouldn’t test, like databse connection options), but from my experience it also a very reliable way to detect what tests you’re missing.