Why 100% code coverage is not the goal

Code coverage is a metric of how many percent of lines of code were executed during a test run. Typically, code coverage would be collected during the execution of a complete test suite (running all the automated tests that were written) to get an overall number of lines of “tested” code in the codebase.

While code coverage is certainly a useful metric to discover untested code, simply aiming for 100% of lines of code covered by tests is not a worthy goal in a test strategy on its own.

To demonstrate that, let’s start with a simple code example (it is valid Python but consider it to be just a pseudocode for the purpose of the demonstration to avoid discussing how such a function should be written in Python):

def is_even(number):
    if number % 2 == 0:
        return True
    else:
        return False

def test_is_even():
    assert is_even(2) == True
    assert is_even(3) == False

The code snippet is a function that should return True or False if the provided number is even or not, followed by a test for said function. After the test is executed the test runner will report 100% code coverage because all lines of code were touched during the test run:

---------- coverage -----------
Name               Stmts   Miss  Cover
--------------------------------------
test_coverage.py       7      0   100%
--------------------------------------
TOTAL                  7      0   100%

Unfortunately, 100% test coverage didn’t help us make sure our code or the tests were any good. The test is not testing whether the function returns correct results for all numbers nor it is testing if the function works with different data types or null values. And because the test doesn’t test things correctly, the function under test might be all wrong too.

There are other situations where the code coverage lies to us a little bit. For instance, a valid code path can be completely missed because multiple tests touch all lines of code, but some specific lines of code are not touched in the right order - missing an edge case.

We might also miss the forest for the trees with the focus on full code coverage. Even if the code and tests are correct from a code perspective, code coverage will never tell you if the program does what it should, based on functional and other requirements.

---

All in all, code coverage is a useful tool to uncover untested code (for instance, during a code review), but it doesn’t make any real guarantees about the tests and, therefore, the code under test as well. That’s why aiming for 100% code coverage might not be worth it: there are probably other things to prioritize over it.

---

Have a nice rest of the weekend,

Petr