I once worked for a manager (a.k.a Scrum Master) who insisted that we should have at least 85% code coverage on our project. If we did not meet the targeted coverage numbers this sprint, then the whole team would face the consequences during the upcoming performance appraisals. In spite of trying to talk to her, she insisted. I would like to believe that the manager had good intent and was trying to encourage developers to write clean, tested code.
However the ground reality was, my colleagues were not convinced about automated developer testing, they lacked basic knowledge and skill to do effective developer testing and the manager always kept them under high pressure. (Otherwise people would slack, she said.)
Now what do you expect to happened when managers push certain (poorly understood) metric down developers’ throats?
Humans are very good at gaming systems and they do a wonderful job.
- Some developers wrote automated tests with ZERO assert statements
- Others wrote complex, fragile test code which was an absolute maintenance nightmare. Their coverage numbers looked impressive.
- Me and my pair wrote a small test which would simply crawl all the code base. We were able to achieve 99% code coverage. (1% intensionally left, so we don’t look suspicious.)
- And so on…
The manager was very happy and so was the team. This went on for some sprints. One could find our manager showing off to her colleagues about how much the team respected her words and what a command she had on us.
Alas, her happiness lasted only for a few sprints. One fine Sprint demo day, I convinced the team to showcase our dirty little secret to all the stakeholders and management.
The event did shake up the management. It also sent out a clear message.
Code Coverage measures which line of code got executed (intensionally or accidentally) when you run the program or tests against the program. Code coverage tells you nothing about the quality of your tests, hence nothing about the quality of the code. Code coverage along with Cyclomatic Complexity can be used by the team to guide them, but not by management to judge the team.