Agile FAQs
  About   Slides   Home  

 
Managed Chaos
Naresh Jain's Random Thoughts on Software Development and Adventure Sports
     
`
 
RSS Feed
Recent Thoughts
Tags
Recent Comments

Don’t be Seduced by Code Coverage Numbers

I once worked for a manager (a.k.a Scrum Master) who insisted that we should have at least 85% code coverage on our project. If we did not meet the targeted coverage numbers this sprint, then the whole team would face the consequences during the upcoming performance appraisals. In spite of trying to talk to her, she insisted. I would like to believe that the manager had good intent and was trying to encourage developers to write clean, tested code.

However the ground reality was, my colleagues were not convinced about automated developer testing, they lacked basic knowledge and skill to do effective developer testing and the manager always kept them under high pressure. (Otherwise people would slack, she said.)

Now what do you expect to happened when managers push certain (poorly understood) metric down developers’ throats?

Humans are very good at gaming systems and they do a wonderful job.

  • Some developers wrote automated tests with ZERO assert statements
  • Others wrote complex, fragile test code which was an absolute maintenance nightmare. Their coverage numbers looked impressive.
  • Me and my pair wrote a small test which would simply crawl all the code base. We were able to achieve 99% code coverage. (1% intensionally left, so we don’t look suspicious.)
  • And so on…

The manager was very happy and so was the team. This went on for some sprints. One could find our manager showing off to her colleagues about how much the team respected her words and what a command she had on us.

Alas, her happiness lasted only for a few sprints. One fine Sprint demo day, I convinced the team to showcase our dirty little secret to all the stakeholders and management.

The event did shake up the management. It also sent out a clear message.

Code Coverage measures which line of code got executed (intensionally or accidentally) when you run the program or tests against the program. Code coverage tells you nothing about the quality of your tests, hence nothing about the quality of the code. Code coverage along with Cyclomatic Complexity can be used by the team to guide them, but not by management to judge the team.

  • This is hilarious, and a very cool example of management not being to perform due diligence appropriately.

    I actually don’t think it’s that hard to get 90+% coverage as long as testing proceeds right along with development. By coverage I mean with appropriate assertions, and the developers have to both buy in and be pretty experienced with testing as well. There is a disconnect in my mind between expecting an application to somehow execute every line in a code, and expecting it to be impossible to otherwise test every line. This is not to say I expect every boundary condition or edge to also be covered, that really would be far, far too much.

    Off to read more articles here.


    Licensed under
Creative Commons License