Play with fire, but don’t get burned

Instrumenting your product’s source code to get an indication of how much of the code was covered during testing is a really, really smart measurement to get.  If you aren’t at least measuring this number, you should be.  Visual Studio provides features that continue to make this easier.  What does this have to do with fire?  Well, code coverage measurements are to managers like fire was to cavemen.  Discovering how powerful these measurements can be is enlightening!  It’s totally awesome!  You can do so much with it!  Grill up some mammoth, roast marshmallows, oh wait, are we talking about fire or code coverage?  But just like fire, with code coverage measurements you need to understand what you have, or you will get burned.  Ok, so play with fire, just don’t start a forest fire – Don’t draw conclusions from data that isn’t fully understood.  Here are some tips so that you can keep it all real when dealing with code coverage numbers.

Know what you are measuring.  You instrument a file, you run your tests, your code coverage is high, you think you are great.  Really?  Do you know what you are measuring? 

How large is your file?  Small files can potentially have high coverage numbers.

Are you measuring the right file?  I’ve seen some mistaken data that was being pulled from the wrong file and therefore more inflated coverage numbers were shown than were actually occurring. 

What is the code and feature you are measuring?  C++/C# code – easy to instrument.  Jscript – a bit harder to measure but possible.  XSLT – is that even possible?  SQL – possible with the right tools.  ETL – still figuring that one out ourselves.  What your features do and therefore what language they are written in (and what technologies they use) have a