In an online Q&A webcast with Scott Hanselman today the topic of Code Coverage was raised. One audience member asked what the optimal level of Code Coverage is. The answer is that there isn’t an optimal level because Code Coverage can be a misleading statistic. Take for example the following method:
public class Math
{
public static int Add(int num1, int num2)
{
return num1 + num2;
}
}
You might then write the following unit test to validate the method:
[TestMethod]
public void AddTest()
{
// ARRANGE
var num1 = 4;
var num2 = 8;
var expected = 12;
// ACT
var actual = MathLib.Math.Add(num1, num2);
// ASSERT
Assert.AreEqual(expected, actual);?
}
This test will pass, and will also yield 100% code coverage on the Add method:
There is a crucial flaw though. What happens if you pass in int.MaxValue and int.MaxValue? To find out, let’s write another test:
[TestMethod]
[ExpectedException(typeof(OverflowException))]
public void AddMaxTest()
{
// ARRANGE
var num1 = int.MaxValue;
var num2 = int.MaxValue;
// ACT
MathLib.Math.Add(num1, num2);
}
Running this test yields the following failure:
Apparently an OverFlowException wasn’t thrown after all. To understand why, let’s write one more test:
[TestMethod]
public void AddMaxExceptionTest()
{
// ARRANGE
var num1 = int.MaxValue;
var num2 = int.MaxValue;
// ACT
var actual = MathLib.Math.Add(num1, num2);
// ASSERT
Assert.Fail("Expected Overflow, received: " + actual);
}
The result is not what you might expect:
In fact, the result of adding int.MaxValue and int.MaxValue into an int yields negative two. The reason for this, to the best of my knowledge, lies in how the .NET framework processes integer arithmetic. The int type is signed, meaning the last bit of it’s 32 bits is reserved for the positive/negative flag. When it’s processed under the covers however, it’s processed as unsigned, so as it rolls up adding int.MaxValue and int.MaxValue until it flips the flag bit producing a result of negative two.
What does this have to do with Code Coverage? Our original test adding 4 and 8 and yielding 12 covers the Add method with 100% coverage. It does not however reveal the unexpected result of adding int.MaxValue and int.MaxValue.
Code Coverage at a low value is a good metric for finding areas of your code in need of more validation. A high Code Coverage number though means nothing without full boundary checking.