I ask this question in a bunch of my talks, mostly because most folks aren’t aware of this. What is the bug in the following method?

public int Add(int num1, int num2)
{
    return num1 + num2;
}

Do you see it? In a simple unit test, the method appears to work as expected:

[TestMethod()]
public void AddTest()
{
    MathHelper target = new MathHelper();
    int num1 = 16;
    int num2 = 16;
    int expected = 32;
    int actual = target.Add(num1, num2);
    Assert.AreEqual(expected, actual); // PASS
}

However, what if we pass in a different set of parameters, say int.MaxValue?

[TestMethod()]
public void AddMaxTest()
{
    MathHelper target = new MathHelper();
    int num1 = int.MaxValue;
    int num2 = int.MaxValue;
    int result = target.Add(num1, num2);

    Assert.Fail("Expected an overflow exception. Received: " + result);
}

Any guesses on the outcome of this test? In actuality, no exception is thrown. The test fails with the output: “Assert.Fail failed. Expected an overflow exception. Received: -2”. The output from the fail assert is “-2”. What!? Is it a bug in the .NET framework? What’s going on?

The answer lies in that I am passing in a signed int. In a signed int, the top bit is flag bit indicating positive or negative, leaving the other 31 bits to indicate the numeric value. When doing binary math that overflows the 31 bits, the 32nd bit is switched as though it is unsigned. In this case a positive becomes a negative and the result becomes –2. Let’s hope the system isn’t trying to deposit $100 into an account with $2,147,483,647.

But what does this mean to the original method? How do we handle this scenario? The point of this experiment isn’t to demonstrate a potential issue with the .NET framework. Rather, I’d like you to take notice of the first unit test. The test does evaluate the method for proper functioning, 100% code coverage would be achieved on the Add method, and yet the issue at hand is not found. And to this I would like to make an important point:

Code Coverage is a misleading metric.

In order to properly test your code, it is important to not only make sure you test as much as possible, but to ensure you also perform boundary checking. If the max values weren’t tested, it might not have been found that the code could erase $2,147,483,749 from some pour soul’s bank account. Well, he’d be poor after the fact. Are here is my second point:

Boundary check everything.

To help solve this problem, there is a neat tool from Microsoft Research called Pex. From the Pex website:

Right from the Visual Studio code editor, Pex finds interesting input-output values of your methods, which you can save as a small test suite with high code coverage. Pex performs a systematic analysis, hunting for boundary conditions, exceptions and assertion failures, which you can debug right away. Pex enables Parameterized Unit Testing, an extension of Unit Testing that reduces test maintenance costs.

What I would really love to see someday is a combination of existing code coverage in Visual Studio Team System with parameter boundary checking in Pex to give a truer sense of test coverage. Maybe for Dev11?