The Right Approach To Testing Software

I've seen various arguments that push for black box testing, or testing with no internal knowledge of the system being tested. Others argue for white box testing, in which the tests are conceived with intimate knowledge of the inner workings of the test subject. And then there's grey box testing, which is somewhere in between.

So what's best? Each of the above (and even other approaches) have their merits, but no one technique on its own is enough.

Black Box
If you pretend you don't know anything about the system under test (or deprive your testers of inside information), the test is typically going to be more of a real-world scenario. But will the test cover everything?

Take for example VMware's recent ESX bug that prevented countless virtual servers from powering on as of August 12, 2008. It was caused by a hard-coded expiration date intended for beta releases that was inadvertently left in. I'm assuming here, but something tells me that the VMware QA group never tested to see if this functionality was disabled in the release build as intended, because they didn't even know it existed.

White Box
Then there's the approach of using all you can find out about the system to be tested (source code, etc.) to more fully test a system. Does that information always help?

Last week I fixed a bug that was not caught in our normal testing. It was a pretty basic problem and something that was found easily with an atypical black box test. Normal testing didn't find it though, because we knew from way-back-when that we didn't have the problem since it was specifically addressed in the source code. This meant there was no test case for it.

Of course, source code changes over time. In this case, the prevention for this problem was shuffled around during code refactoring. While it was still executed, it happened too late in the process. Since it was still in the source code though, it remained untested.

Black and White
Testing is an imperfect science. When you put a bunch of humans together to test code written by another bunch of humans, mistakes will happen. Hopefully the mistakes will be caught and fixed, but it is inevitable that something undesirable will slip out the door.

I haven't seen bug-free software yet, and I doubt I ever will. As perfect as I am, I can't be in all places at all times to stop all the world's bugs. I'll have to work on that though, because imagine the utopia we'd all live in if all software was perfect.

I don't fault a black box test for missing an undocumented scenario. And I don't fault a white box test for missing something documented as not needing to be tested. VMware responded quickly and got a patch out as soon as they could when they found out about the problem customers were having. We were a bit more ahead of the curve and got a patch out before any customers reported a problem. Basically, both cases were just days in the life of technology.

But both cases illustrate what is an all too common problem in my experience - not enough testing. From a quality standpoint, there is no such thing as too much testing. From a business point of view, you have to ship your software at some point and can't test indefinitely.

Each development project finds its own compromise based on its unique situation, needs, and resources. I'm not about to argue with those. But a typical compromise picks a specific test methodology and runs with it.

Just like airplanes have a co-pilot to back up the pilot and the New England Patriots have Matt Cassel to back up Tom Brady, use a backup for your primary testing methodology. Sooner or later, it just might save your trans-Atlantic flight, football season, or just some embarrassment.