Tuesday, July 11, 2006

How are Unit Tests Like Chemical Reactions? Every so often I read someone who thinks because they do unit testing with 100% coverage they don't need to do any other type of testing (regression, system, acceptance). What could possibly go wrong if all the units work? One way to think of a program is a program is like a chemical reaction. A bunch of particles react to make a product. An interesting feature of chemical reactions is the resulting product has nothing to do with the original particles. I think about this as the role model for emergent properties in general. Take as an example our old friend H(2)O: water. We bathe in it, we drink it, we throw it on people when they catch on fire. Let's take a closer look at the components of water. Water consists of hydrogen and oxygen. By looking at the properties of hydrogen and oxygen individually could you predict the properties of water? Let's see. First take a look at this movie of the Hindenburg exploding. The Hindenburg was filled with hydrogen and as you can see it is highly flammable. Oxygen itself doesn't burn, but oxygen is required to make everything else burn. Add oxygen to a flame and you get a lot more flame. So when you combine hydrogen and oxygen what do you get? Shouldn't get a compound that burns and explodes into a fiery hell? You would think so, but you get water instead. To recap:           hydrogen      - think Hindenburg        + oxygen         - burn baby burn        ----------        =   water          - not fiery hell The properties of the elements doesn't tell you what will be the properties of the resulting compound after the reaction. Though not quite as dramatic, programs are like that too. When you combine program elements together you can't really predict the properties of the resulting program when the elements react. The basic reason is the number of paths through the program are now exponentially larger than what you tested when you unit tested the individual elements. And like with elements the really interesting things in a program happen in a running system in the real world where the inputs wash over your system in ways you couldn't imagine from just looking at individual units. For chemical reactions you have to resort to quantum chemistry to know what's really going on and why. In a software system there are usually equally obscure influences on how your system behaves that don't appear in unit testing. In system tests you'll see the effects of load, locks, CPU starvation, memory fragmentation, queue sizes, drops and retries in protocols, bugs in your OS, bugs in your hardware, bugs in the network, bugs in the network hardware, and so on and so on. You'll never see all these reactions in just your unit tests because you aren't testing all the units, you aren't testing all the paths through the system, and you aren't testing the interactions through all the parts you don't even know about. So create the biggest, nastiest, and meaniest test environment you can think of . See how it all reacts. You might be surprised when you get water instead of the Hindenburg. comment[] 12:27:52 PM

© Copyright 2006 todd hoff. Last update: 7/27/2006; 1:27:22 PM.