Looking for Clues
Today I spent some time doing test automation investigations. One of the test failures that I was looking at had the following output log contents:
Fail
I always get sad when I see a test that does this. It means that the person who wrote the test didn't think about helping the person who will be investigating the test when it fails (in this case, me...). It also could be the only time that the test fails in a test run and, with a single word output log, there is no information that could help diagnose the problem. If I can't repro the problem, it likely means there is a bug still lurking that I can't do anything about except hope it shows up again later. I hate that feeling.
I'm an advocate of writing tests that log as much information as possible. Since the team that I'm currently on tests command line tools, that means that, for each tool invocation, I write my tests to log the command line, the output of the program, and the return code. Also, at every failure point in the test, I output a unique informative Error message. That allows me to quickly find the failure point from a test log in the test's source code.
Another example: The command line tools that we test (the Microsoft C++ Compiler and Linker) have a switch, /NOLOGO, that surpresses the printing of the banner (the part that says what version of the compiler or linker is currently running). Many of our older tests invoke the tools with those switches. Whenever I fix a test I remove the /NOLOGO switch because it surpresses information that could be valuable in diagnosing a problem. On one occasion, we had requested a test run of our x86 tools on the 32-bit subsystem of a 64-bit OS. Because some of the newer tests weren't invoking the tools with the /NOLOGO switch, the test investigator (in this case, me again...) could figure out from the banner the tools printed that the test was erroneously run on the 64-bit cross targeting tools in the 32-bit subsystem. With that information, we could request a re-run of the test suites using the correct tools.
So, write your tests to log as much information as possible. The person doing the test investigation (in some cases, it could be you...) will really appreciate it.
8:29:55 PM 
|
|
