Living with code generation.
Just in time for the holidays, I just got a milestone build of my product done! For this milestone release, the core data manipulation engine had to meet certain quality requirements. I thought that I had it done on Friday, but my last set of tests which used data canonicalization turned up a whole raft of bugs that would have otherwise eluded detection. This article describes my experiences using code generation and related techniques while implementing and testing my data manipulation engine.
But first a bit of (intentionally vague) background about my data manipulation engine. It can parse and generate complex binary files. Within these files are hundreds of types, which typically result in thousands of instances at runtime. For this milestone release, my test suite had to ferret out bugs that specifically related to the serialization / deserialization engine. To complicate matters, the data could be [de]serialized into several different formats. This is one of those nasty test matrices where the sheer number of combinations defy any sort of brute-force testing.
Most of my data engine was code-generated using a free code generation tool called gslgen. Using line count as a very simple metric, roughly 10000 lines of codes were generated, and about 1000 lines of code handled the special cases that were beyond the scope of the code-generator.
Most of my test suite was also code-generated. I used a separate code-generator and I defined a special language that let me specify test cases extremely compactly using XML. Currently there are about 150 tests in the suite that I generate from roughly 300 lines of custom test case language. This list is by no means exhaustive given the number of possible combinations, but I'm very happy that complex integration tests using real-world data have resulted in a 100% pass rate. I will back-fill the additional unit tests as the code base evolves.
To run my test suite, I'm using NUnit V2.0. I'm very happy with how effortless it was to integrate NUnit unit tests into my existing build environment. Most of the time during development, I keep the NUnit GUI up and running during development builds to run periodically as I fix bugs. I found that the safety net provided by my unit tests allowed me to aggressively test new bug fixes in my code.
One of the key features of my data engine / test suite is data canonicalization. My data engine generates a single (canonical) representation of the data that is owned by an object, which lets me quickly detect data corruption bugs. My code generator generates a special-case [de]serializer that would parse/serialize an object's fields to/from a stream as XML. Once it was in XML format, it was very easy to run a diff tool[1] against the generated files to see where the problems were. Since the [de]serializer would serialize to/from a canonical data representation, it was very easy to spot bugs in the [de]serializer.
I used this technique extensively in integration testing. A typical series of events in an integration test would be to:
-
Parse an existing file A of binary data to generate an object graph
-
Serialize the object graph as XML 1
-
Parse the XML 1 to generate an object graph
-
Serialize the object graph to file B as binary data
-
Parse file B to generate an object graph
-
Serialize the object graph as XML 2
-
Compare XML 1 and XML 2 for equivalence
I spent a fair amount of effort in my code generator to ensure that the generated source code was highly readable. This made it very easy for me to step through the generated code using a debugger, and to modify that generated code. Contrast this with classic C++ template driven programming, where it was impossible to modify the generated code; instead you must modify the template, which makes it hard to quickly test fixes to a particular instance of a template.
One of the really amazing things about code generation was how quickly I could fix bugs once I found them. This is because algorithms were expressed in exactly one place using code generation.[2] Most of my bug fixes would either be in the code generator or in the supporting class libraries. And I found that once I fixed a bug once, it almost never re-occurred.
Overall I found that the cost of switching to code generation is heavily front-loaded. But once you get the code generator up and running (and also using it creatively in places like generating your test cases) you will find that it pays you back with much higher quality code in the end.
[1] I use WinDiff from the Platform SDK.
[2] This reminds me of Tip 11: Don't Repeat Yourself (DRY) from the most excellent book The Pragmatic Programmer. [IUnknown.com: John Lam's Weblog on Software Development]
10:26:45 AM 
|
