"Our analysis indicates that in some cases, the TPC benchmarks fall reasonably within the range of real workload behavior, and in other cases, they are not representative of real workloads. Some of our findings are (1) TPC-C tends to have longer transactions and fewer read-only transactions than the production workloads, whereas some of the transactions done by TPC-D are much longer but are read-only and are run serially, (2) the production workloads have I/O demands that are much more bursty than the TPC benchmarks, (3) unlike TPC-C, which has very regular transactions, and TPC-D, which has long queries that are run serially, the production workloads tend to have many concurrent and diverse transactions, and (4) TPC-C has no I/O activity involving temporary objects, whereas most of the references for TPC-D are directed at index objects."

These conclusions worried me at first - as someone currently very interested in the TPC-C benchmark - but I now wonder if my worry is misplaced. I believe that the TPC-C benchmark strives to be realistic, but cannot possibly be representative of the wide range of workloads used in IBM's study. I am not an expert in the TPC-C benchmark, but I think the benchmark's intent is to represent a fairly sophisticated order entry system for a wholesale supplier. This will not represent every business' transaction profile. It seems unlikely to be at all representative of the workload in the aeorspace or telecomm industries (which make up two of the ten companies used in the IBM study).

I think IBM's findings are probably valid. In fact, I would expect real-world workloads to vary from the TPC-C benchmark. The important question is: does this matter? Do IBM's findings invalidate the TPC-C benchmark? This is my own question; IBM was clearly focusing on analyzing real-world workloads and comparing them to TPC-C, not trying to invalidate TPC-C. I think the TPC-C benchmark is still valid and useful. From IBM's own study TPC-C is fairly representative of an average business workload. It is very representative of an order entry system's workload. If we assme, correctly, that the purpose of the TPC-C benchmark is to measure the relative performance of one system against another based on a standard workload, and not to determine the absolute performance somebody will see for their particular workload, then it is still a useful benchmark.
9:18:27 AM    Google It!