I believe the current Infiniband hardware is optimized for the 2200 cpus in the system. To increase the number of processors, you would have to improve the communication hardware significantly.

Communication is the real bottleneck, as seen in the efficiency of 80% with 256 processors and the efficiency of 60% with 2200 processors.

The price of the system might scale almost linearly, though, if you could just plug in additional switches. But the cost of the communications hardware could as well increase much faster than linearly when adding new processors.

Rob Barris commented on the huge power consumption estimate: "I would not be surprised if a large fraction of the difference is accounted for by the external cooling machinery - compressors to make cold water for distribution are likely power hogs (as most refrigerators are)." Thanks for this insight. However, isn't this a case of moving heat from one place to another, and thus not too expensive? Of course, this depends on the actual mechanism which is used, and the system at Virginia Tech may well be rather inefficient.