Wow -- I haven't been in a really meaty HCI conversation in a while, and I didn't realize how much I missed it until tonight.
Here's a little known fact, but probbaly won't surprise people once they ponder it for a moment: Microsoft has a central archive of reports from usability tests and user studies on ts products. The reports date all the way back to 1989. I just spent half an hour rooting around in it. The archive is an amazing historical treasure chest; among other things, it contains the usability test reports for Windows going back to 1989, and for Office back to around 1994. It also contains the usability test reports for Microsoft Bob.
(Yes, it was usability tested. It did really well, too. Which is one reason why it was released. It failed for completely different reasons, a topic for another time)
Now, anyone who has worked on a user interface at Microsoft can tell you that there is an amazing machine inside the company that does user studies. Our user study specialists are very detail-oriented and very, very disciplined. They can also be quite creative, and particularly in the last few years as they've dug deeper and deeper into the mysteries of how our customers use our products -- or would LIKE to use them. They have done thousands of tests over the years, and involved tens of thousands of users. The scale of the undertaking, across our entire product line, is simply staggering.
A number of years back, I floated the idea internally that we should release into the public domain the usability tests and user studies that are old enough to no longer be of competitive or strategic value. My thinking was that as historical documents on the process that went into the design of such high-impact products, they would be of significant value to the academic community. And frankly speaking, it would probably get us some well-deserved credit for a lot of hard work that has never really been exposed to a public that likes to complain about our "feature-bloated" products and their personal pet peeve.
In the end, I couldn't get approval to do it, for one excellent, very simple and straightforward reason: because we promised we wouldn't. As a condition of people coming in and consenting to trying out some software and letting us observe, record the session and analyze the results, we promise to keep what happens in the session confidential. It's an important guarantee, because that's the only way for us to get honest, real results. People get embarrassed when they make mistakes in a public forum; if we don't go overboard to reassure people that the session will not be exposed to the public, then we have to expect that they will participate less than fully. It's absolutely sensible, it's the Right Thing for the people who help us with our user studies, and when it was explained to me, I stopped trying to get the usability reports released.
But hey... if you come work for Microsoft as a full time employee, you can peruse the archive yourself!
8:01:50 PM
|
|