So, I think we're in agreement that the media needs some transparency.
And based on what I see being written elsewhere, some PR transparency
appears to be on order as well. So, what about research? In
our industry -- the tech industry -- if there's a part the
business that desperately needs more transparency, it's the research
part. I was reminded of this today when I was forwarded an e-mail
newsletter known as the InternetAcceleration Newsletter from InternetAcceleration.com. In the February 22, 2005 issue is a segment called There they go again. In
saying "We think that the current quality of output in its Magic
Quadrants are both potentially misleading to IT buyers and an abuse of
its brand," the report has some harsh words for tech research outfit
Gartner who owns the Magic Quandrant brand. In fact, so harsh is
the newsletter in levying accusations of impropriety (could this end up
being a transparency test), that I can't help but wonder how Gartner
cannot react. If Gartner sues for libel, then we get to watch
Gartner's image go on trial. If Gartner does nothing, what are we
to think then? The report goes on to say:
"Here's
a shocking example of what one vendor told us: - "we spent about a half hour
with ... [Gartner Research Vice President Name Deleted] ...of Gartner almost
exactly one year ago. We gave him a brief overview of our company and
exchanged pleasantries. As I recall the subject of a "paid relationship with
Gartner" was raised more than once." That wasn't the only response like
that. This situation turns out to be worse than we thought. It appears to
us that not only is Gartner clearly doing insufficient research (hadn't even
spoken to a vendor listed in the MQ for a year!), but that Gartner Vice
Presidents are using vendor briefings as thinly disguised sales calls. It's
pretty intimidating for a vendor to be asked to sign up for research
services IN THE SAME discussion as they're providing input used in a MQ
ranking and some may say unethical."
Whether
or not these stories about Gartner have any merit remains to be seen
(or perhaps we'll never know). But there are many research
outfits in the tech business and in my discussions with certain marcomm
pros that I've know for years and trust, there is no question in my
mind that these sorts of shennanigans are taking place behind the
scenes. While I'm not going to call anybody out for a bar room
brawl, the newsletter reminded me that research transparency is
definitely a discussion that needs to be had. For example,
when presenting scoreboard like research like Gartner's Magic
Quandrants, shouldn't the charts say which of the companies listed in
the chart are also Gartner clients? Or how about when the press
gets pitched on "new, earthshattering" results as a proofpoint of some
vendor's leadership?
Case in point. Recently via e-mail, I received a copy of a press release from Check Point Software Technologies that says:
Check Point Software Technologies Ltd. (NASDAQ: CHKP), the worldwide
leader in securing the Internet, today announced that recent
independent tests conducted by The Tolly Group confirm that Check Point
provides the broadest breadth of coverage and the lowest Total Cost of
Ownership (TCO) for today’s complex security vulnerabilities in
comparison to Cisco Systems, Inc. (NASDAQ: CSCO) and Juniper Networks
(NASDAQ: JNPR).
First, if you're a public
relations professional, please listen to what I'm about to say: I
don't know about other journalists and I know you're just trying to be
helpful, but when you pitch me on customer success stories and
research-based proof-points, my spider senses start tingling.
Call me fickle, but the last people I'm going to trust to provide me
with objective information about your products or services are the
people you're recommending to me. Think about it. What are
the odds that you're going to furnish me with a contact that has
something really bad (read: objective) to say about the product your pitching.
Would I really doing my readers a favor by relying on these sources as
authorities?
Second, this was one of the first research-touting e-mails that I've
received since starting this transparency channel and as I read it, I
realized that, in the name of transparency, it's time to start asking
questions. And I did. In this case I asked the following
question "Was the study by the Tolly Group commissioned by
Checkpoint?" Answer? "Yes, the Tolly Group study was
commissioned by Check Point."
I'm glad the public relations professional was honest. That was a good
move. But, for the research community, this puts the
transparency issue at front and center. Although I'm an
ex-testing lab director, I can't for a minute make a qualitative
assessment about the Tolly Group's research methodologies. I haven't
seen them. But I can say its time to insist on several layers of
transparency.
First, any pitches by vendors or public relations personnel to the
press, analysts, or customers that cite research must absolutely
disclose any relationship that those entities have with the provider(s)
of the research being cited. Second, there needs to be a
review and consensus of what language can be used in these
pitches. In the aforementioned pull-quote from the press
release, the Tolly Group's tests are characterized as
"independent." Now, perhaps everyone who signed off on the press
release including the folks at the Tolly Group have their own
definition of independent and they're certainly entitled to believe
that. But, in my book, when the cited research is commissioned by
the vendor that's pitching me, it doesn't pass my test for
indepedence. It doesn't even come close. Third, as I said
earlier, all published research should be accompanied by disclosure
of the client relationships that are relevant to that research. If
such relationships exist, then I think that calls for an additional
layer of transparency -- one that discloses whether the vendor was
exposed in anyway to the research methodologies prior to the start of
the testing or research. If so, then the nature of that
exposure must be detailed. Was the vendor allowed to provide
consultative input? Was the vendor given any veto power over certain
evaluation criteria? If a comparison was involved, did the vendor
have any influence over the competitive set? I'm sure there are
other questions to ask -- questions that make it possible for consumers
of any research to make their own judgements about the results being
shown to them (very often to help turn a prospect into a customer).
I think you get the drift. Today, you'd be hard pressed to find
any of these practices in use and there's a reason for that. It's
one of this industry's ugly, dirty little secrets and it's time for a
change.
Update: Be sure to read the
comments on this blog entry as well my interactions with Elizabeth
Albrycht. How many more "cathardic" postings from insiders
(former or present) will it take before the issue snowballs into
something bigger?
Finally, I know it has been a while since I last posted, but it has not
been a while since I've been working on this transparency channel. As
you can see by my expandable blog roll on the right side, I've been
experimenting with OPML. I've also been spending a lot of time under
the hood of Radio Userland, trying to figure out the best ways to
connect my e-mail system to it in a way that turns Radio into a
transparency content management system, something that would serve as
the underpinnings of my evolving JOTS specification.
10:55:27 PM 
RadioEdit
|
|
