Waves and MQs: an experienced analyst comments

Mike Rasmussen has blogged a critique of Forrester’s recent Wave on Global Risk and Compliance, in a piece which deserves an audience outside the GRC community.

Mike is the doyen of GRC analysts and, in his time at Forrester, authored two previous versions of this Wave. He’s quite explicitly not getting at the authors of the new update; it’s the process that’s at issue. And the comments are well worth reading if you use any Wave, Magic Quadrant, or similar tool to help your purchasing decisions. Especially if your management board won’t approve a purchase unless these tools “endorse” the choice.

Mike has two criticisms. One of them isn’t a surprise, but it’s worth a reminder. The picture presented in a point-of-time report is, almost by definition, out of date before it’s published. Vendors don’t stand still while a report’s being researched. Indeed, some vendors opt out of supporting an evaluation because they’re close to a new release and don’t want to be judged on the old one.

InformationSpan has a similar issue: if you download our free report on insight service coverage for BI, you’ll find that its assessment of Forrester’s coverage is significantly out of date. It was written last November, and they’ve put out a lot of new coverage this year.

But secondly, Mike comments that the assessment criteria for the GRC Wave haven’t been updated while the discipline of GRC has moved on substantially. Analysis needs to recognise this.

These are process questions. The first one reflects the length of the evaluation process; things go out of date while being evaluated, and vendors sometimes decline to commit the resources. Mike asks whether this can be streamlined.

But the second question reflects the fact that the process was designed for systems delivered into a relatively mature marketplace, where the underlying concepts being modelled in software (for example) aren’t changing greatly.

GRC in the enterprise isn’t primarily about tools: it’s about management discipline and process. Tools simply support the process; tool capabilities develop rapidly as the discipline itself develops. And there are other areas like this.

For example, there’s an emergent unified approach to change management, configuration control and release management (CCRM) – I attended a workshop about this recently. Or there’s architecture, needing new approaches to integrate the benefits of cloud services securely into the enterprise while the first wave of architecture repositories and other tools are still evolving. No doubt you can identify other examples.

This isn’t about the pace of change. It’s whether a process implicitly predicated on a stable environment can cope with changes which are about much more than new technical ways of doing essentially the same thing.

So what’s the way forward? More than Mike suggests, I think. The key must be to separate the market snapshot from the analysis report. A vendor’s vision or ability to deliver don’t change over the timescale of report writing – nor with the actual issue of a new release, though the market’s reaction to it may be significant. So here’s my suggestion.

First: let the analysts get behind the versions to give a well researched, more stable view of the vendors and their contributions in a particular sector. Detach this from the assessment of current releases: they are data for the assessment, in this understanding, but not the core of it. Gartner’s MarketScope, an alternative to their better known Magic Quadrant, targets this: their aim is to provide “an overall market rating that indicates the strength and potential for the market in general. This is particularly important in emerging markets, when … it is difficult to assess the long-term viability or evolution of offerings. In mature markets, MarketScopes provide insight about the ongoing value of products and services“.

And second: make the process continuous, rather than point-in-time. Then it can respond continually not just to new products but to new assessment criteria as the underlying paradigms change. If  new tools were continually tested, and the results added to the database, a Wave or MQ could be dynamically delivered from the most current data. I don’t think anyone does this. Tell me if you know different!

It’s a step beyond what Mike’s asking for: not to streamline the process, but to change it.

So read Mike’s comments, and apply them to your own specialism. Understand the strengths and weaknesses of Waves and MQs as they currently exist. Comment back to me here – does anyone know of an assessment tool which is already dynamic in this way? And always remember, when you’re using any of these analyst tools, that they provide insight – not ready-made decisions!

• The Forrester GRC ?Ripple? … , Corporate Integrity (Mike Rasmussen), 2 July 2009
• Coverage report: Business Intelligence, InformationSpan, Nov 2008 (free download from this page)
The Forrester Wave
Magic Quadrants and MarketScopes: How Gartner Evaluates Vendors Within a Market, Gartner, Jan 2008 (this document appears to be openly available)

Finding vendors: Magic Quadrants and so on

Carter Lusher of Sage Circle reports an interaction between Andreas Bitterer of Gartner and a smaller vendor who’s taken issue with coverage in a Magic Quadrant. Credit, as Carter says, to both parties for engaging in a public discussion about this.

If you’re a user IT organisation with a relevant subscription, you probably use the Magic Quadrant (or Forrester’s equivalent, the Wave) to assess the players in a particular marketplace. If so, it’s well worth reading this post to better understand these methodologies. It appeared over the holiday period, so you might have missed it.

Sage Circle looks at it from the vendor’s angle, for Analyst Relations professionals. Here’s a thought for the enterprise IT team.

If you are inclined to look at emerging or niche vendors, or at Open Source (the issue which triggered Carter’s posting), then the mainstream market assessments probably won’t see them, and the main insight providers probably won’t cover them. The same’s true if you are investigating a niche area. I was asked, once, to find a hosting service experienced with a specialised software product. In its own market area, this application was a leader; but it wasn’t exactly mainstream ERP! There was no analyst coverage at all, within our available subscriptions.

In this situation you can try three things.

First, find a niche insight provider which specifically covers the market, if there is one. Ask InformationSpan: we have the database, and the review is almost complete (I’ve got to “T”!), but one thing I do know now is that not all niches have this kind of specialist coverage.

Second, make your own assessment. Both Gartner and Forrester publish their criteria, though Forrester provide more complete details; if you’re in an area where there is no coverage then look at a related MQ or Wave for ideas what you ought to assess.

If you’re looking at niche vendors in a major area where there is coverage, remember that Forrester make available the full model including data and weightings. You can adjust the weightings to create your own Wave, and if some of those criteria aren’t of interest to you in relation to your niche interests, or you can’t either research or estimate them, then just zero-weight them. You can’t add data to a Wave, but you can work in parallel with it. And validate it with an analyst when you’ve done the work: they might well see something you’ve missed.

Third, you can ask the analyst to do custom research. Again, with a niche product there will be criteria they can’t assess – there won’t be a sizeable user community to ask questions of, for example. This may be useful for a niche product in a major area; in a specialist area it’s likely you’ll know more about it yourself and, because they’ll start from a lower base, it will be expensive. They may even say “no”. But they will work, so far as possible, to their standard methodology and this will make comparison easier; aside from not having to do all the work yourself!

Oh – and a Happy New Year! There’s plenty of talk about managing through the recession. I still like George Colony’s take: including, hire the smart MBAs who are not now going into finance!

• Vendor complains in a very public blog post about Gartner’s Data Integration Magic Quadrant Sage Circle, 29 Dec 2008
Setting the Record Straight Andreas Bitterer, Gartner blog, 28 Dec 2008
• CIO best practices for thriving in a recession Counterintuitive, George Colony, Forrester Research, 24 Sep 2008