Waves and MQs: an experienced analyst comments

Mike Rasmussen has blogged a critique of Forrester’s recent Wave on Global Risk and Compliance, in a piece which deserves an audience outside the GRC community.

Mike is the doyen of GRC analysts and, in his time at Forrester, authored two previous versions of this Wave. He’s quite explicitly not getting at the authors of the new update; it’s the process that’s at issue. And the comments are well worth reading if you use any Wave, Magic Quadrant, or similar tool to help your purchasing decisions. Especially if your management board won’t approve a purchase unless these tools “endorse” the choice.

Mike has two criticisms. One of them isn’t a surprise, but it’s worth a reminder. The picture presented in a point-of-time report is, almost by definition, out of date before it’s published. Vendors don’t stand still while a report’s being researched. Indeed, some vendors opt out of supporting an evaluation because they’re close to a new release and don’t want to be judged on the old one.

InformationSpan has a similar issue: if you download our free report on insight service coverage for BI, you’ll find that its assessment of Forrester’s coverage is significantly out of date. It was written last November, and they’ve put out a lot of new coverage this year.

But secondly, Mike comments that the assessment criteria for the GRC Wave haven’t been updated while the discipline of GRC has moved on substantially. Analysis needs to recognise this.

These are process questions. The first one reflects the length of the evaluation process; things go out of date while being evaluated, and vendors sometimes decline to commit the resources. Mike asks whether this can be streamlined.

But the second question reflects the fact that the process was designed for systems delivered into a relatively mature marketplace, where the underlying concepts being modelled in software (for example) aren’t changing greatly.

GRC in the enterprise isn’t primarily about tools: it’s about management discipline and process. Tools simply support the process; tool capabilities develop rapidly as the discipline itself develops. And there are other areas like this.

For example, there’s an emergent unified approach to change management, configuration control and release management (CCRM) – I attended a workshop about this recently. Or there’s architecture, needing new approaches to integrate the benefits of cloud services securely into the enterprise while the first wave of architecture repositories and other tools are still evolving. No doubt you can identify other examples.

This isn’t about the pace of change. It’s whether a process implicitly predicated on a stable environment can cope with changes which are about much more than new technical ways of doing essentially the same thing.

So what’s the way forward? More than Mike suggests, I think. The key must be to separate the market snapshot from the analysis report. A vendor’s vision or ability to deliver don’t change over the timescale of report writing – nor with the actual issue of a new release, though the market’s reaction to it may be significant. So here’s my suggestion.

First: let the analysts get behind the versions to give a well researched, more stable view of the vendors and their contributions in a particular sector. Detach this from the assessment of current releases: they are data for the assessment, in this understanding, but not the core of it. Gartner’s MarketScope, an alternative to their better known Magic Quadrant, targets this: their aim is to provide “an overall market rating that indicates the strength and potential for the market in general. This is particularly important in emerging markets, when … it is difficult to assess the long-term viability or evolution of offerings. In mature markets, MarketScopes provide insight about the ongoing value of products and services“.

And second: make the process continuous, rather than point-in-time. Then it can respond continually not just to new products but to new assessment criteria as the underlying paradigms change. If  new tools were continually tested, and the results added to the database, a Wave or MQ could be dynamically delivered from the most current data. I don’t think anyone does this. Tell me if you know different!

It’s a step beyond what Mike’s asking for: not to streamline the process, but to change it.

So read Mike’s comments, and apply them to your own specialism. Understand the strengths and weaknesses of Waves and MQs as they currently exist. Comment back to me here – does anyone know of an assessment tool which is already dynamic in this way? And always remember, when you’re using any of these analyst tools, that they provide insight – not ready-made decisions!

• The Forrester GRC ?Ripple? … , Corporate Integrity (Mike Rasmussen), 2 July 2009
• Coverage report: Business Intelligence, InformationSpan, Nov 2008 (free download from this page)
The Forrester Wave
Magic Quadrants and MarketScopes: How Gartner Evaluates Vendors Within a Market, Gartner, Jan 2008 (this document appears to be openly available)

Oracle acquires Sun: what do the analysts say?

As the comments begin to settle down around Oracle’s latest blockbuster acquisition, let’s have a look at who reacted how quickly, and the quality of their comments.

There’s plenty of discussion of the basics: that Oracle is to acquire (“rescue”, according to some coverage) Sun Microsystems. The crown jewels are:

  • the Solaris version of Unix, which Larry Ellison talked up and which is perhaps the most important platform for Oracle’s proprietary software portfolio. Relevant past acquisitions here include PeopleSoft/J D Edwards (Dec 2004) and Siebel (Sep 2005), both hard-fought battles
  • Java, the basis for Oracle’s growing middleware business. BEA Systems was acquired in January last year.
  • Sun hardware … it will be interesting to see what Oracle does with this, as an indicator of the company’s future direction
  • Star Office and Open Office – there’s not much comment about these. But Ellison may well encourage them, the better to compete with Microsoft
  • MySQL open source database, widely used by start-ups and web companies
  • Definitively stopping a link-up of IBM and Sun

When I looked at coverage of the Satyam scandal, the questions were: who reacted first? Who has the most incisive coverage for the enterprise user? Who had the best accessible coverage?

But things have moved on, not least with Gartner’s entry to the blog movement. The major analysts had blog postings up pretty much on the day, and so did a range of other commentators. Some postings are very short and don’t do much more than register that the analyst had seen the news. But some have significant analysis, and I’d commend Darryl Plummer from Gartner, James Kobelius and Ray Wang from Forrester, and Bruce Richardson of AMR who all quickly began to explore the implications. Stefan Ried of Forrester waited a couple of days and then, after a call with Oracle, brought together some of his colleagues’ earlier comments so that’s a useful link.

And at this stage there is not much early stage formal research reporting. There are a couple of reports from the big two (Forrester and Gartner). Forrester haven’t disappointed as they did last time: the quality of accessible coverage in the blogs is high. Gartner’s blog coverage is more random; perhaps this is where Forrester’s approach to blogging, with topic-related rather than personal blogs, pays off. Then there were rapid initial research notes from Forrester and Gartner but not a great deal more.

As with Satyam, there’s no consensus about the future direction of this integration. The main disagreements are:

  • will Oracle divest the hardware business, or leverage it? Significantly, the Oracle press resource talks about delivering “an integrated system—applications to disk”, not “applications to CPU”.
  • will MySQL be used as a route to migrate more customers for Oracle’s database and then killed off? will it be actively supported? or will it be cut loose for the open source community to work with?

So perhaps the main conclusion for insight service users is that the pace and quality of fast-response comment to major events has significantly picked up. And that there were no real surprises about coverage – except that I discovered a new source (GigaOM) which had an incisive article – matching Forrester’s Kobelius for early considered analysis. That’s a source I’ll be watching in future.

• Oracle and Sun, Oracle press and information website
• Forrester: Oracle’s Sun Acquisition Is A Game Changer, 22 April, client research targetted at vendor strategy professionals (i.e. the IT supply side) but relevant to users too
• Gartner: Oracle/Sun Deal Will Change Competitive Landscape in IT, 22 April, client research, short but publicly available
Our Full Analysis of the $7.4B Oracle-Sun Deal, GigaOM, 20 April

To see more of the coverage identified for this survey, click for items tagged in del.icio.us.

Insight service users: assess yourselves!

InformationSpan helps enterprises develop and deliver value from their investment in analyst/research/advisory services (you can see why we coined the phrase “insight services”).

Our new online self-assessment enables you to see how far along the road you are. Visit us at informationspan.com and click the link in the news item, or on the Enterprise page; or visit http://tinyurl.com/bhmfvs.

Providers: how accessible is your meta-information?

I’ve been doing research to update an InformationSpan coverage report on Business Intelligence. I’m struck by the different approaches of providers that help, or hinder, this task.

Remember – I’m not trying to read the content necessarily (though some elements of it are useful). I’m coming at it from the perspective of an enterprise trying to find the best insight services provider for their needs. So I’m trying to find out the depth of their coverage, how important the topic is for them, and how up to date they are. What I really want is meta-information: information about information.

To show what I mean, here are a handful of case studies.

Analyst firm A – a well known global specialist – offers me a guest account. Even without this, I can explore in reasonable depth just using the search box on the home page and the About section of the website. Then, when I sign in, I can see the whole structure of their website as a paying client would see it. I can browse the analyst biographies, undertake searches, and in fact do pretty much anything a client would do except read the premium content research. So I have a pretty good idea how many analysts cover this area, which of them I’ve heard of or encountered, and from the abstracts of the published reports I can see at least some of the companies covered in their writing. I think I can make a pretty good assessment of their coverage, based on what I’ve seen.

Company B has a similar model, and their guest login provides a lot of offers of complimentary access to the full text of quite recent reports – cleverly, using a Flash Player presentation which means there’s no downloadable version but you can see the whole thing. Even better!

Analyst firm C – a global generalist – has a great deal of good content. In fact, BI is one of their primary coverage areas. But if I didn’t know, I might well not realise it. I can see a fully featured non-client home page. But there’s no search box on it, so it’s rather difficult to assess what they’ve got. It turns out that the best route into the information I need is to browse the Analysts section, because their people are indexed by coverage area; or to go through Events. There are links there to reports, but only the barest abstract is visible as a non-client. I did find a key report via a vendor’s website (though I had to use Google to find it) and in the end, from prior knowledge and reasonably successful online research, again I have a fair idea of their coverage. But it was a lot harder work!

Provider D may or may not have content. But since the only information I can find from their web pages, as a non-member, is about the structure of their service and the highest level information about content, and there’s no search function without a login, I simply don’t know!

So if I’m an enterprise trying to find specific coverage, across the marketplace of four hundred or so general and specialist providers, guess which providers simply aren’t going to figure? They can’t be assessed, and they won’t make the cut.

Providers – please, at least, provide a search box so that we can see what you do!

PS – you can see the report via the InformationSpan website. Click the new tab section for Reports. No other links on this posting!