Benchmarking: sources

I’m facilitating tomorrow a Corporate IT Forum discussion on twenty-first century benchmarking. It’s a wide topic. This post is a set of links and some comments, based on the InformationSpan database of 700 research and analyst firms. But I’m always grateful for updates: please comment!

The Forum itself operates a benchmarking service for clients, so there’s a declaration of interest to make but I am not myself a member of it. Primarily this is crowd sourced: it invites members to contribute their own data, and to compare themselves against their peers.

• Computer Economics provides a range of benchmarking data, not all financial. I’d consider it a primary source and worth a subscription. It provides a wide range of data. Major studies include IT Spending and Staffing Benchmarks and Worldwide Technology Trends. Their Management Advisories look at ROI and TCO, Risk Management and other topics. Too many to list here. Take a look for yourself.
• InterUnity Group “provides leading companies with strategy, competitive intelligence, and benchmarking to improve business performance.” It’s not clear what areas of benchmarking are actually covered or whether the focus is primarily financial
• The component services of the Corporate Executive Board will be worth investigating. Using the Researched Sharing model for content, CEB services such as the CIO Executive Board link and correlate information and tools from clients.
• Ventana Research undertakes benchmark research as one of its primary activities, drawing information from its own community, social media and the company’s “media partners”.
• The Data Warehousing Institute undertakes benchmarking in its key area, primarily business intelligence. They publish an annual BI Benchmark Report.

Major insight firms also cover benchmarking. Gartner‘s IT Topics include Cost Optimization and IT Metrics. A search on the Forrester website also shows a wide range of coverage.

This is a rapid post in advance of the event. Look for a wider-ranging Coverage Report from InformationSpan when I’ve time to develop the theme.

ICT professional standards in the UK: what a mess

I teach a couple of Open University courses. In one of them, I’ve just got to the point where we encourage the students to work through the industry skills frameworks. The aim is to benchmark their skills and to identify both longer term career direction and short term professional development targets.

A few years ago it was confusing, but manageable. My first contact with this area was quite some years ago when the British Computer Society began to develop from an academic interest group into the professional organisation it is today. It began to review applications for membership. To benchmark (that word again) applicants’ status and career progression, it needed a framework. Out of this grew the Industry Structure Model, which identified a number of career tracks. This developed into the Skills Framework for the Information Age (SFIA), which is still a great set of definitions for ICT career people. More below, about SFIA.

When I first came back to this teaching, five years ago, the then government had created an enormous, wide-ranging family of National Occupational Standards (NOS). These were divided among a number of defined industry sectors and Sector Skills Councils. Some of the areas were fairly obvious, like Engineering. Others, perhaps less so, like Contact Centres. The general principle was a good one: that in the main, skills were only defined once. So, anyone whose role included management looked to the Management framework. It wasn’t re-defined in every profession. Anyone who used IT (and I mean, used as a user) could benchmark those skills against the IT User NOS standard. These “generic” skills were, as it were, imported into the professional portfolio which defined actual roles in real organisations.

Well, what have we now?

1. Originally, there was the overall IT Professional Competency model  (e-skills Procom). This has been discontinued so far as I can tell. It now exists only in the National Archive – under the “NVQ” section although Procom is not an NVQ framework (!).

Procom provided a framework of seven disciplines:

  • Sales and marketing
  • Business change
  • Programme and project management
  • Solutions architecture
  • Solution development and implementation
  • Information management and security
  • IT service management and delivery

2. Of these, disciplines 4, 5, 6, 7 are represented in the IT/Telecom Professional NOS of 2009. The SSC, e-skills UK, still exists and this framework is still current on the e-skills website. These are, though, hidden in a link right at the bottom of the page. Currently, look for “NOS” in the purple footer.

The IT/Telecom Professional framework categorises capabilities at five levels: Junior Technician; Associate Professional; Professional; Lead Professional; Senior Professional. It categorises its criteria according to Performance; Knowledge; and Understanding.

Alongside this, e-skills maintains the IT User NOS which is valuable for almost anyone, We all use IT user skills. This framework defines three levels: Foundation, Intermediate, and Advanced. The Advanced level overlaps into the IT Professional framework, covering user application development (Access, say, or Excel). This is also the framework where you’ll find user skills with software, be they office tools or specialised business applications.

3. The Skills Framework for the Information Age (SFIA) still exists and is now at version 5. It’s available as a spreadsheet download.

SFIA defines the following skill areas:

  • Strategy and architecture
  • Business Change
  • Solution development and implementation
  • Service Management
  • Procurement & management support
  • Client interface (i.e. sales & marketing)

It defines levels from 1 (junior) to 7 (which equates to senior management or CIO). Not all cells in the model have definitions at all levels: for example, within Strategy & Architecture the cell “Corporate governance of IT” begins at level 6. SFIA does have the advantage that it encompasses management to the most senior levels as well as technical capabilities.

4. Since late 2012 there is the IT Skills Academy. It is itself confusing.

First, it references a full set of role descriptions in its Standards section. The rubric says that “The IT Professional Standards have been organised and aligned to the relevant SFIA skills and levels.”. What this actually means is that the Standards are not aligned to SFIA, but there is a correlation table showing where matches have been identified.

They are not aligned to the NOS either. Again, some areas map across although the names are not quite the same. The disciplines here are:

  • Architecture, Analysis & Design
  • Business Change
  • Information Management and Security
  • IT Project Management
  • IT Service Management and Delivery
  • Sales & Marketing
  • Solution Development & Implementation
  • Transferable Competencies (three flavours: Personal, Business and Leadership).

The sub-categories of each discipline have definitions from Level 3 to level 6. The definitions are, like the NOS, divided as Performance; Knowledge; Understanding.

The Transferable section is well worth having. With the change to the NOS database overall, these general skills are now much harder to find elsewhere.

5. The Skills Academy website also offers the Professional Profile. This matches the categories and levels (3-6) of the Framework, but the descriptions are considerably simplified with a handful of “Do you do these things?” criteria.

6. Finally there is what you get to from the new NOS website. Searching this website is now far inferior to what used to be provided. The Search delivers only PDF documents for individual “cells” in the overall model, with titles such as “Software Development Level 5 Role”. Note the use of “Level 5” which is not the categorisation used in the NOS. The content appears to be cloned from the NOS, but the sub-elements have been reorganised and you have to look at the content to infer that Level 5 equates to Professional.

There’s no link, as there used to be, back from these framework documents to the Sector Council or to the overall Suite, and there’s no search which will identify appropriate suites for a capability (as was the case on the old NOS website). Link to Search for indexes for both “Occupations” and “Suites”, but this assumes you already know what you’re looking for …

This is a horribly confused and confusing situation.

• IT Professional Competency model  (e-skills Procom), in the National Archive
• e-skills NOS page: look for links to IT/Telecom Professional and IT User frameworks
Skills Framework for the Information Age (SFIA)
• IT Skills Academy: IT Professional Standards, and the simplified My IT Professional Profile tool
• See: National Skills academy framework backed by UK employers, Computer Weekly, 4 Oct 2012
• The NOS website is now maintained by the UK Commission for Employment and Skills (UKCES). The former URL ( redirects here.
• The NOS Search page is indexes, not searches. It has tabs for Organisations, Occupations and Suites.