|
|
Effective Benchmarks IT Can Use to Gauge Success
(January 30, 2009)
|
 |
In this roundtable discussion, each high-level IT executive was asked to
comment on benchmarks that he or she regularly uses to gauge IT success.
Some that were mentioned included:
• IT expenses in capital as percentage of revenue
• Users per IT professional
• Successful releases into production
• User perceptions and confidence
• Customer satisfaction
• Willingness to recommend to another client
• Number of tickets
• Time spent on Maintenance of Business (MOB) vs. new
development/projects
• The cost to maintain and operate organization systems equipment
(MOOSE)
• Percent of work that is rework (within scope)
• How many customizations in the ERP?
• Time to market where IT is the product
No one set of metrics is right for all firms, but developing,
incorporating, and tracking metrics over time is an essential management
tool.
End-User Survey
A popular and effective benchmark is the end-user survey. To make this
tool effective some companies are designing more scientific surveys and
engineering questions to provide feedback of what the company is doing
well, not just what is wrong. Many times surveys involve a five point
scale; most attendees agreed that expecting a perfect five as an average
is unrealistic and understand that a number slightly lower than perfect
will be the most realistic goal. How the number changes over time may be
the most important information.
It is also important to know which type of user survey is most effective
for the company. Which type of customer does the IT department service?
High-level business users or low level employee concerns. A “5” from
executives is more important than from typical employees, where a “3”
might be sufficient. One has to weigh the costs of trying to get to a
“5” versus the benefits.
Many times when a user’s service ticket is resolved, an automatic survey
is generated with multiple choice answers as well as open ended
questions. It may be important to make sure that the same person does
not receive the same survey twice in the same month. Also when
outsourcing IT services, it is best to monitor surveys internally.
Assess survey results frequently and report the overall process at least
quarterly to executive management.
Publishing survey results builds accountability with users filling out
the surveys; they feel their opinion is valued and are more likely to
fill out surveys in the future. Making IT benchmark results transparent
in an organization can be very effective for an IT department. Other
service departments will have little opportunity to blame their own
internal performance failures on “system failures” if the facts say
otherwise. This may take the form of incorporating results on the
company intranet home page or regularly publishing results in managerial
reports.
Free and Inexpensive Resources
Smaller firms may prefer to keep their benchmarks simple such as
tracking the number of complaints or ticket volume, or even forming
focus groups. Inexpensive, effective survey tools suggested for small
companies include surveymonkey.com as well as the Google Docs Survey
tool. There are many useful benchmark resources on the internet.
Infotech (www.infotech.com) offers an “IT Budget and Staffing Report”
that provides a summary on 60 different benchmarks IT can use in an
organization. The 2007-2008 report is currently free on their website.
NOREX (www.norex.net) is a site where you can share IT experiences with
a large number of organizations. You can hold regional forums or
symposiums and it is inexpensive; for about $5,000 anyone in your
company can access the site’s vast resources. In comparison, APQC wanted
$2,500 for just four surveys from one of our participants, with the
lowest cost being $800. An advantage to the NOREX data was having the
ability to understand what assumptions go with various benchmarks. For
example, in thinking about capital expenses, is depreciation included?
Comparing ratios of a firm that does include versus a firm that does not
can be grossly misleading.
Other free and inexpensive IT data resources include mining your current
vendor list for average IT costs, signing up for services like Gartner
research on a trial basis, or searching the internet for companies that
have purchased and published Gartner research on their sites already,
such as in an industry newsletter.
Other Benchmarks
Tracking the proportion of time an IT department focuses on Maintenance
of the Business (MOB) is another effective benchmark. How much time is
spent maintaining internal IT services and how much is spent outside of
that time, such as on new upgrades to software or switches? By
forecasting dashboards and analyzing them frequently a company can track
IT management over time.
The cost to Maintain and Operate Organization Systems Equipment (MOOSE)
is a benchmark that totals all salaried IT staff. If your company
stopped selling orders, how long could your IT department operate?
Quantifying this annual number allows a firm to compare its own
performance annually rather than to other companies who could have
significantly different variables. This benchmark can help you determine
whether you want to spend time on new implementation or maintaining
current operations.
Another benchmark used relates to the effectiveness of software
development. Measure the amount of rework needed after concluding a
project (of things that were in scope), or how many times you need to
return to fix problems (break fixes). Of course metrics can work both
ways. User-initiated scope creep needs to be measured as well.
Other more common benchmarks include dividing total IT expense by total
company revenue as well as calculating the number of users supported
divided by each IT professional. Regularly monitoring these measurements
builds history so the company can measure information services over
time.
Another form of benchmark is to see whether or not ROI projections have
panned out. Most firms in the room did not do this very often. Mainly
this was done only for the biggest projects. One participant related an
instance where no one showed up to the “year later” review. IT can show
costs, but to show the benefits on the business side remains elusive.
Some organizations try to capture best practices and measure against
them. But these can differ across organizations dramatically. Whereas
one organization may think that four software releases per year is a
good target, another firm may release numerous mini-releases of patches,
security upgrades, and other minor fixes on an almost continuous basis,
especially is the software is being used as a service online.
Distinguishing between what is an effective practice in your company
versus what is an industry “best practice” is important.
Linking Compensation to Metrics
Many local companies have improved individual and team goals by linking
small proportions (anywhere from 2% to 10%) of annual compensation with
IT metric outcomes. Other approaches link customer satisfaction surveys
to compensation; sometimes a certain number of surveys must be reported
before a bonus will be awarded. Companies have reported improvements in
IT benchmark measurements after compensation was linked to metrics.
Next Meeting (March 27, 1:00 pm to 4:00 pm)
Several potential topics were discussed for future ITEE meetings,
including:
• Data protection & privacy
• Collaboration technology, including fielding a reliable desktop
videoconferencing experience
• Changes in the web infrastructure, requirements for Web 2.0 and Web
3.0 in the future (examples: buy online, pick up at store; analytics;
web advertising)
The topic we chose for next meeting is The Open Source Software
Proposition.
This summary was prepared by Sara Lucas, The
University of Akron | | | |
