Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

‘If you can’t measure it, you can’t manage it’. (Thorp 1998)

IT-related organisational transformation can engender changes in business processes and work practices, which may ultimately deliver value by reducing costs, increasing output quality, enabling new product development or improving customer service (Brynjolfsson and Hitt 2000). Indeed, Gregor et al. (2006) found that variations in the level of organisational change can explain the differential effects of computer use on productivity across organisations. The recognition that most benefits from IS/IT come from changes in the way an organisation does business and not from the introduction of the new technology itself (Marchand and Peppard 2008) has engendered a whole new field of management practice and enquiry: benefits management. Benefits management (BM) has been defined as ‘the process of organizing and managing, such that the potential benefits arising from the use of IT are actually realized’ (Ward and Elvin 1999). A number of previous studies have attempted to highlight the need for proactive management of organisational change in formal benefits realisation approaches to improve the outcomes of systems development projects (Remenyi et al. 1997; Changchit et al. 1998). Indeed, Doherty et al. (2012) consider the instigation of an organisational change process that complements the new information system’s functionality as one of the defining characteristics of the benefits realisation approach. However, it is difficult if not impossible to proactively manage the process of benefits realisation management without effective tools to measure and monitor benefits.

Against this background, the primary aim of the current study was to review the academic literature that investigates different aspects of benefits measurement in order to identify and critique the benefit measurement approaches currently available. In so doing, we wanted to understand the extent to which current measurement techniques are fit for purpose in terms of facilitating the effective deployment of benefits realisation management approaches. Moreover, we were keen to identify the key priorities for future research regarding IS-/IT-enabled benefit measurement and management. The structure of this chapter is as follows: The next section provides a conceptualisation of IS-/IT-enabled benefits and discusses possible benefit classification schemes. The process of identifying and reviewing key contributions to the benefits measurement literature is then explained, before an overview of the main themes emerging from this review is presented. The chapter concludes by proposing an agenda that identifies the many areas in which future research projects could be fruitfully conducted.

2 Conceptualising IS-/IT-Enabled Benefits

Before we can discuss how benefits might best be measured, it is important to review how the term IS-/IT-enabled benefit has been conceptualised in the extant literature. This task is not as easy as it may sound, because there are few clear definitions, and many studies simply identify examples of the types of benefit that IS/IT investments deliver or classifications of these benefit types (e.g. Farbey et al. 1993; Giaglis et al. 1999; Shang and Seddon 2002), rather than attempting to explicitly defining this term. For example, Sanchez and Robert (2010) argue that all benefits can be classified as tangible or intangible depending on the ease with which they can be quantified. Remenyi et al. (1993) have defined ‘a tangible benefit as one which directly affects the firm’s profitability’. The word directly, in this definition, attempts to draw a distinct line between tangible and intangible benefits. For example, it is fairly clear that an IT system contributing to a direct cost reduction is more tangible than one that is designed to deliver management information, which may, over time, facilitate improvements in organisational decision-making. However, Murphy and Simon (2002) argue that the delineation between tangible and intangible is often quite fuzzy. In this context, quantifiable benefits are differentiated from tangible benefits in that quantifiable benefits may be measured easily but may, or may not, directly affect a firm’s profitability.

Some authors have attempted to devise benefit classifications that attempt to further decompose the broad categories of tangible/intangible or direct/indirect. For example, Williams and Parr (2006) have decomposed tangible benefits into financial and non-financial benefits. Giaglis et al. (1999) attempt to sidestep the issue of the degree of alignment between benefit tangibility and benefit directness by presenting a matrix of IS benefit types. In their matrix they differentiate between the following four benefits types: hard benefits (e.g. cost reduction), intangible benefits (e.g. improved decision-making), indirect benefits (e.g. the implementation of a LAN allowing future systems to be implemented if required) and strategic benefits (e.g. a new business strategy or better market positioning of the firm).

Strategic benefits have also been included in two other benefit classification schemes. Farbey et al. (1993) include strategic benefits as one of the five possible categories of generic benefits from IS/IT that they constructed from empirical study of project evaluation in 16 organisations. They also include management benefits (e.g. flatter organisational structure), operational benefits (e.g. timeliness and accessibility of data), functional benefits (e.g. enforcement of regulatory or legal requirements) and support benefits (e.g. improved recruitment and retention processes). These five categories are based on Mintzberg’s (1983) view of the structure of an organisation. Ward and Daniel (2006) comment that whilst the first three categories remain relatively easy to understand, the last two categories are difficult to distinguish reducing the effectiveness of the framework.

A further classification is proposed by Shang and Seddon (2002). Their framework is based on the potential benefits available from ERP system implementations and has some similarities to Farbey et al.’s classification also including strategic and operational dimensions. The first three categories of Shang and Seddon’s (2002) framework are based on Anthony’s (1965) classification of management activity into three levels: operational, managerial and strategic. To these three levels they add IT infrastructure and organisational benefits. Shang and Seddon’s framework has the advantage of being more clearly defined and derived from practitioner assessments of realised benefits, rather than planned or potential benefits. They also start to consider which types of measure may be more appropriate for measuring different benefit types differentiating between tangible and intangible benefit measures.

Whilst classifications are useful in that they provide useful insights into the types of benefits that may be realised from IT implementations, it is often difficult to apply them in practice, as the boundaries between categories are often indistinct. Against this backdrop, other academics have attempted to define benefits, rather than categorise them. For example, Thorp (1998) defines a benefit as ‘an outcome whose nature and value are considered advantageous by an organization’. Bradley (2006, p18) provides a similar perspective defining a benefit as ‘an outcome of change which is perceived as positive by a stakeholder’, and Ward and Daniel (2006) continue the emphasis on stakeholders, by defining benefits as ‘an advantage on behalf of a particular stakeholder or group of stakeholders’. Sanchez and Robert (2010) provide a greater emphasis on the measurement aspect of benefits in their definition, stating that ‘benefits are measurable improvements perceived to be a value by one or more of the stakeholders’.

Conceptualisations and definitions of benefits both provide useful academic insights, but they say very little about how benefits are actually measured, in practice. Consequently we initiated a systematic literature review to investigate and critique the benefits measures that had been reported in the academic literature.

3 The Academic Literature on Benefits Measurement

Having reviewed some of the ways in which the benefits of IS/IT have been conceptualised in the literature, it is important to turn our attention to the rather more practical issue of what we know about the reality of measuring benefits. This chapter draws on a sample of references drawn from the literature on benefit measurement and management. Due to the multidisciplinary nature of the interest in the benefits from IT projects, we searched Web of Science, ABI/Inform, Business Source Premier, EI Compendex (Engineering Village), Scopus, Google Scholar, Inspec and IEEE electronic databases for English language journal peer-reviewed articles that were published between January 1990 and June 2012. We searched for articles, using a range of terms that included benefits realization, benefits management, business benefit, benefits evaluation, business value, value engineering, value management or value realization and measurement, measures, assessment or valuation in their title, abstracts or keywords. We combined the above search terms using lemmatisation, where available, and wildcard and truncation to pick up alternatives and to account for UK and US spellings. This search generated about 100 articles, which were imported into RefWorks. The titles and abstracts of these articles were examined to classify the literature and we ultimately identified 27 articles that focused on measuring benefits or IS/IT, which we now use as the focal point for the remainder of this chapter. Initial reading and forward and back citation analysis of key articles identified several additional papers that were included in the review resulting in a total of 32 articles for analysis (see Table 16.1).

Table 16.1 Academic articles on measuring IS-/IT-enabled benefits 1990–2012
Full size table

An initial observation from the list is that the number of relevant articles, identified over this 20 year a time period, is relatively low, and it highlights the extent to which the concept of IS-/IT-enabled benefits measurement has been neglected and remains underdeveloped. This exercise also provided some interesting insights into the research methods used to study benefits measurement. The findings presented in Table 16.1 indicate that the most common method adopted, within our sample of papers, was the survey strategy, usually across multiple firms, to investigate one of the following:

  • The relationship between IS/IT and the delivery of business value (e.g. Nevo and Wade 2010; Jeffers et al. 2008; Powell and Dent-Micallef 1997)

  • The validity of an author-developed measure (e.g. Tallon and Kraemer 2007; Love et al. 2005; Mirani and Lederer 1998; Klein et al. 1997)

  • The approaches taken to measure benefits by practitioners (e.g. Gacenga et al. 2010)

Author-developed measures were the most common focus of the conceptual papers (Sanchez and Robert 2010; Rehesaar and Mead 2005; Kleist 2003; Smith and McKeen 1993), and the relationship between IS/IT investment and the delivery of business value was the primary concern of papers that adopted a financial or economic modelling techniques. A small number of studies had adopted a case study or action research approach to investigate aspects of benefit measurement compared to those following a survey design. These studies tended to focus more on softer aspects of benefits measurement addressing topics such as:

  • Intangible benefits (e.g. Sharif and Irani 2006; Murphy and Simon 2002)

  • Disbenefits (e.g. Fox 2008)

  • Different stakeholder perspectives on benefits assessment (e.g. Buccoliero et al. 2008)

In addition to shedding light on the different research approaches reported in the literature, our study also provides important new insights into the specific types of measure that can be adopted. The remainder of this section critically discusses the two most common types of measure, namely, objective and perceptual measures.

4 Objective Measures of IS-/IT-Enabled Benefits

Nevo and Wade (2010) argue that the business value of IT (BVIT) is central to the IS discipline. Many researchers have attempted to apply economic measures to assess whether the anticipated benefits of IS/IT investments are being realised. Such objective measures include profitability, productivity, costs, quality, operative efficiency, consumer surplus and Torbin’s q (Lin 2009). Many researchers have attempted to investigate how such objective measures are affected by organisational IT adoption practices. For example, Jeffers et al. (2008) report that whilst some studies have shown a positive relationship between IT spending and firm performance (Hitt et al. 2002), others have reported more mixed or negative associations (Nevo and Wade 2010; Lin 2009). Researchers have responded to these mixed results in two ways. One stream of research has focused on building process-oriented models of value creation. These scholars argue that IS/IT investments can only be measured at the intermediate process level as it is at this level that IT-enabled contributions can be seen (Barua and Mukhopadhyay 1995; Ray et al. 2004). A second stream of research considers organisational resources and investments complementary to IT (Powell and Dent-Micallef 1997; Brynjolfsson and Hitt 1998; Bresnahan et al. 2002). However, it has been argued that the application of objective, accounting-oriented measures has significant weaknesses. For example, Smith and McKeen (1993) argue that return on investment is not suitable for capturing intangible aspects that may be essential for firm survival in an industry. Revenue growth rates are attractive because systems that are designed to offer new products or services are designed to increase market share. However, isolating an IS/IT investment contribution to revenue growth is likely to be problematic making linking individual IS/IT project costs with subsequent benefits difficult. Smith and McKeen (1993) argue that there are similar problems of isolation when attempting to apply return on management or profits as a measure of business benefits. However, despite such measurement problems, organisations still seem focused on gathering objective measures even though the business reasons for investing in these systems are often to provide intangible benefits (Petter et al. 2012). Consequently, although some authors are retaining objective measures (Lin 2009; Martin-Oliver and Salas-Fumas 2012), others have moved to perceptual measures.

5 Perceptual Measures of IS-/IT-Enabled Benefits

Several researchers have attempted to move away from objective measures and offer alternative instruments for assessing benefits from IS/IT investments. These approaches mainly rely on perceptual measures of benefits and the quality of respondent’s recall rather than objective measures. For example, Mirani and Lederer (1998) developed an instrument to measure three dimensions of organisational benefits: strategic, informational and transactional using a Likert scale ranging from not a benefit to very important. More recently, Jeffers et al. (2008) and Nevo and Wade (2010) have relied on survey harvests of self-reported practitioner data on various aspects of firm performance and IS/IT contribution. Tallon and Kraemer (2007) provide a useful contribution to this debate as they assessed the efficacy of executives’ perceptions of IT impacts at both the process and firm levels. They compared data collected via a perceptual survey of 196 executives views on their firm’s IS/IT investment and performance with financial data for the same firms obtained from Standard and Poor’s Compustat database. They conclude that whilst perceptions were not a perfect proxy for hard-to-find objective measures, they were ‘sufficiently accurate, credible, and unbiased as to constitute a viable approach to IT impact assessment’.

An alternative approach to evaluating the performance of IS/IT and therefore the delivery of benefits is proposed by Skok et al. (2001) who advocate the use of importance–performance (I–P) maps. I–P maps are matrix-based techniques that seek to capture stakeholder perceptions of importance and performance, using Likert or numerical scales, which make them easy to analyse and interpret. Skok et al. (2001) assessed the application of this technique within a case study organisation and found it to be simple to administer and interpret, although it sacrifices depth for breadth of coverage. An alternative approach to addressing the challenge of assessing the value of IS/IT investments in a context of human, organisational, social and technical complexities is provided by Sharif and Irani (2006). They apply fuzzy cognitive mapping (FCM) to an IS evaluation at a single case study organisation. This approach is essentially a mind map representation of a system or set of causal statements, within a situational context, that could effectively be enumerated in terms of a simulation algorithm.

These studies indicate that adopting alternative non-accounting-based measures of IT-enabled benefits can be useful for researchers, especially when attempting to capture intangible benefit levels. However, Murphy and Simon (2002) caution that it is important to ensure that any proxy measures are agreed with business managers, to ensure that they are considered valid throughout the business. The work of Klein et al. (1997) suggests that as well as considering which measures are most appropriate for particular benefits, the nature of the technical functionality should also be considered. Objective, accounting measures may be preferable for systems that are concerned with transaction processing and thereby likely to deliver tangible benefits. However, systems that are designed to improve decision-making and have less tangible organisational impacts require perceptual-based measures or proxy measures to be able to effectively assess the delivery of business benefits. Consequently, Tallon and Kraemer (2007) recommend the use of both objective and perceptual data when assessing IS/IT impacts.

6 Benefits Measurement in Practice

Although most researchers have attempted to independently measure the benefits of organisational systems, others have attempted to assess the benefits management practices of IS professionals and a number have gathered data on current benefits measurement practices in use. For example, Berghout et al. (2011) conducted eight case studies of financial service organisations in the Netherlands to investigate the adoption of lifecycle cost and benefit management practices. They report that financial measures (e.g. return on investment) or nonfinancial indicators (e.g. achieving a strategic match) were used for IS/IT investment justification. However, they found that the cost/benefit analyses of almost all of their case study organisations were incomplete. Further, in most justification processes, investment goals were not set in a way to allow evaluation afterwards. There was a tendency to formulate qualitative (‘better than the current situation’) rather than quantitative goals (‘an improvement of 15 %’), which would complicate evaluation. They also report that there was little proactive measurement of benefits during the realisation or exploitation stages of a project with few organisations adopting service-level agreements to evaluate performance between the IT department and users or linking operation costs and business value. They conclude that few organisations conducted an evaluation of their IS/IT investments largely because of setting nonmeasurable goals and the difficulty of isolating the influences of other investments.

Slightly better experiences are reported by Gacenga et al. (2010) in their study of IT service management benefits and performance in Australian firms. They found that the balanced scorecard was the most popular measurement technique adopted by managers supported by metrics at the process level. However, they also report that many of their respondents reported difficulties in measuring and reporting benefits in their organisations, which Gacenga et al. explain through a lack of adoption of performance measurement frameworks. A further study, also conducted in Australia, has examined the uptake of benefits management and evaluation practices (Love et al. 2005). Love et al. (2005) develop benchmarking metrics at three benefit levels (strategic, tactical and operational) for SMEs implementing IS/IT investments. They also argue for the inclusion of benchmarking metrics for risk factors associated with IS/IT implementations (such as reluctance of employees to adapt to change) that may inhibit the delivery of benefits. They conclude that strategic benefits from IS/IT investments vary across industry sector and claim to have developed measures that should allow SMEs to assess their performance against similar organisations. However, Love et al. (2005) only apply these measures to assess the overall performance of SMEs in Australia rather than for application at the project level for individual SMEs. They also do not provide guidance on how these items should be measured for individual IS/IT investments, identifying several items that were considered intangible and therefore difficult to measure (Ashurst et al. 2008). Consequently, it would appear that many IS practitioners are not applying benefits measurement in practice, possibly because the challenges of benefits measurement have yet to be resolved.

7 Conclusions: The Future of Benefits-Oriented Research

In his much-cited book, The Information Paradox, Thorp (1998) emphasises the critical importance of measuring of IT-enabled benefits for ensuring the effectiveness of the benefits management processes stating ‘if you can’t measure it, you can’t manage it’. Our study, which has examined and synthesised the benefits measurement literature, has raised some important concerns about the ability of organisations to measure the benefits being realised from their IT investments, and it brings into question their ability to effectively manage the realisation of benefits. More specifically, the studies of the broad benefits measurement approaches adopted by practitioners indicate that there are significant deficiencies in their current practices: too often benefits are estimated at the outset of a project but then not systematically reappraised once the system has gone operational (Ashurst et al. 2008). The ability of organisations to effectively measure the benefits of IT is also called into question by the studies that have attempted to measure the relationship between IS/IT investment and the business value realised: the very variable results produced by these studies suggest that many organisations are neither measuring nor managing the benefits of their IT investments effectively.

In addition to highlighting serious concerns about the practice of benefits measurement, this study has also identified some significant gaps in the literature. Whilst there have been many studies that have attempted to model the IT value relationship or explored the broad evaluation approaches being adopted, there has been a dearth of research that explicitly investigates how these measures are actually operationalised within organisations and how effective they are in supporting practitioners to monitor and manage the process of IS-/IT-enabled benefits realisation. This is a very important oversight, because unless we know how benefits are currently being measured, it will be very difficult to discern how these approaches can be improved. Consequently, despite considerable interest among industry and academia in leveraging benefits and value from IS/IT investments, good results are not going to be consistently achieved without further research into the development of reliable and accepted measures. Against this backdrop, it is important that a new research agenda be established for benefits measurement, to ensure that these significant gaps are filled, as a matter of some urgency. Key issues on this agenda would include:

  • Empirical studies to investigate the application of benefits measurement techniques at the process and project levels and how these methods can be used to inform adjustive action and regular monitoring of benefits realisation. Such studies would be valuable, as they will help to bridge the theory-practice gap in benefits management (Ashurst et al. 2008).

  • Studies to investigate how objective and perceptual measures can be effectively combined to proactively measure and monitor the delivery of benefits during and following the completion of systems development projects (Tallon and Kraemer 2007). In particular, it would be valuable to investigate how such measures could be embedded in existing systems development methods to ensure that they become a core aspect of IS/IT projects rather than a nice idea that is put to one side once the business case is accepted.

  • Research that involves longitudinal data would also be helpful to assess the strengths, limitations and accuracy of existing and new benefits measurement techniques. Research methods such as case study or action research would be particularly pertinent for these studies to get a richer interpretation of the benefits journey for both academics and practitioners and thereby increase the likelihood of realising business benefits from IS/IT investments (Doherty et al. 2012).

To conclude, if the unacceptable high levels of information systems failure and underperformance are to be successfully addressed, then it is essential that members of the academic and practitioner communities actively engage with this agenda sooner, rather than later.