Evaluating Information Systems

Zahir Irani, Peter Love

• Publisher: Routledge
• Year: 2008
• Print ISBN: 9781138152649
• eBook ISBN: 9781136404863
• Topic: Business
• Subtopic: Business General
• Index: Business

The adoption of Information Technology (IT) and Information Systems (IS) represents significant financial investments, with alternative perspectives to the evaluation domain coming from both the public and private sectors. As a result of increasing IT/IS budgets and their growing significance within the development of an organizational infrastructure, the evaluation and performance measurement of new technology remains a perennial issue for management. This book offers a refreshing and updated insight into the social fabric and technical dimensions of IT/IS evaluation together with insights into approaches used to measure the impact of information systems on its stakeholders. In doing so, it describes the portfolio of appraisal techniques that support the justification of IT/IS investments. Evaluating Information Systems explores the concept of evaluation as an evolutionary and dynamic process that takes into account the ability of enterprise technologies to integrate information systems within and between organisations. In particular, when set against a backdrop of organisational learning. It examines the changing portfolio of benefits, costs and risks associated with the adoption and diffusion of technology in today's global marketplace. Finally approaches to impact assessment through performance management and benchmarking is discussed.

Chapter 1

Information technology and systems justification

A. Gunasekaran, E. W. T. Ngai and R. E. McGaughey

Introduction

Globalisation, together with the development of information technology/information systems (IT/IS), have had a tremendous impact on the way organisations function. These developments influence the strategies, tactics and operational decisions of organisations. The application of marketing principles to the transfer and commercialisation of technology is a vital policy issue in the highly competitive global marketplace (Drury and Farhoomand, 1999). IT has spread to the extent that it is hard to envision any organisation in an industrialised nation, and even in a developing nation, not using some form of IT/IS. The rapid growth of IT investments has put pressure on management to take into account investment risks and payoffs in decision-making (Kim and Sanders, 2002). Comprehensive but understandable methodologies are needed to solve the complicated project justification problems arising from the complexity of new technologies.

Selecting and effectively pursuing the right IT/IS investments can be a key factor in sustaining corporate viability and prosperity (Bacon, 1992). Managers have employed various methods for evaluating the cost and benefits of IT/IS investment, ranging from simple computational formula to very complex techniques that blend quantitative and qualitative analysis into one. Quantitative techniques have been used mainly for capital investment decisions, but they do not necessarily capture the entire impact of new technology adoption (Shank and Govindarajan, 1992). Justification difficulties are linked to the intangible nature of the benefits promised by IT, such as improved communication and control, enhanced capabilities, and competitive advantage (Senn, 1989), and likewise to costs that are difficult to quantify.

The original version of this chapter has been published in European Journal of Operational Research, Volume 173, No. 3 (2006), 957–983.

Existing models for IS project selection typically do not reflect interdependencies among criteria and candidate projects. Careful and deliberate consideration of these interdependencies among criteria captures valuable cost savings and benefits that accrue to organisations from the use of IT/IS. Lee and Kim (2000) argue that IS project selection problems are multi-criteria, decision-making (MCDM) problems. They suggest that it is very important to incorporate multiple opinions in order to understand the interdependence among criteria and candidate projects. They further suggest an improved IS project selection methodology, which reflects interdependencies among evaluation criteria and candidate projects, using an analytic network process (ANP) within a zero-one goal programming (ZOGP) model. Their rather complex approach highlights the importance of considering the implications of IT/IS from multidimensional perspectives and the combined impact of IT/IS in all areas.

The level of investment and the associated risks involved in planning and implementing IT/IS, coupled with less than stellar results reported by many companies, highlight the need for improved understanding of the IT/IS justification process. A company's success, or lack thereof, in this important area can have a significant impact on its competitiveness. In the literature, there is a considerable volume of research reported on IT/IS evaluation, but a balanced and practical framework has yet to be established. What is proposed here is a simple, balanced, and more complete framework for IT/IS evaluation.

The chapter is organised as follows: First, IT/IS evaluation and justification is discussed. Second, the details of the research methodology are presented. Third, the classification scheme used for reviewing the selected literature available on IT/IS evaluation and justification is discussed. Fourth, a brief review of the literature on IT/IS evaluation and justification is presented. Fifth, a framework has been developed for the design and implementation of IT/IS justification. Finally, it concludes.

IT/IS evaluation and justification

IT investment studies are important, especially in the current business environment, because of the large sums of money spent on IT/IS projects as well as the risks. The benefits are expected to be of strategic value. In the published literature there is little consensus about why some IT projects succeed and others fail. A common criticism is that companies do not give due consideration to intangibles which produce long-term strategic and operational benefits (Anandarajan and Wen, 1999). Anandarajan and Wen (1999) concluded that IT investment evaluation is very complex, and because IT becomes more sophisticated over time, we may never have a complete understanding of the full range of costs and benefits associated with it. This suggests that there is a need to develop a balanced approach to evaluating investment in IT/IS projects. For example, if the evaluation relates to IT/IS in services, one might need to assign more weight to intangibles, and if it is in manufacturing, one might need to assign higher weight to tangibles. With services or manufacturing, different situations will require different weights be assigned to financial, non-financial, tangible and intangible criteria. For example, when evaluating IT/IS projects in marketing, more weight might be given to financial performance and intangibles like customer satisfaction, but an IT/IS project in manufacturing might necessitate more weight be assigned to non-financial performance measures like capacity utilisation, and intangibles like flexibility.

The use of traditional appraisal techniques to justify investments in IT and IS has received a lot of attention in recent years. This is due to the increased worldwide investment in IT/IS. Companies need to justify the investment in IT/IS taking into account costs and benefits, both short-term and long-term. Small and Chen (1995) and Alshawi et al. (2003) argue that a lack of management guidelines for use in investment decisions could encourage decision-makers to:

refuse to implement an IT infrastructure that could be beneficial to the long-term competitiveness of the organisation;

invest in IT as an ‘act of faith’ or

use creative accounting (assigning arbitrary values to benefits and costs) as a means of passing the budgetary process.

The justification process is a major concern for those organisations considering the development of an IT/IS infrastructure. Weak justification may be putting the competitive advantage of many companies at risk. The reason is management's inability to evaluate the holistic implications of adopting new technology, both in terms of the benefits and costs. The adoption of new technology, especially IT/IS, can clearly be one of the most lengthy, expensive and complex tasks that a firm can undertake. The level of investment and high degree of uncertainty associated with the adoption of this technology implies that issues involving project justification should assume great importance (Irani et al., 2002).

Although there is a considerable body of literature on IT/IS evaluation, gaps still exist regarding the application of the research findings. The following are some of the deficiencies discovered in the existing literature:

No precise definition of various performance measures and metrics based on the organisational characteristics.

No clear distinction among strategic, tactical and operational performance measures and metrics.

Identification and measurement of intangibles and other non-financial performance measures relevant in IT/IS justification is problematic and often neglected.

No good framework for assessing the implications of IT/IS on organisational performance measures during the implementation of IT/IS.

Absence of risk measurement and management models for IT/IS implementation.

No proven optimisation models for IT/IS evaluation.

No good framework for selection of suitable tools or techniques for IT/IS investment evaluation.

The current literature is reviewed in an effort to develop a contemporary framework that sheds more light on these deficiencies and moves us in the direction of bridging these gaps.

Research methodology

A literature review was the research methodology employed for developing the proposed framework for IT/IS evaluation. The literature review focused on articles in scholarly journals published over the last 14 years. In addition to classifying the literature on IT/IS evaluation, techniques and tools used to model and analyze IT/IS projects environments were identified. The following were the major objectives of this study:

Highlight the significance of IT/IS justification and its implications for organisational performance.

Identify the major performance criteria that should be considered in evaluating and justifying IT/IS.

Identify the techniques and tools used for evaluating and justifying IT/IS projects.

Focus attention on evaluating the implementation stage of IT/IS projects.

The literature review was undertaken to identify articles and provide information that should be helpful to researchers and practitioners studying, or involved in, IT/IS evaluation. The literature on IT/IS evaluation and closely associated topics was reviewed and classified in the following sections.

Classification of the literature on IT/IS evaluation

There are several survey-type articles on the evaluation of IT/IS investment. Included among them are the works of Ezingeard et al. (1999), Irani (2002) and Irani and Love (2001). Most articles focus on different types of performance measures such as strategic, tactical and operational, tangibles, intangibles, financial and non-financial. Some articles focus on the techniques and tools used in evaluating IT/IS projects and investments; however, there are few that provide a comprehensive review of articles ranging from the conceptual, to IT/IS investment evaluation, to evaluating the implementation of IT/IS projects. From this review of literature on IT/IS evaluation, a generic framework is proposed which could assist researchers and practitioners in studying or evaluating the costs and benefits of IT/IS.

IS can improve organisational efficiency and effectiveness, thereby providing competitive advantage. Irani (2002) argues that, although many of the savings resulting from IS are considered suitable for inclusion within the traditional accountancy framework, it is the intangibles and non-financial benefits, together with indirect project costs, that complicate the justification process. Irani (2002) reviewed the normative literature in the area of IS justification, and then proposed a set of conjectures. Many managers now appreciate the wider strategic implications of developing a robust and responsive IT infrastructure, yet businesses still wrestle with the dilemma of how to assess, quantify and accommodate the implications of infrastructural investments within traditional methods of appraisal. Managers still struggle with identifying and measuring the strategic implications of IT/IS. There is no suitable framework to guide them.

The literature on IT/IS justification has been divided into four major areas (gleaned from the literature review) that are considered essential for developing and managing IT/IS justification (see Table 1.1). The four major areas include: (i) general IT/IS evaluation and justification concepts, (ii) evaluation criteria for justifying IT/IS projects, (iii) techniques and tools used for evaluating and justifying IT/IS projects and (iv) evaluation of the implementation of IT/IS projects. The proposed framework for IT/IS justification addresses these four areas. The major classifications and articles that come under each are presented in Table 1.1. The details of the four areas of interest are discussed in the next section.

Table 1.1 Classification of the literature available on IT/IS justification

Classification criteria	References
General IT/IS evaluation and justification concepts	Clemons (1991); Willcocks (1992a, b); Sethi et al. (1993); Hitt and Brynjolfsson (1996); Grover et al. (1998); Ballantine and Stray (1999); Lubbe and Remenyi (1999); Bharadwaj (2000); Chircu and Kauffman (2000); Devaraj and Kohli (2000); Larsen and Bloniarz (2000); Sircar et al. (2000); Massey et al. (2001); Alshawi et al. (2003); Khoumbati et al. (2006); Connell and Young (2007)
Evaluation criteria emphasised for justifying IT/IS projects	Chokhani (1992); Davis et al. (1992); Banker et al. (1993); Belcher and Watson (1993); Powell (1993); Ballantine and Stray (1999); Drury and Farhoomand (1999); Ryan and Harrison (2000); Sarkis and Sundarraj (2000); Tallon et al. (2000); Croteau and Bergeron (2001); Gunasekaran et al. (2001); Kushniruk et al. (2001); Love and Irani (2001); Benaroch (2002); Chatterjee et al. (2002); Hitt et al. (2002); Irani et al. (2002); Kazanjian...

Table of contents

• Front Cover
• Half Title
• Title Page
• Copyright
• Contents
• Preface
• List of contributors
• Information systems evaluation: A crisis of understanding
• 1 Information technology and systems justification
• 2 Revisiting the content, context and process of IS evaluation
• 3 How IT evaluation methods are used: Examining case research from an ANT perspective
• 4 IT project evaluation: Why more formal evaluation is not necessarily better
• 5 Don’t mention the war: Managing disbenefits
• 6 Spot the difference: Evaluation of custom and package applications
• 7 Post-implementation evaluation of IT systems: A close review of practice
• 8 Evaluation of information service providers in support of the fixed-income market
• 9 Evaluating actors and factors associated with healthcare information systems
• 10 Evaluating the governance structure for public sector IT: The UK National Programme in the Health Service
• 11 Strategic alignment and a culture innovation: Using the SPRINT methodology to meet two challenges of information age government
• 12 Social dimension of IT/IS evaluation: Views from the public sector
• 13 Benchmarking e-Government: Improving the national and international measurement, evaluation and comparison of e-Government
• 14 Evaluating e-Government infrastructure through enterprise application integration (EAI)
• 15 t-Government for benefit realisation: A research agenda
• Index