1 Introduction

In light of technical limitations and considerations of management structure, the introduction of enterprise information systems in the past was mostly constructed according to functional areas needs. Although the needs for the internal operations of each department might be supported by an independent information system, different information technology systems were adopted by all departments. Likewise, different operating systems and hardware were used, and issues on overall enterprise and cross-department information integration were ignored. Because of this, each department became an isolated information island. The internal information in the enterprise cannot be exchanged or reconciled through computer systems. Furthermore, the overlapping input tasks also considerably affected the efficiency of such information systems.

The emergence of the Enterprise Resource Planning (ERP) system solves the problem described above because it acts as a management system, integrating all the information of the enterprise, including the marketing, sales, and planning process. This helps enterprise managers to generate the best decisions. The ERP system is a single software, which can integrate all departments and functions within the enterprise (Umble et al. 2003). It can also be regarded as a software module family where each module shares a database and closely connects with each other in order to support the operational procedure of the enterprise (Hammer 2002).

The introduction of the ERP system is not a simple process. Enterprises must first have a clear prospect of available resources and future visions. They also must understand what effects and values will emerge once the ERP is introduced, and consider if these outputs will match future visions and goals. If necessary, enterprises also need to conduct business process reengineering to achieve such benefits. Therefore, the process of introducing the ERP system is relatively complex and extremely risky especially since implementation failures are sometimes reported (Bingi et al. 2001; Aloini et al. 2007). With respect to the capital investment, the introduction of the ERP system is considered an investment project that entails an enormous amount of capital. The cost of such a system could range from hundreds of thousands up to several million dollars. Other investments involved in the ERP project such as labor force, hardware, database, users’ training, and enterprise reform management are all considered as necessary costs. It is estimated that enterprises around the world spend about 1 billion dollars on ERP systems every year (Yusuf et al. 2004). In addition, McHugh (2000) cited a survey in March 2000 of executives in charge of the ERP system’s introduction in 100 American enterprises revealed that merely 1/3 of the executives made a positive evaluation, and only 37% of the respondents could identify the tangible effects of the system on the business’ finances.

The introduction of the ERP system is relatively risky, and its efficiency might not be obvious. Thus, after enterprises swarm to introduce the ERP system, the major issue is how to effectively manage the system in order to allow it to fully accomplish the expected performance (Hunton et al. 2003). According to Kaplan and Norton (1996), if one cannot measure it (i.e. the ERP system) then one cannot handle it. Therefore, for enterprises intending to better manage and measure its IS/ERP system performance, it is wise to start measuring from the start of the system introduction/use so that they would have a better knowledge of their enterprise performance and can evaluate whether the IS/ERP system actually produces its intended benefits. Moreover, the management and performance appraisal of information technology differ from those involved in other financial investment projects, in which the relationship between purpose and effectiveness is mostly simple, clear, and quantifiable. The effectiveness of the enterprise brought on by investment in information technology includes many invisible manifestations of effectiveness (Brynjolfsson and Hitt 2000). The difficulty of performance appraisal and the management of information technology investments cannot be measured by the traditional accounting performance assessment method.

Although the academe has presented various studies (see (Norris et al. 2000; Poston and Grabski 2001; Nicolaou 2004)) with respect to the effectiveness of the ERP system’s introduction, the studies tended to focus on the verification or categorization of effectiveness, sometimes using a case company to validate the effectiveness generalized by past literature. There is apparently a lack of complete, objective, and measurable effectiveness assessment models and appraisal methods. Thus, enterprises that have already utilized the ERP system could not recognize if the project’s introduction was efficient.

Therefore, the main objective of this research is to generalize the collected previous literature related to the effectiveness of the ERP system’s introduction, and use the Balanced Scorecard to design a complete effectiveness measurement method for the ERP system, which is validated using a real case. The goal is for enterprises to use an objective and quantitative method to clearly examine the benefits associated with the system’s introduction and avoid high risks involved in the process.

The subsequent paper consists of five sections. First section contains the discussions on ERP system management and the Performance Appraisal Theory. In the second section, the methods used in this research are explained, and an in-depth description of the process on how the model in this research is established is given; and in third section, data related to the case company is used to validate the model. The fourth section summarizes the study and provides the research conclusions.

2 Theoretical background and discussion

2.1 ERP system management

Akkermans et al. (2003) believed that the ERP can offer the following functions. First, it can replace outdated, old systems by using integration, the latest technology, and a maintainable software. Second, the system offers enterprises a framework of transaction, and allows specific operational procedures for projects within the enterprise scale to be closely connected with each other. Third, the ERP system can also help transform a function-oriented organization into an operation-oriented one. It mainly accomplishes these assignments through financial, manufacturing, logistics support, distribution, and human resource modules. These five major modules can operate independently for implementing specific functions of an enterprise as well as connect with each other. Therefore, the ERP system can execute the original function as well as fulfill the goal of integrating the internal data of the enterprise through data exchanges among the modules in order to accomplish the daily operations, which are considerably significant for the enterprise (Umble et al. 2003).

According to the collection and reorganization of the previous literature related to the effectiveness of introducing the ERP system, the reasons for enterprises’ introduction of such a system included, but was not limited to, global operational management, close connection of each functional data system, reduction of operational costs, upgrades of enterprise operational efficiency, enhancement of enterprise decision-making quality and management efficiency (Reinhard and Bergamaschi 2001). These are mostly intended to solve problems with data integration and increase the enterprise’s competitiveness. However, can the ERP system, with its powerful functions, really solve these problems and bring huge benefits for enterprises? According to some local and overseas literature, once an enterprise introduces the ERP system it can realize the benefits associated with it. These benefits include the ease of saving and receiving of data, integration of operational processes, visibility of data, and increase in overall enterprise operational activities quality (Olhager and Selldin 2003). The ERP can also integrate corporate information and reach the corporate goals of rapid delivery of goods, lower costs, internationalization, and improvement of the whole enterprise’s performance (Yen et al. 2002).

Based on the study of Deloitte, the benefits of ERP’s include reduction of stocks, trimming of labor force, increase of output, improvement of order management, reduction of IT and purchasing costs, improvement of cash flow management, increase of profits, reduction of transportation and logistics costs, reduction of system maintenance requirements, improvement of the ratio for immediate goods delivery, reinforcement of the visibility of corporate information, offering of the latest or best operational procedure, improvement of response time to customers’ needs, reduction of costs out of expectation, close connection among the systems, increase of flexibility, data sharing in the whole company, solution of the Y2K problem, and finally, improvement of overall corporate efficiency (Majed et al. 2003).

Other benefits of the ERP system suggested by Yusuf et al. (2004) include the improvement of supply chain management through e-communication and e-commerce, reducing operational costs, offering the information needed by the clients, and management’s ability to treat external suppliers, corporate alliances, and clients as a virtual enterprise.

2.2 Performance appraisal theory

Performance appraisal is an initial and critical part of management (Evans et al. 1996), as it can clearly describe the past and current situations, and function as the reference for future management (Stadtler and Kilger 2000). Therefore, in order to manage the ERP system well, the enterprise must initially have a proper performance appraisal model to assess its ERP system.

After collecting and consolidating the past literature pertaining to performance appraisal, it was discovered that the performance appraisal theory has been consistently evolving since its inception. In the beginning, it was conceived of as Univariate Effectiveness Measures; however, Steers (1977) indicated that Univariate Effectiveness Measures tended to merely assess one facet of performance and could not reveal the whole situation. Besides, it was difficult to define and select, but very easy to be affected by individual researchers’ subjective factors. Thus, the theory evolved into Multivariate Effectiveness Measures. However, Steers still point out that Multivariate Effectiveness Measures usually lack mutual indicators. In addition, there was also no mutual principle for selecting performance appraisal indicators, which tended to be difficult to quantify and define in terms of relative weighting. In addition, the position of assessors would also affect the selection of indicators.

The theory further evolved to include financial statement analysis. However, Booth (1996) believed that this method focused on short-term assessment instead of long-term appraisal. Enterprises would tend to sacrifice long-term competitive advantages for short-term benefits, and it would be easy for them to neglect other critical information when their only focus was the financial aspect. This assessment system merely stressed the performance of the departments instead of satisfying customers’ needs. The theory was thus turned into overall analysis. Although overall analysis considered all aspects of the enterprise, it could not usually convert the enterprise’s overall strategies and goals into performance appraisal indicators.

Epstein and Manzoni (1997) indicated that there will be three major trends with respect to the development of a prospective performance system. One, future performance systems established by the enterprise will support the implementation of strategies. Two, future performance systems must include non-financial indicators in order to replenish financial indicators, and three; these systems must be promoted to the departments or districts, which actually create the performance for the organization.

According to the study of Gaiss (1998), the performance system developed by modern organizations must connect with the prospective strategic goals of the organizations. Thus, the overall analysis evolved into the strategic performance assessment. Kaplan and Norton proposed the concept of the Balanced Scorecard in 1992, which included four facets: learning and growth, customers, internal process, and finance. These not only involved overall performance assessment, but also combined corporate vision and strategies. It has become an emerging tool for enterprises to properly evaluate overall performance.

Milis and Mercken (2004) organized and compared traditional capital investment assessment mechanisms, such as payback period, ARR (Accounting Rate of Return), ROI (Return on Investment), IRR (Internal Rate of Return), NPV (Net Present Value), and other emerging methods and techniques in their article. They indicated that these mechanisms were difficult to use to explain the associated intangible costs and benefits before and after the IT solutions. In addition, Clemons and Weber (1990) pointed out that most of these mechanisms were at the stage of conceptualization and could not yet be accepted by the public. Therefore, Milis and Mercken (2004) finally and enthusiastically recommended the use of the Balanced Scorecard as the proper assessment mechanism to evaluate the investment project of information technology.

3 Research method and the construction of the ERP performance assessment model

3.1 The research method and design

This research adopts a series of research methods, tools and approaches to collect and analyze data, and to propose an ERP effectiveness assessment model. The research methods and approaches used are the Grounded Theory, and case study, whereas the research tools utilized are the balanced scorecard, analytical hierarchy analysis and fuzzy logic. The flow chart of this research is as shown in Fig. 1.

Fig. 1
figure 1

Flow chart of the research

Full size image

The descriptions of the flow in Fig. 1 are given as below.

3.1.1 Stage 1–2: Grounded theory and balanced scorecard—synthesizing the list of effectiveness of ERP system

Grounded theory is a research method which allows the investigator(s) to play the role as the primary instrument of data collection and analysis. Its end result is a theory emerging from (or is “grounded” in) the data; and it is useful to practice, has its referent specific, everyday world situation (Merriam 1998).

In order to construct the Effectiveness Assessment Model for the ERP System’s introduction, the Grounded Theory is used because no theory is assumed in advance in this study. Instead, they allowed the theory to be presented through the data. The Grounded Theory uses the statements and concepts in the original data and applies the methods of reorganization, analysis, constant comparison, and coding. The stage of open coding is used to analyze, examine, compare the data, and further name the phenomenon. The same phenomenon can be categorized into one group. The phase of axial coding is done to connect the subcategory and the main category according to their respective characteristics through deduction and generalization, and allows them to be correlated with each other. Therefore, the Grounded Theory collects and analyzes data through a systematic method, and is considered to be a critical means to reorganize qualitative data (Strauss and Corbin 1990). It can also be the most scientific and rational method among the different qualitative methodologies (Hammersley 1989).

This research analyzed the meaning of different kinds of effectiveness concepts reported in the use of the ERP system, then combined those with similar concepts, and refined them into a single intuitive concept of effectiveness of the ERP system. Similarly, the authors subsequently analyze, evaluate, and simplify all other effectiveness concepts and group the same concept into the same category.

The introduction of the ERP system should be combined with the existing corporate vision framework, organizational procedure, and strategic principles. The Balanced Scorecard can manage overall performance evaluations and combine the vision and strategies of the enterprise.

The Balanced Scorecard (BSC) is a performance management and measurement tool; it is a concept for measuring whether the micro operational activities of a company are aligned with its macro objectives in terms of vision and strategy. Its underlying rationale is that measuring an organization’s performance mainly based on the financial perspective is not sufficient as this effort cannot directly influence financial outcomes (Kaplan and Norton 1992). It proposes that managers to select measures from three additional categories or perspectives: customer, internal business processes and learning and growth (Kaplan and Norton 1992). This stage also focuses on the four major facets of the Balanced Scorecard, and allocates each category found from the Grounded Theory stage to these four facets according to their respective characteristics in order to construct the effectiveness framework of ERP system introduction, which is mainly based upon the Balanced Scorecard.

3.1.2 Stage 3–5: Questionnaire, analytical hierarchy process and fuzzy theory—designing and refining the performance assessment indicators

Performance assessment indicators of different effectiveness facets extracted from the previous literature related to performance indicators could have been incorporated with subjective views. Therefore, we administered the questionnaire to collect the opinions of professionals from the academe, industry circles, and government agencies, which can filter out and improve these performance indicators. After calculating the Content Validity Ratio (CVR) of each performance indicator using the Analytic Hierarchy Process (AHP), we can obtain the salient performance evaluation indicators as the base for judging the performance of each facet.

The Analytic Hierarchy Process (AHP) is a mathematically based theory method for selecting competing solutions/ activities using distinct criteria, which can be quantitative or qualitative (Marakas 1998). The AHP offers a systematic way to weight multiple criteria aim to achieve the organizational goals by evaluating alternative solutions. The AHP can solve non-structural problems and is mainly applied to support decision making. Since different enterprises have different views when it comes to the significance of each performance indicator, this research adopts the AHP method in order to calculate the relative weight of each performance indicator.

Through administering questionnaires, this research gives understanding to the enterprises’ perceptual differences with regard to assessment indicator levels, and uses the Fuzzy Theory (Zadeh 1965). Fuzzy theory permits the gradual assessment of the membership of elements (in the real unit interval [0, 1]) in a set (Zadeh 1965). It makes use of approximate reasoning rather than strict rule for set membership, and modeling how humans obtain information from imprecise information and vague phenomena. A major goal is to simulate normal human reasoning, knowledge and experience in a way that can allow the computers to behave less precisely and logically than the traditional computer methods require (Turban and Aronson 2001). Fuzzy theory uses the fuzzy inference logic in order to solve for factors of uncertainty in human thoughts to transform qualitative data into quantitative data for calculating the effectiveness of the ERP system introduction.

3.1.3 Stage 6: Case study—validating the proposed model

A case study acts as an empirical inquiry under realistic conditions and applies the observed evidence to obtain the conclusion. It is not only a method to collect data or a design feature, but it is a considerably complete research method that includes design logic and specific data collection as well as analysis (Yin 1994). The method mainly accesses the incidents, personnel affairs, and activities of research targets through the perspectives of actual participants (Gall et al. 1996). Thus, it allows people to evaluate realistic situations and is also an extremely valuable scientific research method, helping researchers obtain more practical data for constructing theories (McCutcheon and Meredith 1993). In order to ensure the quality of a case study, Yin (1994) suggested that it is necessary to consider the following four measurement standards (see Table 1).

Table 1 The case study method dealing with four research design tests Yin (1994)
Full size table

In summary, this study utilizes an integrated approach to investigate the research problem of ERP system performance assessment model. The grounded theory is used because no theory is assumed in advance in this study. This method is used to analyze, examine and compare the data found in the previous literature. These performance indicator data are then categorized based on the four Balanced scorecard (BSC) perspectives. BSC is used as the performance measurement tool because it can provide overall performance evaluations and combine the vision and strategies of the enterprise. Subsequently, the Analytic Hierarchy Process (AHP) is used to calculate the relative weight of each performance indicator. The fuzzy theory is adopted to transform qualitative data into quantitative data for calculating the effectiveness or performance of the ERP system introduction. The glossary for the terms and jargons used in this study is given in the Appendix Table 19.

3.2 The construction of the ERP performance assessment model

3.2.1 Analytical result of the grounded theory

Since there are various reports on the effectiveness of the ERP’s system introduction in academia, this research first collected previous literature, which mentioned the effectiveness of the ERP’s system introduction, and then managed open coding in the Grounded Theory with respect to different effectiveness concepts in the existing literatures (Sharda et al. 1988; Appleton 1997; Poston and Grabski 2001; Gale 2002; Hunton et al. 2003; Nicolaou 2004; Matolcsy et al. 2005). One hundred sixty-three conceptualized results were identified and were transformed into 25 mutually exclusive items covering different perspectives. The detailed results of the 25 effectiveness are shown in Appendix Table 20.

Subsequently, this research allocated these 25 items of effectiveness into the four facets of the Balanced Scorecard and constructed the effectiveness framework for the ERP’s system introduction. Kaplan and Norton (1992) indicate that financial facet includes some index used to indicate whether an organization’s business operations are resulting in improvement of the bottom line. Customer facet consists of index that can be used to measure an organization’s performance from the customer perspective. Internal process facet focuses on the core competencies. Learning and growth facet contains index for evaluating an organization continuous business improvement. Hence, this study employed these criteria and allocated each item to suitable facet according to its characteristics. For example, in the internal process facet, the internal operations in the enterprise are stressed; and the critical additional value activities, which can result in the supply chain performance improvement and reinforce shareholders’ value, are also particularly emphasized. Improving the performance of the operational procedure (the 10th component of effectiveness) is certainly part of the internal operation of any enterprise, and better operational effectiveness (the 17th component of effectiveness) implies better stakeholders and shareholders value. Thus, these two items of effectiveness can be allocated within the internal process facet. The detailed results of coding are shown in Table 2.

Table 2 Result of axial coding
Full size table

According to the data reorganization and analytical results, there were more items of effectiveness allocated in the learning and growth facets than for the financial and internal process facets. This means that enterprises’ introduction of the ERP system is not only beneficial in the sense that it can upgrade current corporate performance with common investment projects, but it is also more beneficial for the future growth and competitiveness of the enterprises.

Based on the analysis of individual effectiveness, it is determined that once enterprises introduce the ERP system, assistance in e-dealing procedures will not only considerably reduce operational costs but will also improve the performance of operational processes. The response time to customers will also be reduced. In addition to providing information for decision-making, the ERP system allows the information to be more accurate and timely. The ERP system, which emphasizes cross-department information system integration, also makes information-sharing among departments easier and connections among the departments better. As to corporate productivity, the business volume and profits also increase due to the effectiveness of the system.

3.2.2 Designing the performance assessment indicators according to the grounded theory result

In terms of literatures related to performance indicators, this research developed 43 items for effectiveness measures (or performance indicators) of the ERP’s introduction based on effectiveness (Kaplan and Norton 1996; Booth 2000; Lipe and Salterio 2000; Banker et al. 2004; Milis and Mercken 2004; Dilla and Steinbart 2005). For example, “reducing the time to react” in the customer’s facet can be measured by the duration of “response time to customers’ needs” and the “ratio of immediately responding to customers’ inquiries”. Meanwhile, “offering more accurate and immediate information for decision making” in the learning and growth facet can be measured by the “information accuracy ratio” and the “information dealing time”. The detailed information is shown in Appendix Table 21 below.

3.2.3 Refining the performance indicators through the expert questionnaire

The Expert Questionnaires were distributed to the pioneers of the academe, industry circles, and governmental units related to ERP. We filed and filtered out the 43 performance assessment indicators generated from the previous stage. This stage is intended to find the key performance indicators, which the scholars and experts agreed upon to be used for the evaluation of the effectiveness of the ERP system’s introduction. There were 20 questionnaires distributed and nine valid returns, resulting in a 45% rate of return. The nine experts include three professors and six top-level managers. Based on the order of importance for each measurement indicator from the experts, this study employed the Content Validity Ratio (CVR) to check whether the indicator is critical. As there are nine experts in this study, the CVR should be greater than 0.78 in order to be selected (Lawshe 1975). The formula to compute the CVR is: CVR = (n-N/2)/(N/2). The symbol “n” indicates the key factors considered important, but not absolutely relevant by the experts. The symbol “N” represents the number of experts.

Based on the responses made in the abovementioned questionnaire, this research finally filtered out 21 key performance indicators of effectiveness as agreed upon by the experts and scholars. As to the financial facet, the researcher filtered out nine items including gross margin, net profit ratio, and revenue growth ratio, to name a few. Customer facet includes four items, for example the response time required by customers and the ratio of immediately responding to customers’ inquiries. The internal process facet involves four items such as reduction percentage of unexpected shutdown time and order dealing time. Finally, for the learning and growth facet, the researcher filtered out four items including accurate information ratio and information dealing time. The definitions of each performance indicator are described in Table 3.

Table 3 Key performance indicators of the effectiveness of the ERP system’s introduction
Full size table

3.2.4 The prototype of the ERP performance assessment model

Finally, based on the 21 filtered KPIs and on the knowledge of the AHP and Fuzzy Theory, the research constructed a prototype of the assessment model for the effectiveness of the ERP system’s introduction. The prototype includes five steps as shown in Fig. 2. The detailed description is as follows:

  1. Step 1

    Based on 21 KPIs reorganized and generalized by this study, the researchers must modify the operational definitions of indicators and the duration of data according to the characteristics and actual situations of different industries and enterprises. We designed a “Weight confirming questionnaire of the ERP system’s performance assessment indicators”, in the format as shown in Tables 4, 5, 6, 7 and 8, in order to get accurate data about the internal enterprise’s views toward the relative significance of each performance indicator. Based on the returned questionnaire, the authors enter the relative importance of each facet and its sub-criteria into the Expert Choice or other similar software to compute the Consistency Ratio (CR value must ≤0.1), and calculate the relative weight of each performance assessment indicator. For the step by step details on how to use the Expert Choice system or AHP approach can be obtained from this website: http://en.wikipedia.org/wiki/Analytic_Hierarchy_Process. The computed weight for each indicator (KPI) forms the basis for vector \( \mathop {\text{A}}\limits_\sim \).

  2. Step 2

    In the same survey instrument as in Step 1, we designed a “questionnaire of perceptual “differences for performance assessment indicator levels of enterprise’ introduction of the ERP system” to understand the enterprises’ views on the perceptual differences of each performance assessment indicator level, as well as construct its membership functions. In order to take care of the potential fuzziness in the data provided by respondents, we further partition each assessment level into two halves as illustrated in Table 9. It is assumed that the respondents indicated that the performance assessment level of gross margin is excellent, and should fall between good and excellent performance. Thus, a line is drawn between these two zones, as shown in Table 10.

Fig. 2
figure 2

Application steps figure of the assessment model

Full size image
Table 4 Pairwise comparisons of the weight (importance) of the four facets of Balanced Scorecard ERP system performance assessment indicators
Full size table
Table 5 Pairwise comparisons of the weight (importance) of the nine performance assessment indicators of the financial-facet
Full size table
Table 6 Pairwise comparisons of the weight (importance) of the four performance assessment indicators of the customer-facet
Full size table
Table 7 Pairwise comparisons of the weight (importance) of the four performance assessment indicators of the internal process-facet
Full size table
Table 8 Pairwise comparisons of the weight (importance) of the four performance assessment indicators of the learning and growth-facet
Full size table
Table 9 General format of the questionnaire of perceptual difference performance assessment level
Full size table
Table 10 Perceptual difference assessment for Gross Margin
Full size table

As for the membership function, it is set up based the relative frequency of each of the 10-part (as in Table 9) for each KPI, using the formula of X/N, where N is the total number of questionnaires and X is the number of times a group (for a KPI) was selected.

  1. Step 3

    Also in the same survey instrument as in Step 1 and 2, the true value, highest standard and lowest standard of each KPI are collected. By using these three measures, the performance division group value for each KPI is calculated based on the following formula:

    $$ {\text{Performance}}\,{\text{division}}\,{\text{group}}\,{\text{value}} = \frac{{true\,value - lowest\,standard }}{{highest\,standard\,{\text{ - }}\,{\text{lowest}}\,{\text{standard}}}} \times {\text{10}} $$
  2. Step 4

    We can now construct the performance assessment set \( \mathop {\text{R}}\limits_\sim \) by referring the performance division group value of a KPI (obtained from Step 3) to the KPI membership function (obtained from Step 2),.

  3. Step 5

    Using the weighted vector set of Step 1 and the performance assessment level set in Step 4, we obtained the fuzzy evaluation set for overall KPIs performance using the following formula:

    $$ \mathop {\text{A}}\limits_\sim \circ \mathop {\text{R}}\limits_\sim = \mathop {\text{B}}\limits_\sim $$

Where \( \mathop {\text{B}}\limits_\sim \) is the overall performance fuzzy evaluation sets, and ◦ is the Composition Operator. The corresponding value in vector \( \mathop {\text{B}}\limits_\sim \) for each set of performance score (e.g. see Table 11) is then plotted in a graph. The areas, Ai, under the graph are then computed. And, the gravity measure for each area, Ai, is calculated using this gravity formula according to its shape.

Table 11 Transformed relationship between performance scores and situations
Full size table

Finally, we applied the operation of M (•, +) to obtain the performance scores of the whole enterprise for each facet, using this formula \( \overline {\text{X}} = \frac{{\sum {\begin{array}{*{20}{c}} {\overline {{{\text{X}}_{\text{i}}}} } & {{{\text{A}}_{\text{i}}}} \\ \end{array} } }}{{\sum {{{\text{A}}_{\text{i}}}} }} \). Based on Table 11, the performance after the introduction of the ERP system can thus be recognized. For instance, if the score of performance is 65, then the situation of performance is “Good performance”.

4 Case study

This research expected to construct an objective and quantifiable effectiveness assessment model for the ERP system’s introduction. This assessment model is expected to not only be a theoretical research finding, but can also be applied to real and empirical circles. The prototype of this model is verified in a case company from the stainless steel secondary processing manufacture industry.

4.1 Background of the case company

The case company was established on Jan. 11, 1992 with a capital of 26 million New Taiwanese Dollars. At its inception, the company exclusively dealt with exports of stainless materials and other manufacturing transactions. It was later transformed to manage the business of manufactured stainless materials. In 1999, the company moved from Taichung Industrial Park to Changhua Chuan-Hsing Industrial Park. In 2000, it founded a stainless pipe professional manufacturing plant in Chang-pin Industrial Park, and became involved in the stainless pipe business. With the expansion, the case company continued to purchase advanced facilities to enhance its market competitiveness. The case company has been upholding the managerial ideas of “integrity, stability, growth, responsibility, and specialty” in the pursuit of corporate sustainable development and increases in client satisfaction. The company also expanded with a new scope of stainless materials, externally enhanced service quality and high competitiveness, internally reinforced working efficiency, and upgraded professional techniques. The company also passed the ISO9001:2000 edition of international quality system certification, and has established an image of maintaining excellent product quality.

By the end of 2004, the company’s capital grew to 353 million New Taiwanese Dollars. Around that time, there were about 100 employees. The company’s major clients included Chieh-mao, Chang-ching, Yu-lung, and Sheng-hsiang. The main suppliers were Yi-lien, Tang-jung, Tung-meng, Chian-hsin, and Asia Chemical. The major competitors were Hsin-kung, Chen-yu, Yun-chiang, etc. The business volume in 2004 was about 435 million, and the business objective for 2005 was sales of 60,000 tons of stainless materials.

The original information system of the case company was based on divisional stages to integrate corporate information. During the process of integration, it was inevitable to face differences in time and precision. Thus, in order to efficiently integrate corporate information, after assessing the factors of completeness of the software system, availability of human resources and introduction expenditure, and the suppliers’ supports exclusively for the steel industry, the case company decided to use the WorkFlow ERP system of Data Systems, the largest local ERP vendor in Taiwan. This ERP system was introduced in 2002 and it provided stock management, order management, purchasing management, manufacturing, account receivables and payables, note capital, personnel affair management, and other special purpose module such as card zipping. The company expects to completely reduce the business errors of the personnel, and upgrade the accounting closing time with consistent speed.

The process of ERP system introduction to the company occurred through slow employees’ education at the initial stage. At the phase of formal introduction, the company completely integrated to the ERP system and abandoned the old information system. During the introduction process, the top executives, such as the chairman, were all very supportive of the ERP system. The executives of each unit were also considerably cooperative and supportive.

With respect to effectiveness after introduction, the case company indicated that although the ERP system is a complete software set, they could not merely focus on certain modules. The precision of distribution module and reduction of human errors, were more prominently improved. Therefore, as to the financial facet, only account receivables and accounting management were further improved and the gross margin ratio or revenue was not as prominent. This revealed some reduction of errors and the precision of information offering. The non-financial facet showed a better effect, particularly with distribution operations. The company can rapidly reduce order-dealing time, create scheduling and delivery times, and improve client satisfaction. In addition, the employees’ operational errors were also relatively reduced, and self-learning capacities were improved. The operational process of the case company was also improved, which reinforces the image of the company. Thus, competitiveness was strengthened with the introduction of the ERP system.

4.2 Preparation for obtaining the related assessment model data

After interviewing the case company three times, the definitions of assessment indicators and the duration of data were adjusted as given in Appendix Table 22. Different questionnaires were distributed according to the departments based on their expertise, to their top and middle management. The main content of the questionnaire is divided into three parts: (1) understanding the respondents’ views toward the relative significance of the ERP system performance assessment indicator (the first step of the assessment model), (2) understanding the respondents’ views toward the evaluation level of each performance assessment indicator (excellent, medium and very bad) and their corresponding performance (the second step of the assessment model), and (3) understanding the respondents’ views toward the performance indicators. Doing this required the respondents to reply by scoring and obtaining the true value of a qualitative indicator by means of the average (the third step of the assessment model). However, the options of each section are based upon the fields and experience of each department, and the options related to the four major facets of the Balanced Scorecard allocated in the questionnaire for each department.

Subsequently, the same group of personnel in the general manager’s office was invited to provide their views toward organizational performance in another interview survey. The goal of this interview is to collect the subjective views of the corporate executives and use it to verify the result of our proposed assessment model. The measures were requested to evaluate their satisfaction level for the significance of the performance indicators at the case company (denoted as satisfaction) and their importance in comparison with its major competitors (denoted as importance) in each assessment indicator (this was measured through the five-point Likert scale, in which 1 refers to not very important/very unsatisfied and 5 means very important/very satisfied), see (Govindarajan 1988; Govindarajan and Fisher 1990). Finally, by multiplying the score of satisfaction and importance, the performance scores were obtained for this indicator. Because there are five indicators under each facet, the performance scores of all five performance indicators were then added to obtain the performance scores of the facet before standardization. The following formula was used to standardize the performance scores of the facet between 0 and 100 points.

$$ {\text{Performance}}\,{\text{score}}\,{\text{after}}\,{\text{standardization}} = \frac{{{\text{performance}}\,{\text{scores}}\,{\text{before}}\,{\text{standardization}} - 5}}{{120}} \times 100 $$

4.3 Results of the assessment model in the case company

The following describes the operational process of the assessment model in the case company:

4.3.1 Consistent ratio and weight of each performance indicator

This research used the Expert Choice 2000 software to compute the weight of each performance indicator. From the returned 24 questionnaires, 14 were found having the CR values to be less than or equal to 0.1. These valid questionnaires were subsequently used to calculate the integrated pair-wise comparison matrix and to obtain the weighting for each facet and performance indicator. The results, which are automatically computed by the Expert Choice 2000 software, are shown in Table 12.

Table 12 Weight of each facet and each performance indicator
Full size table

4.3.2 Construction of the associated function of the performance assessment indicators

Ten equal parts, ranging from “very bad performance”, “bad performance”, “average performance”, “good performance”, and “excellent performance” were provided in the questionnaire of perceptual differences of the performance assessment indicator evaluation level as the basis for group division. By using the statistics on the total number of questionnaires (N) and the number of times each group was selected (X), the relative frequency of each group (i.e. X/N) is then computed. This relative frequency refers to the level of membership. The authors then regarded the average value of each group as a representative value. The representative value refers to the x-coordinate, and relative frequency refers to the Y-axis. The authors drew these in a graph, connecting each point into a curve using the Richardson extrapolation method to expand the scale from 0 to 10 and from left to right. The curve obtained is called the membership function. The membership function of gross margin is shown here in Fig. 3.

Fig. 3
figure 3

Membership function of gross margin

Full size image

4.3.3 Transformation of performance division group value

Using the formula as discussed in the Section 3.2.4, the division group value is calculated. As for the quantitative indicator, the financial information from 1999 to 2004 were obtained and the year 2004 was considered as the true value. The best and the worst financial performance of each indicator from 1999 to 2004 were then considered as the highest and lowest standards, respectively. With regard to qualitative indicators, the researcher used a questionnaire to obtain the respondents’ subjective views on the performance of each indicator in 2004 as the true value, with 10 points as the highest and 0 as the lowest. The results are shown in Table 13.

Table 13 Division group value of performance indicators
Full size table

4.3.4 Constructing the performance assessment sets \( \mathop {\text{R}}\limits_\sim \)

This step used the division value and membership function of the performance indicators to map into the membership function value, as well as to collect all of the membership functions included in the performance evaluation set. For example, the division value of the gross margin is 5.88; and based on the membership function of gross margin shown in Fig. 3, the corresponding membership function values to the three membership function curves (i.e. very bad, medium, and excellent) are, respectively, (0.5, 0.5, and 0.5). Thus, this performance assessment set became \( \mathop {\text{R}}\limits_\sim \) = {0.5, 0.5, 0.5}. The same operation was also applied on the assessment set of the other performance indicators. Through this, all of the assessment sets were obtained.

\( \mathop {\text{R}}\limits_\sim = \left\{ {\begin{array}{*{20}{c}} {0.5} & {0.5} & {0.5} \\ {0.23} & {0.5} & {0.5} \\ {...} & {...} & {...} \\ {0.0625} & {0.125} & {0.5625} \\ {0.} & {0.09375} & {0.6875} \\ \end{array} } \right\} \), detailed information is shown in Table 14.

Table 14 Performance levels
Full size table

4.3.5 Managing fuzzy judgment by using the gravity method

The following used overall performance as an example to describe the operational process of this step. Using the results of \( \mathop {\text{A}}\limits_\sim \) and \( \mathop {\text{R}}\limits_\sim \), we calculated the overall performance fuzzy evaluation set \( \mathop {\text{B}}\limits_\sim \) of the case company.

Overall weight sets\( \mathop {\text{A}}\limits_\sim = \left\{ {\begin{array}{*{20}{c}} {0.09} & {0.065} & {...} & {0.026} & {0.023} \\ \end{array} } \right\} \)

Overall indicator assessment set\( \mathop {\text{R}}\limits_\sim = \left\{ {\begin{array}{*{20}{c}} {0.5} & {0.5} & {0.5} \\ {0.23} & {0.5} & {0.5} \\ {...} & {...} & {...} \\ {0.0625} & {0.125} & {0.5625} \\ 0 & {0.09375} & {0.6875} \\ \end{array} } \right\} \)

Overall performance fuzzy assessment set is: \( \mathop {\text{B}}\limits_\sim = \mathop {\text{A}}\limits_\sim \circ \mathop {\text{R}}\limits_\sim = \left\{ {\begin{array}{*{20}{c}} {{\text{0}}{\text{.069575}}} & {{\text{0}}{\text{.22916915}}} & {{\text{0}}{\text{.5584198}}} \\ \end{array} } \right\} \)

Three attributes of “very bad”, “medium”, and “excellent” levels were designated to represent the following range of performance facet scores: 0 ∼ 33 points, 33 ∼ 67 points, and 67 ∼ 100 points respectively. The indicator assessment membership in the fuzzy assessment set for performance was considered as the Y-coordinate and the assessment equal part middle point (e.g. if excellent represents 67 ∼ 100, so its middle point is (100-67)/2 = 83.5) as the X-axis. This was used to draw the indicator assessment for membership in the diagram. Each point was connected with straight lines and expanded externally to the zero point, i.e. (100, 0). A general performance assessment diagram was drawn. Figure 4 shows the overall general performance assessment diagram resulting from this process.

Fig. 4
figure 4

General assessment diagram of overall performance

Full size image

The diagram in Fig. 4 is then divided into four zones (where zone 1 and 4 are triangles, and zone 2 and 4 are trapezoid shapes) and applied the polygon square measure (or area) formula to obtain a measure of Ai and the individual gravity measure \( \overline {{{\text{X}}_{\text{i}}}} \), where i is zone area. For example, square measure of A1 is 0.574 (16.5 × 0.069575÷2); A2 is 5.004 [(0.069575 + 0.22916915)*(50–16.5) ÷2]; A3 is 13.192 [(0.5584198 + 0.22916915)*(83.5–50) ÷2], and A4 is 4.607 (16.5 × 0.5584198÷2). On the other hand, gravity measure of \( \overline {\text{X}} \) 1 is 11 [(0 + 16.5 + 16.5)/3]; gravity measure of \( \overline {\text{X}} \) 4 is 89 [(83.5 × 2 + 100)/3]. The formula of the whole gravity \( \overline {\text{X}} \) is: \( \overline {\text{X}} = \frac{{\sum {\begin{array}{*{20}{c}} {\overline {{{\text{X}}_{\text{i}}}} } & {{{\text{A}}_{\text{i}}}} \\ \end{array} } }}{{\sum {{{\text{A}}_{\text{i}}}} }} \). The overall gravity position is the performance score. Table 15 shows the calculation process of the overall performance score.

Table 15 Calculation process of the whole performance score
Full size table

Thus, the overall performance score is 64.55, which is a “good performance” according to Table 5. The same operational process was also applied to the performance scores of each of the other four facets. The overall performance scores and the specific performance scores of each facet were then transformed into general performance scores using Table 5. The reorganization is shown in Table 16:

Table 16 Reorganization of performance scores and the performance of the case company
Full size table

4.3.6 Result of case verification

In order to verify (1) whether the model actually matched the managerial subjective views toward the performance of the whole and each specific facet after the introduction of the ERP system, and (2) if the objectivity and quantification of this model replaces the past subjective assessment toward performance, this study compared the performance generated from the subjective assessments with the results of this model. Thus, another interview survey was distributed to the same sample group, who was the top and middle management of the case company in 2005 to collect their subjective views by rating their satisfactions with each performance indicator in 2004 and rating the importance of each performance indicator to the company. Each evaluation was based on 5-Likert-scale. Therefore, the final computed range of scores was between 1 and 25. After the questionnaires were returned, the performance scores of each facet were computed. The detailed information is shown in Table 17.

Table 17 Reorganization of performance scores of each attribute
Full size table

The authors compared Table 16 (performance scores and performance calculated through the assessment model) and Table 17 (performance scores and performance calculated through the subjective replies of top managers in the case company) and the result is reorganized as shown in Table 18.

Table 18 Comparison of the performance by different assessment methods
Full size table

According to the data found in Table 18, the performance scores questionnaire based on the manager’s subjective responses in general are lower than the performance scores generated by the assessment model. However, the score of overall performance is not significant different.

With regard to each assessment facet, the performance scores of the internal process facet and the learning & growth facet, regardless generated from the model or from subjective views, take the first and second positions in the sequence of performance. The performance levels are labeled as “good performance”, so it can be surmised that the performance of the case company was accurately obtained.

As to the performance scores of the financial facet, the financial facet calculated through the assessment model is lower than the subjective views. The reason might be that during the period of our collected data (from 1999 to 2004, i.e. before and after 2 years of the ERP system introduction), the rise and fall of performance was too extreme (for example, the highest profit growth ratio is 1939.78% and the lowest is −94.58%). As to the performance of the customer facet, although both are not at the same performance level, the performance score calculated through the assessment model is 60.10 points, which is just above the threshold of the average performance level. This shows the difference is not enormous and can still be acceptable.

With regard to the whole performance, although the performance level is different, the performance score generated from the responses of the general manager-level executives is 59.90 points. This is very close to the threshold of “good performance”, and is not so different from the 64.55 points calculated through the assessment model. Thus, the result is still pretty closed and acceptable.

5 Conclusions

This research applied the Grounded Theory to reorganize the coding of different effectiveness concepts mentioned in the literature, and found 163 items of effectiveness, which were distilled into 25 items. Based on this, 43 performance assessment indicators were designed for the 25 items of effectiveness but only 21 KPIs were confirmed through the Expert Questionnaire. Using these 21 KPIs, the balanced scorecard, AHP and the Fuzzy Theory were used to develop the prototype for the effectiveness assessment model of ERP system introduction.

Subsequently, a case study was used to verify this prototype. After comparing the research findings and the results generated from subjective view questionnaires, it was discovered that the difference between the two certainly exists since the measurements are different. However, the assessment model produced and applied in this research can generally compute the performance of the case company. The assessment model is scientific and all subjective aspects can be transformed into quantitative data. Thus, this model should be a better assessment model for evaluating the effectiveness of ERP system introduction in the future. When assessing the effectiveness of ERP system introduction, the prospective enterprises can apply this model to replace the past subjective and non-scientific evaluations. For academics, it is expected that this research would offer the basis for cross-case comparison studies on ERP system performance. Alternatively, they can do an in-depth study on real performance after the enterprise-initiated introduction of the ERP system, and explore if the enterprises have fulfilled their original performance goals after introducing the system.

In addition, due to restrictions in time, resource obtainment and sources of information, this research merely focused on one case to manage the verification of the model. In order to expand the generalizability of this assessment model, it can be further tested in different industries and companies to produce an effectiveness assessment model of ERP system introduction, which can be widely applied to all companies across different industries. In addition, future studies can conduct a thorough survey in an effort to develop a more comprehensive, objective and quantitative performance indicators to enhance the proposed effectiveness assessment model of ERP system introduction. Finally, future research can compare “Before ERP” and “After ERP” performance in the same way and provide case company an objective result to evaluate effectiveness of such IT implementation.