1 Introduction

For more than two decades, knowledge management (KM) has received attention as a crucial factor for enterprises in gaining and maintaining a competitive advantage [1, 2, 28, 31, 32, 39] to survive in an increasingly turbulent business environment. Thus, to assess KM activities becomes an important issue with various measures proposed in a growing body of research on such topic [10, 37, 40].

However, there occurs unconscious confusion between KM performance and KM effectiveness [9, 37]. Sproles [33] had distinguished the Measures of Performance (MoP) from the Measures of Effectiveness (MoE). According to him, MoP measures the internal characteristics of a solution; MoE measures external parameters that are independent of the solution, i.e. a measurement of how well the problem has been solved. Thus, to evaluate knowledge management performance (KMP) is to assess how KM works in itself and to evaluate knowledge management effectiveness (KME) is to assess how KM contributes to the organization. And this paper attempts to assess both KMP and KME, here named as the KM implementation (KMI) by establishing an evaluation model.

The steps of developing the model include: (1) review related literature to establish a conceptual framework; (2) design a pilot questionnaire to filter out proper samples; (3) survey the selected samples by a more advanced questionnaire developed according to the framework to collect objective data; (4) examine the validity and reliability of data through various statistic tests; (5) finalize the comprehensive KMI assessment model by testing through regressions.

Briefly, Sect. 2 presents the theoretical background. Section 3 formulates research hypothesis. The measurement matrix is constructed in Sect. 4. Section 5 derives and verifies the evaluation model with some conclusions drawn in Sect. 6.

2 Theoretical background

To evaluate the performance of KM, the American Productivity and Quality Center [4] and Arthur Andersen Business Consulting [5] developed a knowledge management assessment tool (KMAT) in 1995 to assist enterprises to select the appropriate type of KM. KMAT is composed of 5 elements: strategy and leadership, culture, technology, measurement and knowledge management process. Four key success factors were proposed: procedures of KM adopted, persons involved in KM, supporting organizational structure for KM and information technology utilized in KM.

Allee [3] partitioned organizational KM activities into four categories which can be further extended into organizational KM activity measures. Those categories are knowledge creation, knowledge retention, knowledge sharing and knowledge innovation. Similarly, Teece [35] stated that superior performance depends upon the ability of firms to innovate, to protect knowledge assets and to use these knowledge assets.

But Kalling [18] argued that knowledge is not always utilized and that utilized knowledge does not always result in improved performance. Thus, he suggested dividing the concept of KM into three instances: development, utilization and capitalization. Lee et al. [20] defined five components (i.e. knowledge creation, knowledge accumulation, knowledge sharing, knowledge utilization, and knowledge internalization) to use in determining the knowledge circulation process (KCP) and introduced a new metric of knowledge management performance index (KMPI) for assessing the KM performance of an enterprise. And they found that KMPI increases when KCP efficiency improves.

Chen and Chen [9] summarized KM evaluation methods used in KM performance evaluation from 1995 to 2004, into eight categories: qualitative analysis, quantitative analysis, financial indicator analysis, non-financial indicator analysis, internal performance analysis, external performance analysis, project-oriented analysis, and organizational-oriented analysis, together with their measurement matrices for different research and problem domains. It was comprehensive and extensive, but ignored the difference between effectiveness and performance of KM.

Tseng [37] partitioned the activities of KM into three processes- KM strategic, KM plan and KM plan implementation and explored the KM performance matrix. But as the KM performance was measured by financial and non-financial indexes, their study actually evaluated the effectiveness of KM as conceptualized in this paper.

Generally, KM researches that focused on identifying, storing and disseminating process related to knowledge in an organized manner have rarely done empirical work [2, 41]. And those empirical researches on KM have not clearly identified KMP and KME [9, 37]. Since there are evidence that KM positively influences the organizational performance [8, 11, 18, 20, 22], it is valuable to investigate how managers can initiate KM more effectively so as to transform the good impact of KM performance onto KM effectiveness.

3 Conceptual framework and research hypothesis

Previous research [23] has revealed that SMEs’ poor KMI results are caused by their management’s ineffective decision making in management support, culture, technology, infrastructure, measure, KM process control, etc. Thus, in consideration of SMEs’ limited resources and conditions available, making sense in decision making appears more valuable especially under today’s increasingly turbulent global environment.

This paper integrates Sense-making Methodology [13] into knowledge management to develop a holistic framework covering all the fundamental elements of KMI. Sense-Making Methodology as proposed here assumes that there exist gaps between people, things, spaces, and times and people are making sense of their world all the time. It conceptualizes information as “that sense created at a specific moment in time–space by one or more humans”. For more than 20 years, Dervin’s [13] approach was developed focusing on better understanding the ways people make sense out of information under changing conditions by mandating methods of question framing, data collection, and analysis that can be used in conducting research. Although rooted in the communication field, this study extends its application into KMI.

Specifically, it contains three sense-making modules of KMI [23]. Firstly, the enterprise makes sense of what is currently happening in their organizational environments in order to share a meaningful interpretation that serves as a context for organizational activities. Secondly, the enterprise makes sense in the process of managing knowledge through the four sense-making modes of intra-subjective, extra-subjective, inter-subjective and generic-subjective [12]. Thirdly, the enterprise makes sense in KMI decision making by aligning KM strategy closely to the overall business strategy and applying KM systems as project management.

Following this logic, the paper partitions KMI into three stages: environment analysis (EA), KM activity planning (KMAP) and KMI decision making (KMIDM). And the initial conceptual framework is depicted as in Fig. 1. The outcome is the KM Measurement Index (KMMI), and the assumed causal factors are EA, KMAP and KMIDM. The constructs that are employed to quantify the four variables are explained in the next section.

Fig. 1
figure 1

Initial conceptual model

Full size image

Since there are researches that revealed KM outcome is significantly influenced by KM activities [20, 27, 29], it is reasonable to assume firms with good environmental analysis, implementation planning and implementation decision making would achieve good KMI. Thus, the specific research hypotheses are formulated as follows:

  • RH1: If the EA is good, the KMP, KME and KMMI are positively influenced.

  • RH2: If the KMAP is good, the KMP, KME and KMMI are positively influenced.

  • RH3: If the KMIDM is good, the KMP, KME and KMMI are positively influenced.

4 Measurement and modeling

Since there is no statistical index can be used directly to measure all the variables above, a detailed description of all the adopted measurements is provided below.

Firstly, if KM is to be successfully implemented, the knowledge of the external environment that the enterprise confronts and the comprehension of the internal process it undertakes are required [36] in order to reach a common understanding of its strength, weakness, opportunities, and threats in conducting KM activities [25, 38] and to achieve a sustainable competitive advantage [19, 24]. Thus, EA is investigated from both external and internal aspects. For external analysis, three constructs including globalization, competitor and cooperator are measured, while for internal analysis, three constructs such as culture, technology and infrastructure are assessed. The specific items adopted in both analyses are selectively presented in Table 1.

Table 1 The selected constructs of EA
Full size table

Secondly, a good understanding of various KM activities allows and guarantees a systematic and comprehensive planning [15]. So far, several researchers have proposed different frameworks of KM cycle. Ruggles [30] summarized eight main activities of KM cycle: generating, accessing, using, embedding, representing, facilitating, transferring and measuring. Newman and Conrad [26] organized the process into four activities: knowledge creation, retention, transfer and utilization, through which data, information, knowledge and meta-knowledge are transformed from one state to another. Beckett and Murray [6] introduced the top-level conceptual framework with five types of activities as knowledge utilization, transfer, transformation, representation and assurance. Based on those researches, the study measures the most essential items for KMAP as depicted in Table 2.

Table 2 The selected constructs of KMAP
Full size table

Thirdly, Fliaster [14] proved that the top management support and senior levels commitment are crucial to a successful KMI. The core KM task of management is to define the ultimate goal of KM to create value closely aligned to the overall business strategy [7]. The top manager’s KM vision and resolution should be well distributed to the employees to gain their supportive acceptance for change and effective commitment for implementation [21, 29]. Sense-making [13] would suggest the inclusion of a KM project management related construct which we call KMIDM as shown below. Because of its mature discipline KMI could be controlled more systematically and completely, from the front-end planning, startup and operation, to project control as well as project performance measures. The adopted constructs and items are selectively provided in Table 3.

Table 3 The selected constructs of KMIDM
Full size table

Fourth, the KMP is evaluated from the movement of knowledge flows in terms of knowledge communication, application and learning; while KME is investigated from the effects of knowledge flows such as financial benefits, innovation, decision making and competitive advantage, etc. All the items involved to measure KMMI are shown in Table 4.

Table 4 The measurement of KMMI
Full size table

Finally, the initial conceptual model is transformed as shown in Fig. 2.

Fig. 2
figure 2

The integrated conceptual model

Full size image

5 Data analysis and regression results

The population of this paper is targeted at SMEs with employment between 50 and 250, following EU standards. The specific targets are senior managers or the directors of KM department of these enterprises due to their key roles in organizational activities [34]. The questionnaire was sent initially through e-mail and followed up by telephone call to improve the response rate. Personal face to face method and spot observation are selectively used for the accuracy of the responses. All participation in the study is voluntary. The subjects are free to withdraw at anytime without obligation and free to decline to answer any questions they do not wish to answer.

After the pilot survey, a total of 100 SMEs from China were selected for further research. And finally, 65 feedbacks were suitable for analysis, yielding an ineffective response (including non-response) rate up to 35 %. To ensure no possible bias, a telephone survey for non-respondents was conducted to find out their reasons for not responding. The results show that 33.1 % of them do not understand the topic of KM and 26.4 % have not experienced any KM activities yet, so they are unable to answer the questionnaire [22]; 40.5 % do not have time to fill it, or lost the questionnaire, etc. Thus, it implies that the 65 respondents seem to be reasonable for the study.

For those collected data, the derived high value of 0.908 for the Kaiser–Meyer–Olkin Measure of Sampling Adequacy indicates that it is superb [17] to allow for the application of factor analysis.

Factor analysis was used to test the validity of constructs by combining redundant items and deleting inappropriate ones [16]. The Varimax with Kaiser Normalization Rotation method was used here to explain the items’ variance. Items with extraction value through Varimax less than 0.4 will be removed [16].

Cronbach’s α (alpha) was used to test the internal consistency or reliability of the constructs. Factors with Cronbach’s α value greater than 0.7, indicate that internal consistency is guaranteed for the measurement constructs.

The test results show that the reliability of the constructs was significant as Cronbach’s alpha was greater than or equal to 0.70, and all the derived Varimax values of the surveyed items were greater than 0.4. Only I8 “Increased bid win rate through repository of previous bids/tenders” and I12 “Inventory reduction” are removed since most of the answers of these two items are left empty which may be due to their inapplicability in the surveyed SMEs.

The judgment scores for the importance/preference of criteria are based on Likert scale, with the rating from 1–5. The weighted factor values for each sample company are provided in Table 5.

Table 5 Weighted factor value for each sample company
Full size table

The regression results are obtained by running Eviews 6.0. Thus, the above mentioned integrated conceptual model is transformed into Eq. (1) with OLS estimation results shown in Table 6.

Table 6 The OLS estimation results
Full size table
$$ {\text{KMMI }} = \, 0. 6 3 7 4 6 7 { } + \, 0. 2 8 20 60*{\text{EA }} + \, 0. 2 4 9 4 3 9*{\text{KMAP }} + \, 0. 2 3 4 3 3 7*{\text{KMIDM}} $$
(1)

According to the results, R-squared is 0.79 (near to 1.0), which means nearly 79 % of the variance of the KMMI is explained by the involved independent variables, in other words, the derived regression model is efficient. Also F-statistic equals to 74.995 [>F0.05 (3, 61)], which means under the significant level of 0.05, the model is robust. Thus, EA, KMAP and KMIDM could be regarded as the Granger Causality towards KMMI. Moreover, the Coefficients of EA, KMAP and KMIDM are all positive with p values very close to 0, which suggests the EA, KMAP and KMIDM positively and significantly contribute to KMMI. Thus, the three afore-formulated research hypotheses are verified. In other words, when the firms carry out good environmental analysis, KM activity planning and implementation decision making, their KMI will be most probably good. Also as the coefficient reflects the strength of the corresponding independent variable’s contributions to the dependent variable, it indicates that EA (0.28) contributes to KMMI most with KMAP (0.25) less and KMIDM (0.23) least.

Additionally, Table 7 shows the correlation within the KMMI dependent variables. As shown, the Pearson correlation value between KMP and KME is as low as 0.29, which indicates that both KMP and KME explained different aspects of KMI in SMEs and also there are some positive associations between them. The strength between KMP and KMMI (0.803) is equal to that between KME and KMMI (0.803), which means both KMP and KME are similar important in KMI.

Table 7 Correlation between KMP, KME and KMMI
Full size table

6 Conclusion and recommendation

In this study, the established model by using empirical approach can be used as a comprehensive matrix for enterprises to assess or predict their KMI performance. Then from the prediction results, the management can properly adjust their whole strategy of KMI in terms of all the aspects of the three critical factors. Based on the statistical evidence, the higher the efficiency of the three factors of KM reaches, the better the performance and effectiveness of KM can be achieved. According to the relative value of the derived coefficients, EA contributes to KMMI most, with KMAP less and KMIDM least. Thus, during KMI, the enterprises should attach different weight to different part accordingly, thus to best utilize their limited resources. Based on the derived samples, the significance of KMP and KME in KMMI are similar. But with sample size enlarged in the future, the weight balance may be changed, which needs further confirmation. Also more detailed investigation of regressions on KMP and KME will be in our future work.