1 Introduction

E-learning is emerging as the new paradigm of modern education. This study considers e-learning as it applies to the workplace or organizational environment. Due to its flexibility to access, just-in-time delivery and cost-effectiveness, e-learning has been adopted by organizations, especially by small and medium-sized enterprises for support of learning and training in the workplace (Sambrook 2003; Driscoll 1998). With the increasingly significant role of professional skills and expertise in organizational development, practices and studies on workplace e-learning have received increased attention (Zhang and Nunamaker 2003). On the other hand, with the emergence of Web 2.0 technologies, there has been a recent transformation of e-learning from a central controlled education system to an interactive and conversational learning network. Web 2.0 is regarded as a new kind of technologies that are increasingly utilized and integrated with e-learning applications for active knowledge creation and sharing in learning communities (Alexander 2006; Mason and Rennie 2007). Web 2.0 enables learning to take place through participation and engagement in social networks.

However, the recent emergence of Web 2.0 applications leads to a plethora of promises by enthusiastic technologists and pedagogues. There is a lack of overall strategies for the use of Web 2.0 technologies in e-learning applications. As addressed by Rollett et al. (2007), the Web 2.0 e-learning applications have been developed without profound business models. Most applications are performed poorly in helping individuals develop required expertise to improve their performance, or make their social interactions integrated with their learning practice, and ultimately fail to serve the organization’s goal for success in the knowledge economy (Ran and Wang 2008; Roy and Raymond 2008; Tynjälä and Häkkinen 2005; Moon et al. 2005). Though various social tools have been used in workplace e-learning applications, they are less effective than expected in building organization’s intellectual asset or facilitating collaborative learning. In particular, most Web 2.0 e-learning applications have failed to align learning with organizational goals and individual needs in a systemic way. In the meantime, dominance of technology-oriented approaches makes Web 2.0 e-learning development less goal-effective, and accordingly makes them perceived to be poor in quality and design. While using Web 2.0 applications in workplace e-learning, it is crucial to consider how we can shape an education system which meets the needs of learners and the wider needs of the organization or the society (Attwell 2007).

In this study, we use organizational learning theories and pedagogical principles to investigate the requirement and solution for the design and development of Web 2.0 e-learning applications. First, we examine the requirement of integrating organizational, social, and individual perspectives in the development of Web 2.0 e-learning systems. This considers the needs of directing knowledge creation, sharing, and social networking in light of corporate missions, individual needs, and social context. In doing this, a systemic and rational approach is vital in which considerations on organization, pedagogy, and technology must been integrated and balanced. Second, a key performance indicator (KPI) oriented approach is proposed as a solution to fulfill the requirement. A key performance indicator represents a set of measures focusing on the aspects of organizational and individual performance that are critical for the success of the organization. A KPI model shows a clear picture for each individual in the organization what is important and what they need to do.

By integrating Web 2.0 technologies with the KPI-oriented approach, we are able to: 1) set up organizational goals and link the goals with required expertise to be developed by individuals; 2) align the building of knowledge network with the business model by linking learning resources to a set of competences to be developed and a group of people who learn and contribute to the knowledge network through knowledge creation, sharing, and peer evaluation; and 3) make social networking and knowledge sharing more effective by identifying each individual’s work context, expertise, learning need, performance, and contribution. In this way, corporate training, individual learning, together with knowledge sharing and social networking are integrated and facilitated via making sense of organizational, social, and individual requirement and linking them in the learning environment. The mechanism of the approach is explored and elaborated with conceptual frameworks and implement details. A prototype e-learning system has been developed to demonstrate the effectiveness of the approach.

The rest of the paper is organized as follows. Section 2 introduces the background and relevant theories. Section 3 establishes a holistic understanding of workplace e-learning based on the problem addressed, and identifies the requirement of workplace e-learning development. An overview of the KPI-oriented approach for developing a Web 2.0 workplace e-learning system is presented in Section 4. We implement and evaluate the approach using a prototype and relevant experiments in Section 5. Related work is discussed in Section 6 before we conclude this paper in Section 7.

2 Background

2.1 E-learning in the workplace

E-learning is defined as “the use of computer network technology, primarily over or through the Internet, to deliver information and instruction to individuals” (Welsh et al. 2003). E-learning is generally quoted when learning procedure is involved with information and communication technologies. Nowadays e-learning is emerging as a new paradigm of modern education, especially for small and medium-sized enterprises. Workplace learning refers to learning or training undertaken in the workplace (Craig 1996). The field of workplace learning is also known as Training and Development, Human Resource Development, Corporate Training, and Work and Learning (Craig 1996; Piskurich et al. 2000; Driscoll and Carliner 2005; Smith and Sadler-Smith 2006). Workplace learning can be summarized as the means, processes, and activities by which employees learn in the workplace from basic skills to high technology and management practice that are immediately applicable to workers’ jobs, duties, and roles. Increased intensive competition, industrial change and globalization have forced organizations to search for new ways to improve competitive advantage. Continuous innovation becomes a goal in which knowledge is seen as the core resource and learning is viewed as the most important process. Practices and studies on workplace learning have received increased attention due to the increasingly significant role of professional skills and expertise in organization development.

2.2 Relevant theories

E-learning has attracted considerable interests and increased divergence of theoretical perspectives in the past decades. Most e-learning applications are based on the objectivist learning model. The objectivist learning model is based on the stimulus-response theory, where learning is the transfer of knowledge from the instructor to the learner; instructors control the learning process and assess whether knowledge transfer has occurred (Leidner and Jarvenpaa 1995). Recently, constructivist learning model has been adopted in e-learning development (Akhras and Self 2000). The constructivist learning theory views learning as a process in which learners actively construct or build new ideas or concepts based upon current and past knowledge; instructors should let learners participate in meaningful activities so that they can generate their own knowledge (Schunk 1991). More recently, the theory of connectivism states that learners are actively attempting to create meaning through engagement in networks; learning is the process of creating connections and developing a network (Siemens 2005).

Theories specific to workplace learning can be categorized into four groups: adult learning, organization learning, Community of Practice (CoP), and knowledge management. Adult learning theories form the basis for the design of e-learning practice in work environments. Andragogy and self-directed learning are two fundamental parts in adult learning. The implication of adult learning theories in workplace context is that, learners would be motivated once learning objectives have been rationally set and would meet their needs (Knowles et al. 1998). According to self-directed learning theory, learning programs should be designed to give emphasis to self-directed learning to help learners make sense of the workplace and their experiences at work (Merriam 2001).

Organization learning within the domain of organizational theory investigates how an organization continuously and effectively learns and adapts to the environment. Organizational learning concerns both the ways individuals learn in organizational context and the ways in which organizations can be said to learn as organizations (Easterby-Smith et al. 1999). The organizational theory implies that learning should be noted beyond individual level. Its pedagogical focus is on organizational systems, structures, policies, and institutional forms of memory to link individual and organizational learning.

In relation to organizational learning, Community of Practice (CoP) presents another perspective of learning. CoP starts with the assumption that engagement in social practice is the fundamental process by which we learn and become who we are (Wenger 2000). The preliminary analysis unit of CoP is neither the individual nor social institutions but rather the informal “communities of practice” that people form as they pursue shared goals of interests over time.

Knowledge management (KM) represents another discipline in relation to organizational learning and CoP. It refers to a range of approaches and practices used by organizations to identify, create, represent, and distribute knowledge for reuse, awareness, and learning (Nonaka and Takeuchi 1995; Spender 1996). The focus of KM is on the management of knowledge as intellectual asset, the conversion between tacit and explicit knowledge, and the development and cultivation of the channels through which knowledge flows and transfers. Recent research has motivated the integration of knowledge management with e-learning for organizational strategic development. How knowledge management and learning apply to and affect organizations is a complicated, yet important question that requires a variety of conceptual, methodological, and technical approaches (Wang and Yang 2009).

2.3 Web 2.0 and e-learning

Computers and other Information and Communication Technologies (ICTs) have provided people a wide variety of activities and experiences that support learning. Nowadays Internet has become the core platform which places learners at the centre and facilitates informal consumption, creation, communication, and sharing of knowledge. This change has increased the emergence and use of Web 2.0 applications as social software. Web 2.0 refers to an expected second generation of Web technologies that allow people to create, publish, exchange, share, and cooperate on information and knowledge in a new way of communication and collaboration (O’Reilly 2005). With its culture of networking, sharing, and collaboration, Web 2.0 is fundamentally altering people’s relationships and activities with information and knowledge. As a matter of fact, Web 2.0 technology has been widely applied to learning to enhance social communication and knowledge transfer in virtual learning environments. However, the recent emergence of Web 2.0 applications leads to a plethora of promises by enthusiastic technologists and pedagogues. The Web 2.0 applications have been developed without profound business models (Rollett et al. 2007). There is a lack of overall strategies for the use of Web 2.0 technology in e-learning applications. While there is no doubt that the interactive software, electronic resources, and Internet-based communication tools should be considered in training and education initiatives, there are arguments on efficiency and effectiveness (Aczel et al. 2008).

With a further review on the root of the problem, it is found that e-learning development tends to focus on technical issues of design and ignores organizational, social, and pedagogical aspects that are necessary for effective e-learning programs in the workplace. Most applications are lack of pedagogical underpins on the use of e-learning, and fail to understand learning behavior that takes place in the organizational and social context (Tynjälä and Häkkinen 2005; Moon et al. 2005). The dominance of technology-oriented approaches makes e-learning practices less goal-effective, and accordingly makes them perceived to be poor in quality and design.

3 Requirement analysis of Web 2.0 workplace e-learning

Despite the ever increasing practices of Web 2.0 e-learning in the workplace, many of them are performed poorly in motivating employees to learn. Significant gaps exist between corporate interests and learner needs when it comes to e-learning (Servage 2005). To organizations, e-learning is generally designed without meeting organizational vision and mission. To individuals, although knowledge can be learned by participating in e-learning programs and social networks, more often, individuals do not think e-learning is helpful since knowledge learned does not help improve their work performance. Though various social tools have been deployed in e-learning systems, they are not effective in building organization’s intellectual asset or facilitating collaborative learning in the community. As a result, most e-learning applications fail to meet the needs of learners and ultimately fail to serve the organization’s quest for success in the knowledge economy. To solve the problem, we need to establish an in-depth understanding of workplace e-learning, based on which we may investigate what workplace e-learning requires and how workplace e-learning systems should be developed in line with these requirements.

3.1 Understanding workplace e-learning

Although there has been a diversity of theories related to workplace learning, the purposes of workplace learning stay similar, which has been summarized by Boud and Garrick (1999): to improve performance for the benefit of the organization and the learner, and to improve learning as a social investment. In order to have a better understand of workplace learning, we start from fundamental elements of learning environment addressed in (Illeris 2003). The foremost is the learner, which is the chief actor in the learning environment; the other three elements refer to the learning surroundings, including the learning content, the social context, and other learning stakeholders such as parents or society. In workplace settings, the learners are employees in the organization; the learning content is the knowledge and expertise required in the workplace; the social context considers groups and teams in the workplace; and other learning stakeholder is the organization. An effective workplace learning environment should take the four elements into consideration.

Learner

Employees are adult learners with distinctive learning characteristics. They are goal-oriented, practical, autonomous, and self-directed (Knowles et al. 1998). Employees in an organization have distinct job responsibilities which require different types and levels of expertise. Even assigned with an identical task, employees would have different learning needs and expectancies as a result of different educational background, working history, and learning performance. Individuals would learn if learning can satisfy their learning needs based on the personal and organizational development request.

Organization

Different from formal learning in educational institutions, learning in the workplace serves for organizational goals and needs, and focuses on organizational systems (e.g., job system and reward system), structures, policies, and institutional forms of knowledge which link individual and organizational learning. Organizations wish individual learning can be transferred back to job and utilization of new skills to enhance organizational performance. Moreover, workplace learning is a dynamic process which both influences and is influenced by the dynamic changes in organizational structures and practices (Bontis et al. 2002).

Social context

Workplace learning environment is a knowledge society that builds upon community of practice. Learning in the workplace can be understood as social networking between learners, which allow the creation and transfer of knowledge among individuals, groups, and organizations. Knowledge created by individuals are amplified and crystallized during the process of informal communities of social interaction within the organization, and interaction between the organization and its surroundings (Nonaka et al. 1995).

Learning content

Workplace learning content is more contextual and dynamic than that in typical school settings. The learning content is contextual in that knowledge in the workplace is disseminated within an organization and arises from employees’ daily activities and interaction with the working environment (Raelin 1998). Meanwhile, employees and organizations have to think new ideas and adjust learning process in the aim of improving organizational performance. Moreover, to facilitate learning practice, knowledge assets (e.g., learning materials, assessment packages, and discussion messages) accumulated through workplace learning processes should be well organized, updated, and maintained for continuous learning, which may refer to co-creation, mixing, and re-publishing of content in Web 2.0 applications.

3.2 Requirement of workplace e-learning

In relation to the above elements in workplace e-learning, there has been a diversity of studies emphasizing on different elements or aspects. Due to lack of a holistic understanding of workplace e-learning and appropriate conceptual and methodological tools for implementation, e-learning in the workplace remains a fragmented, complex, and challenging area of research and practice (Servage 2005; Collin 2006). Researchers originating in systems theory conceptualize the learning organization from a macro-system perspective. Systems thinking is a holistic approach to analysis that focuses on the way that a system’s constituent parts interrelate and how systems work over time and within the context of larger systems (Jackson 2000). The systems thinking approach contrasts with traditional analysis, which studies systems by breaking them down into separate elements. It is a framework for seeing interrelationships rather than things only, and for seeing the “structures” that underlie complex situations. Using the systems thinking approach, it is found that workplace learning is composed of a highly complex set of variables (e.g., learners, activities, outcomes, organization, and contexts) and their interactions (e.g., the organizational context upon learners’ motivations) (Smith and Sadler-Smith 2006).

Based upon the above analysis, we address the requirement of integrating organizational, individual, and social perspectives in the development of e-learning applications. To fulfill the requirement, we need to start from the key elements in the learning environment and to facilitate the integration of and interaction between the elements, as shown in Fig. 1.

Fig. 1
figure 1

Key elements and their interactions in workplace e-learning

Full size image

Learner

The learning system should be able to support learners’ self-directed learning by guiding their learning process. In particular, the system should be able to help learners determine their learning needs and objectives based on personal and organizational development request. Moreover, the system should provide appropriate learning resource and learning instructions for learners to develop specific workplace skill.

Organization

The learning system should be able to reflect the organization’s learning needs, aligned with organizational mission and vision, job design, and reward system. When the organization rebuilds its organizational structures or redefines job responsibilities, corresponding changes in the learning system should be allowed.

Social context

The system should be able to provide an environment that makes learning take place through participation and engagement in the learning community. In addition to communication and content co-creation and sharing, individuals should be able to make sense of the social learning community by identifying other peers about their expertise for more effective knowledge sharing and social networking.

Learning content

The system should be able to manage (capture, organize, publish, retrieve, and update) learning content or knowledge assets accumulated from daily practices. The system should be able to boost learning motivation and potential by providing learning content relevant to individuals’ learning needs. Meanwhile, the learning content should be continuously renovated with changes in learning reource and changes from learning needs in the organization’s internal and external environment.

4 Proposed workplace e-learning approach

To meet the above requirement of e-learning in the workplace, a key performance indicator (KPI) oriented approach is proposed in this study. Performance measurement is crucial for organization development, therefore is a main drive for employees’ learning activities. A KPI represents a set of measures of organizational and individual performance that are critical for the success of the organization. KPI is a flexible and popular approach for conducting performance measurement in organizations. The mechanism of why and how we use the KPI-oriented approach and integrated it with Web 2.0 technologies in the development of workplace learning systems is elaborated as follows.

4.1 Why KPI-oriented approach

Performance measurement is used by organizations as a procedure to improve performance by setting performance objectives, assessing performance, collecting and analyzing performance data, and utilizing performance results to drive performance development (Baker 1995; Slizyte and Bakanauskiene 2007). KPI is an approach which can be used to assess almost any aspect of work performance, whatever financial or non-financial, depending on individual organization’s design. In a KPI performance measurement system, organizational vision and mission are interpreted into clear defined key performance targets for each business unit; official expertise and capabilities required for each position in the unit are defined according to unit goals and objectives. A KPI model shows a clear picture for each individual in the organization what is important and what they need to do. As a performance measurement approach, KPI has special meaning to workplace learning by considering organizational strategy, structure, and systems (e.g., job system and reward system). KPI bridges the gap between an organization’s mission and vision with and its employees’ targets. In this way, KPI is able to make organizational goals accomplishable and help employees set up rational learning objectives based on their knowledge gap.

Furthermore, KPI can facilitate social networking among individuals by identifying their work context, learning needs, expertise, and performance. A KPI-oriented social learning network implies that detailed competency and performance profiles of learners can be used for exploring collaboration opportunities. The social exchange theory (Constant et al. 1994) suggests that there is a relationship between a person’s effect (satisfaction with a relationship) and his/her commitment to that relationship, that is, his/her willingness to contribute and sharing knowledge. The reasons underlying the motivation and commitment to share knowledge include personal need, reputation, altruism, and reward (Hall 2001). These elements of social exchange intertwine and play a complex role in communities. The KPI model can facilitate and direct collaboration and interaction among community members based on a common purpose, i.e., competency performance development in the work context. This common purpose can stimulate participation to learn, share knowledge, and support each other.

In this way, KPI is used as a systemic scheme to drive learning activities, organize knowledge assets, and facilitate social networking in line with a set of competences to be developed by a group of people who learn and contribute to the knowledge network (Ran et al. 2008, Ran and Wang 2008). Accordingly, KPI is used in this study for integrating organizational, social, and individual perspective in the development of workplace e-learning systems.

4.2 KPI model

A model of key performance indicator (KPI) is designed based on an organization’s mission and vision, organizational structure, and job system. It consists of three levels including organizational level, business unit level, and position level. Key performance indicators on the organizational level are defined according to business goals and strategies of the organization. Derived from the organizational KPIs, the unit KPIs for each unit can be specified. Based on the unit KPIs, the KPIs for each job position within the unit are defined; the KPIs for a position are further broken down into a set of items that measure the performance of relevant capabilities required for the position. For performance measurement to be effective, the measures or indicators themselves must be accepted, understood, and owned by employees as well as their managers. Therefore, the building of a KPI system needs cohesion and integration of different strategies as well as tight cooperation among managers and employees from different unit and at different level of position (Baker 1995). KPIs of a position in one unit can be reused in other unit for a similar position; performance items of one position can be reused by other position where similar capabilities are required. In this study, due to the space limitation, we focus on the KPIs at the position level which have more close relationships with learning in the workplace.

The KPI at the position level consists of three components: KPI item, rating criterion, and KPI value. KPI items are a set of capabilities required for a job position. For example, oral and written communication skills might be two KPI items defined for a sales job position. For each KPI item, rating criterion is set up to assess related performance indicators at different proficiency levels. The proficiency level achieved by an employee is called a KPI value for a certain KPI item. An employee’s performance measurement result is a set of KPI values of his/her job position. Tests or quizzes can be used to assess how an employee achieves a certain KPI item. For impartiality and objectivity reasons, most organizations use 360 degree feedback to assess employees’ performance. It means that the employee’s performance could be assessed by the employee him/herself, his/her supervisor, his/her subordinate, or his/her peers, in addition to taking standard tests. Each appraiser gives the employee a set of KPI values, and each appraisal is given a certain weight. As a result, a set of KPI values can be calculated to evaluate the employee’s overall work performance. The KPI model at the position level is illustrated in Table 1.

Table 1 A KPI model at the position level
Full size table

4.3 KPI-oriented Web 2.0 learning environment

KPI helps employees make sense of their work context, required expertise, and social community; and accordingly help them set up rational learning objectives, access relevant knowledge artifacts, and get engaged in social networks. The mechanism how KPI works as a drive for social and individual e-learning in the workplace is depicted in Fig. 2.

Fig. 2
figure 2

KPI-oriented workplace e-learning architecture

Full size image

In the system, learners are assigned to one or more positions. Learners are able to access, create, publish, and evaluate learning resources. They are also able to discuss and communicate through the discussion forum and communication tools. Based on the KPI model, a set of capabilities are identified for each position. Moreover, each capability is linked to a number of learning objects, which are categorized into different types such as articles, books, web pages, and video files. Learning objects are created by the experts based on remixing or syndication of learning materials contributed by learners or directly uploaded by the training manager. Discussion messages can be processed as a type of special learning material. Each capability is also linked to an assessment package, which includes assessment methods (e.g., tests, quizzes, and peer evaluation) and rating criteria. Based on the assessment result, each employee is provided a KPI identification, i.e., a set of KPI values that represent his/her expertise and proficiency level with a position, stored as a part of the learner profile.

4.3.1 Ontology-based KPI model

To conceptualize the KPI model and manage relevant resources such as learning objects and assessment packages ontology technology is used as a formal approach for knowledge representation. Ontology is defined as a formal representation of a set of concepts within a domain and the relationships between those concepts. In the context of computer and information sciences, ontology is “the specification of a conceptualization” (Gruber 1995) and can be used to model, represent, and share knowledge. According to the expertise or capabilities identified in the KPI model, ontology is used to define all the capabilities with their relationships (e.g., prerequisite, composition, and relevance). A part of the capability ontology is outlined in Fig. 3, which describes the expertise required for the position of “Junior Tester” in a software development company. In this example, if a learner intends to acquire the capability of “Testing Specific Skills”, he/she needs to acquire two composite capabilities, “Bug Reporting” and “Test Execution”; before acquiring the two capabilities, he/she must acquire the prerequisite capability “Testing Basic Concepts and Definitions” which involves three components “Testing-related Terminology”, “Theoretical Foundation”, and “Relationships of testing to Other Activities”.

Fig. 3
figure 3

An example of capability ontology

Full size image

In addition to identifying relevant capabilities required for a specific position, the capability ontology can be used to reason out the learning path based on the relationships between the capabilities. Relevant learning instructions are specified for support of reasoning. The reasoning of an individual learning process should also consider the learner’s learning needs according to his/her knowledge gap, preference, or interest (Carchiolo et al. 2007). For example, if the learner has already acquired the expertise of “Theoretical Foundation”, he/she may skip this component in his/her learning process. The knowledge gap can be identified via appropriate assessment methods including the tests provided by the system, the learner’s self-estimation, and peer assessment.

4.3.2 KPI-oriented social learning network

In our system, Web 2.0 technologies (e.g., discussion forum and tools for communication, publishing, and co-editing) are used to support learners to create, publish, exchange, share, and collaborate in the learning community. To improve the effectiveness of Web 2.0 in workplace learning, KPI is used to reflect personal need, reputation, altruism, and reward, which underlie the motivation and commitment to co-create and share knowledge. First, learners are able to create and publish learning materials, as well as add relevant KPI annotation to the materials. By using KPI as the index, learners are able to share, access, and aggregate knowledge assets in a more systemic way by linking the knowledge assets in line with business model or work context. Second, learners can become more goal-oriented in the learning process by using KPI to identify their learning needs in capability performance development. Driven by a clear motivation and common purpose, the system encourages voluntary and active participation. Third, learners are able to know about and interact with each other based on their work context and expertise represented in their KPI profiles. Based on KPI profiles of other peers, learners are able to find peers of similar interests or background, or locate experts with high reputation of particular expertise. Four, learners’ contribution of knowledge to the knowledge network is well recognized and evaluated by peers. The recognition and evaluation may improve participation and knowledge sharing. Five, discussion and social communication in the learning community can be directed by linking their topic to relevant competency or expertise, which makes the social networking more effective and goal-oriented. Six, peer assessment on individual’s capability performance is conducted as a part of collaborative learning, which encourages engagement and interaction in the social learning network. Using the KPI-oriented approach, self-directed and socially constructed learning activities in the workplace are effectively directed and facilitated via the integration of organizational interests, individual needs, and social context, as shown in Fig. 4.

Fig. 4
figure 4

KPI-supported social and individual e-learning in the workplace

Full size image

5 System development

To demonstrate the effectiveness of the KPI-oriented approach, a prototype system has been developed for the company Peanut, a medium-sized software company. The organization structure of the company and its job positions are outlined in Fig. 5. In this prototype, we focus on e-learning development in the Testing unit. Testing is a technique for evaluating software product quality by identifying defects and problems; it is an important and mandatory part of software development. Based on the company’s policy, the Testing unit of this company has defined “Bug Found” and “Bug Returned” KPI as the standards for measuring productivity and quality of software development. The KPI model of this prototype is constructed based on the company’s policy as well as IEEE standards of software test introduced in (Bertolino 2001). A part of capabilities of software testing are presented in Fig. 3.

Fig. 5
figure 5

An organizational structure with positions

Full size image

5.1 Implementation

The prototype is built using Java programming tools. SQL Server is used for database implementation. Moreover, Hibernate technique is used for developing persistent java objects, and Struts technique is adopted to separate the programming codes into model, view, and controller. Based on the KPI model, Protégé is used to construct the ontology in machine language. Protégé is a free open-source Java tool developed at Stanford University for editing of ontology and knowledge framework; it provides a powerful environment and plug-in API for developing customized knowledge-based applications (Noy et al. 2000). In this prototype, ontologies that model capabilities of different positions together with the relationships between the capabilities are specified. Moreover, Protégé API is used to develop a graphical interface of the capability ontology. To visualize the relationships between the capabilities, a semi-open-source component — JGraph is used for display and auto layout of the capability ontology in a graph. With the help of XML parser, the ontology specified in Protégé is linked to the capability items stored in the database. Moreover, each capability is associated with relevant resources including learning objects, assessment packages, and discussion items stored in the database.

In the prototype, two platforms are provided, one for individual learner, another for training manager and expert. The user interfaces enable different roles of user to access the learning system via Internet. The architecture of the system is depicted in Fig. 2. To synchronize the operations between the two platforms, JDBC technique is used. Moreover, JESS engine is used for reasoning the learning path for individuals to facilitate self-directed learning. The inference is made based on the relationships between capability items, the learner’s knowledge gap, and relevant learning instructions that have been transformed into JESS rules.

A set of screenshots are presented in Fig. 6. The capability ontology is visualized in graphs for easy communication of the learning context. The learner is able to locate learning resources related to a specific capability by clicking the capability in the capability graph. The learner’s performance can be evaluated by self-estimation or peer assessment, in addition to taking standard tests. The performance results can be combined and transformed into KPI values. Moreover, the KPI value is represented with a color in the capability graph, which indicates his/her overall proficiency level of the capability. Moreover, learners are able to share and evaluate learning objects as well as participate in discussion and communication. During discussion and communication, learners are able to locate peers or experts according to their background, expertise, or their contribution to the learning community.

Fig. 6
figure 6

Screenshots of the proposed e-learning prototype

Full size image

5.2 Evaluation

To evaluate the effectiveness of the prototype, we invited a number of employees who are currently working with or worked before with the Testing unit of the company to participate in the experiment. Two parallel prototypes were used for evaluation: the prototype system developed by using the KPI-oriented approach, and another one developed based on a traditional approach without KPI support. The participants were divided into two groups (KPI Group and Reference Group) using two different e-learning prototypes respectively.

The evaluation was conducted based on Donald Kirkpatrick’s model which is proposed for evaluating the effectiveness of a training program (Kirkpatrick and Kirkpatrick 2006). The evaluation includes four levels, reaction (how participants react to the learning system), learning (knowledge learning or skill development by using the application), behavior (transfer of learning into change of behavior by using the system), and result (organizational and individual outcome as a result of the training program). The methods used for evaluation consists of a mixture of quantitative and qualitative data gathering approaches. The data collected includes participants’ learning outcome reflected in the pretest and post test, and participants’ perception on the system obtained through a questionnaire survey and interview.

In the experiment, 28 employees were enrolled and assigned to the two groups, 14 for each group. Each participant was assigned to a role as learner, expert, or manager. The data collection process includes several steps. At the beginning, participants were asked to finish the pretest. After using the system for a month, participants were asked to finish the post test and questionnaire for their evaluation of the system. The score of the pretest and post test was measured as the number of questions which were answered correctly. The learners’ evaluation of the system through questionnaires was measured in a Likert scale (from 1 strongly disagree to 7 strongly agree). The evaluation result with the test scores are presented in Table 2.

Table 2 Evaluation of the system
Full size table

In relation to the reaction level, the KPI-oriented system is found to be more effective in terms of learning requirement and functional support for workplace learning. In relation to the learning level, the KPI-oriented system is perceived to help learners obtain more knowledge than the non-KPI system. In relation to the behavior level, the KPI group found the system more helpful to enable learners to put learning into practice, and encourage learners to participate in social networking. In relation to the result level, the KPI group perceived that the system provide more positive outputs to both individuals and organization. Moreover, the result of the pretest and post test scores shows post test score mean is higher than pretest score mean of both KPI Group and Reference Group, while there is no significant difference in post test score between KPI Group and Reference Group. The result is acceptable, since other factors of the learners (e.g., learning capability and efforts) and learning environment (e.g., Internet speed) may affect the result, in addition to the learning system.

After the above evaluation, the two groups exchanged the use of the prototype for 2 weeks, i.e., the participants who were asked to use the e-learning prototype with KPI support at the beginning were asked to use the non-KPI system at this stage, and vice versa. The purpose of this arrangement is to provide participants different learning experiences with two prototypes for further comparison. Finally, an interview was conducted to collect the participants’ comments on the learning systems. It is found that 80% of the participants feel the KPI-oriented prototype more effective in facilitating individual and social learning and helpful for utilizing knowledge learned into work practice. 90% of the participants preferred the KPI-oriented prototype, which provides more effective learning experience to improve employees’ work expertise. However, the participants were not quite satisfied with the usability and interface of the systems. Also, they felt the time of the experiment was not enough for them to get familiar with the functions, in particular in using the KPI-oriented system. Based on these initial findings, we will make necessary modification and improvement of the system for further experiment and evaluation.

6 Related work

KPIs are used to help an organization define and measure progress toward organizational goals. KPIs are typically tied to an organization’s strategy. In education, key performance and quality indicators have been mainly used to guide teaching and learning development to make institutional practice aligned with objectives and strategic plans of high education (Day and Bobeva 2006; Tapinos et al. 2005). KPIs together with other performance measurement approaches have also been used to set up frameworks for assessment of impact of knowledge management (KM) on organizational performance in education (Rodrigues and Pai 2005; Taylor 2001), business (Carrillo et al. 2003), and other settings. The frameworks provide a solid basis for developing KM strategies that are not only coherent but also consistent with the overall objectives of an organization.

The other related work includes competency-based learning. In recent years, competency-based learning has become a widely used approach by organizations, where learning is driven by development of specific competencies for dealing with needs and challenges in competitive environments (Lucia and Lepsinger 1999). The competency method seeks to identify the ideal combination of skills, knowledge, and attitudes, and is noted as a tool to structure and facilitate communication between education and the labour market (Green 1999). Evidence suggests increased usage of competency method by organizations to drive workplace learning initiatives in the USA and more recently in the UK (Garavan and McGuire 2001). The increased usage of competencies is also reflected in the burgeoning academic literatures. Korossy (1997) use the competency performance approach to extend the theory of knowledge space. Based on Korossy’s approach, Ley and Albert (2003) investigate a method for creating and testing tailored competency management models for actual work and tasks in organizations or institutions.

Though competency and performance are not new concepts, the literature in the concept and main issues is fragmentary due to the research interests distributed in multiple disciplines. There is a lack of a widely acceptable understanding of competency and performance, and a systemic approach for connecting competency and performance (Sicilia and Naeve 2007; Parmenter 2007). The implementation of competences and performances oriented learning systems for improved effectiveness and efficiency poses a number of significant challenges. More substantial efforts are required to investigate the main requirements and develop concrete solutions for competence performance oriented learning.

Moreover, though some studies have addressed the issue of using competency as the structure to construct organization or group memory, they have ignored the role of competence performance management in building social learning networks. This study overcomes the limitation by using competency and performance profiles of people for exploring collaboration opportunities in social learning networks. In particular, we have presented a systemic approach based on a KPI model to: a) link learning resources to a set of competences to be developed by a group of people who learn together and contribute to the knowledge network; and b) enhance social interaction and knowledge sharing in the learning community by identifying each individual’s work context, expertise, learning need, performance, and contribution.

7 Conclusion

This study addresses the problem of Web 2.0 e-learning development in the workplace. The main questions of the study are what workplace e-learning requires and how workplace e-learning systems can be developed in line with these requirements. We investigate the problem from a variety of aspects, and address that workplace e-learning systems should be able to integrate organizational, individual, and social perspectives. The development of Web 2.0 workplace e-learning environment should consider the alignment of individual learning needs, organizational objectives, and social networking. To achieve this, a KPI-oriented approach is proposed and integrated with Web 2.0 technologies to enhance the effectiveness of self-directed and socially constructed learning practice in the workplace. Key performance indicators are set up to assist organizations to clarify training objectives and organizing socially constructed knowledge assets, help individuals make sense of their work context and required expertise, and accordingly help learners set up rational learning objectives and access relevant knowledge artifacts. Meanwhile, KPIs help individuals communicate in relevant work context, and make their knowledge sharing and social networking more effective and consistent with the business model. Using this approach, a prototype system has been developed to demonstrate the effectiveness of the approach.

Web 2.0 technologies have offered a broad range of new possibilities for learning in educational institutions and organizations. Web 2.0 allows people to create, publish, exchange, share, and cooperate on information and knowledge in a new way of communication and collaboration. However, it is hard to transmit the essential attributes of Web 2.0, such as voluntariness, trust, and self-directness, into many of existing applications (Rollett et al. 2007). We are facing the challenges of maintaining these essentials that really defines Web 2.0. The KPI-oriented approach presented in this study attempts to explore a way to pursue the essential of Web 2.0 in workplace e-learning. First, the KPI model makes the Web 2.0 learning environment more goal-oriented, and therefore makes the participants more voluntarily engaged in learning by a common purpose to improve work performance. Second, the KPI profile of each individual recognizes the expertise and reputation of the participants, which improves the trust in Web 2.0 learning. Third, the knowledge contributed by peers is harnessed and well organized based on the KPI model, which enhances further aggregation, sharing, and retrieval of collective intelligence in the knowledge pool. This avoids a common problem of information overload in Web 2.0, and improves the self-directness of learning in Web 2.0.

A limitation of this work is that the learning development has focused on short term learning needs related to job performance. In workplace settings, learning should go beyond the aim to enhance job performance in the short term, and also articulate the need to enhance personal and career development in the long term (Smith and Sadler-Smith 2006). Investigations towards the long term needs of workplace learning will be investigated in the future work.