Keywords

1 Introduction – Competency Management

In this paper we explore the literature on competency management frameworks in Canada and elaborate on how competency functionality is being approached for the Learning and Performance Support (LPSS) program at the National Research Council of Canada (NRC). The paper is based on the results of literature and market reviews as well as user feedback through surveys and usability studies.

A competency is defined as a set of characteristics of an individual that are observable, measurable and predictive of superior performance in a given role. They define how people get their job done [1]. Others have suggested that competencies include a combination of observable and measurable knowledge, skills, abilities and personal attributes that contribute to enhanced employee performance and ultimately result in organizational success [2].

Competencies act like a bridge to connect job requirements with the desired skill set (of an individual) through appropriate training and development [3]. Competencies balance both the theoretical and practical experiences and can be developed through knowledge management within organizations by human resource development activities such as: competence recognition (through interviews or other means); acquiring competence (through e-learning or other training/learning methods); assessing competence (through monitoring within e-learning or other training/learning methods); and optional utilization of competence (through coaching/mentoring) [3]. Competencies are developed into competency frameworks within organizations or are based on existing national competency frameworks. The difficulty with this approach is that competency frameworks tend to be specific to particular organizations and countries.

Competency-based education is a topic frequently mentioned in relation to competency management, prior learning experience assessment and the informal learning paradigm: “Competency-based education” (CBE) refers to a method of organizing learning based on the achievement of competency, rather than on a defined set time. The two basic methods and forms of CBE are: (1) course-based where students work on their own time through a set of competencies organized into a course; and (2) direct assessment where students demonstrate the achievement of predefined competencies based on prior learning or experience” [4]. The 2016 Gartner report on business trends impacting higher education claims that CBE is currently at the peak of inflated expectations on the digital learning technologies hype curve. The other findings by Gartner related to the status of CBE in North America include the following observations [4]:

  • CBE is becoming increasingly popular among institutions that are interested in increasing student outcomes, such as mastery of the subject matter, graduation rates and preparation for the world of work.

  • Interest in competency-based education is very strong in the U.S. and is growing globally, as it delivers scalable, any-pace education and connects with competence and mastery initiatives in other countries.

  • Currently, most of the activity around CBE is at the planning level since only a relatively small number of institutions have actually implemented it.

  • There is a close logical relationship between CBE and outcomes - or performance-based funding, a funding formula to reward a defined policy outcome. Performance-based funding should not be confused with the concept of competency-based education (an education model built on mastery). Most of the experience with CBE has been at institutions with a nontraditional student body. As CBE is implemented at institutions with a more conventional 18- to 24-year-old student population or with a less-disciplined student body, new challenges are likely to emerge.

In addition to the United States, the area of competency management, frameworks, competency-based education and competency matching is a very active and extensively funded research area in the EU. There are several EU funded projects that develop technologies for competency management and competency based education such as TENCompetence (Building the European Network for Lifelong Competence Development), PROLIX (Process-oriented Learning and Information eXchange), TRACE (TRAnsparent Competence in Europe) and WATCHME (Workplace-based e-assessment technology for competency-based higher multi-professional education). EU projects on competency management and competency based education develop technologies focused on lifelong competence development, personal competency management, authoring tools to help users organize and coordinate learning environments, activities, competencies and learning paths, as well as ePortfolios to help lifelong learners to reflect on the competences and competence profiles they have acquired [5,6,7,8,9,10].

Regrettably, the area of competency management and competency-based education does not get similar attention and research funding from the Canadian government and industry, with the exception of healthcare industry. The next section of our paper provides an overview of competency frameworks, competency management and competency-based education in Canada.

2 Canadian Competency Frameworks

Competency frameworks are most commonly used within the medical profession including medical education, professional training and accreditation –and in the public service. Details of the medical and professional competency frameworks in Canada are addressed in the sections below.

2.1 Medical Competency Frameworks

In 1986 a physician’s strike in Ontario raised concerns about the public perception of doctors and their competence, and the importance in ensuring that medical education fits societal needs [11]. This resulted in the 1996 introduction of the Canadian Medical Education Directives (CanMEDS) Framework by the Royal College of Physicians and Surgeons of Canada (the Royal College) that implemented a competency framework for Canada’s postgraduate training programs. The CanMEDS framework has since been modified for use in other countries [12]. CanMEDS currently outlines seven roles: medical expert, communicator, collaborator, manager, health advocate, scholar and professional. Some of these roles are easy to teach and assess while others are more challenging. There is also a “tension between the need to meet accreditation standards and the development of meaningful assessment tools” [11].

Competency-based medical education (CBME) has its share of issues. Competencies are intrinsically contextual and culturally specific – that “individual views of competence communication, collaboration, professionalism and advocacy will be historically contingent, situational, changeable, and inevitably different from those from other backgrounds and cultures.” There is also a need for medical training to align with community wants and needs [11].

CBME includes measuring competencies within medical simulations. For simulations involving interpersonal and communication skills, psychometric tools are used – the encounter is recorded and rated on a 7 point scale. Checklists are used for procedural skills, pre and post tests are used to assess knowledge and observational and non-observational tools are used to measure competencies. Observational tools include global rating scales, checklists (checking “respect for tissue, efficiency of time and motion, instrument handling, knowledge of instruments, use of assistants, flow of operation, forward planning and knowledge of specific procedural steps”). Non-observational tools include computer based measurements such as scores “generated based on errors, economy of movement, and time to complete the task” [13].

Issues with simulations and competencies “include the challenges and costs of obtaining and using appropriate simulation software and hardware, concerns about validation of simulations as an educational tool, and difficulty in creating normative standards for grading performance.” To summarize, potential barriers to simulation and competencies include: access, cost, instructor availability, educational validity, assessment and outcome measurements [13].

Another implementation issue is that when competencies are considered in medical simulations sometimes important competencies are neglected and there is a need to include a greater degree of competency assessment. For example, when competencies are assessed during medical simulations of orotracheal intubation, force is usually not considered. In addition, traditional measures of observation and checklists are not always the best way to assess intubation competencies. In these cases, Garcia and colleagues reviewed additional assessment measures of force applied, number of attempts, time to intubation and hand position using checklists, and additional measures such as “sensors, transducers, and special pressure sensitive films secured to the laryngoscope” [14].

2.2 Professional Competency Frameworks

The bulk of the Canadian literature on competencies deals with medical education; few other professions and sectors in Canada employ competency based education and competency frameworks. For example, the competency framework has been developed for public service sectors in Canada based on the existing job classification system in place within the Canadian public service.

Bonder [15] outlines the implementation of a competency framework in the Canadian Federal Public Service. The framework was based on the existing job classification system in place within the Canadian public service. While the article indicates numerous core competencies for all public sector employees (making up more than 140 competency profiles), only “client focus” competency is listed and detailed within the article. The competency framework is available through the CBM (Competency Based Management) Web Suite 2 which is accessible to all employees and provides information on CBM and on various tools such as the national competency dictionary, competency profiles, competency self-assessment questionnaires for employees, assistance on developing learning plans and the “National Learning Inventory” which links all departmental learning and development activities to competencies. The site also provides online tools and information for managers to assist them in applying competencies to staffing and other HR processes.

In addition to public service workers, competency models are currently in place for professional training programs for new lawyers. Paquette [16] writes on the design of competency software for a professional law training program. The final system presents the users with a list of competencies from which the users then select their own competencies and performance level. Next the system provides the users with a global summary of their competency levels in order to show gaps and identify strengths and weaknesses. Finally the system presents the users with a plan of action by providing them with resources.

2.3 Competency-Based Education and Prior Learning Assessment

Competency-based education (CBE) and prior learning assessment (PLA) are both important strategies and companion tools for post-secondary education programs. CBE focuses on what students should learn rather that where or when the learning occurs, while PLA presents strategies for evaluating informal learning [17]. The Canadian Forces (CF) is one of the few large Canadian organizations with a Prior Learning Assessment and Recognition (PLAR) program. Other countries with PLAR in the military include Australia, New Zealand and South Africa. The UK has a national qualification framework but PLAR is underutilized. PLAR is not used in the United States military.

PLAR implementation in other Canadian workplaces, besides the Canadian military, is limited. Not recognizing prior and informal learning does have significant consequences for the Canadian economy. Simpson and Vollick’s report [18] on a 2001 study from the Conference Board of Canada concluded that if the experiential learning of Canadians was fully recognized, between $4.1 and $5.9 billion in income could be generated. This same study found that Canadian post-secondary education has established barriers that prevent the recognition of credentials other than those gained through formal education.

2.4 Implementation Challenges

Our literature review showed that there are significant challenges with the implementation of competency-based education and management in Canada, including lack of a national competency classification system, non-uniform implementation of accepted medical competency frameworks, such as CanMEDS, and the fact that competency assessment methods vary widely and could be unreliable. Another challenge in broad implementation of competency frameworks is that they are not universal and generally not applicable outside of specific countries, provinces and/or professions and need constant revisions. In addition, researchers observed some significant user resistance to implementation of competency frameworks in industry and in medical training [11, 12, 19, 20].

Competency management practices are closely intertwined with learning management. The functionality of both competency and learning management systems are frequently included in a single software application, or two software systems might be integrated. The tools can be used to schedule training sessions, keep track of attendees, and create reports on who completed each session [21]. New learning management systems (LMS) incorporate social media, training, certification and mentoring systems. Similarly, highly evolved learning and performance ecosystems are made up of a combination of talent management, performance management and knowledge management; they also provide access to experts, social networking and collaboration, and structured learning [22, 23].

3 Competency Implementation Within LPSS

The National Research Council of Canada (NRC)’s Learning and Performance Support (LPSS) program implements adaptive and personalization strategies and develops software components for learning, training, performance support and enterprise workforce optimization. These technologies are designed to benefit NRC clients and their users by: facilitating lifelong learning, reducing learning and training costs, reducing demands on physical infrastructure, enabling streamlined and rapid skill development, reducing time to competency, supporting informal, personal and personalized learning, increasing learner engagement, optimizing sustainable workforces, and increasing operational performance and productivity [24].

The LPSS program is developing a learning and performance support suite of tools that will maximize a users’ potential, by enabling them to manage and achieve competencies by matching their skills and expertise to stated customer or employer needs. The tools will help to understand training needs by automatically collecting and analyzing learning and performance reports to show gaps between existing competencies and learner or employer needs. The goal is to improve efficiency of training by using learning records and performance analytics to recommend the most useful learning services and resources specific to workplace environments and competency profiles. LPSS technologies will aid in lowering the cost of learning by enabling access to a wide range of learning services and resources from multiple providers from within the context of relevant multiple workplace environments and productivity tools. LPSS tools originated as a web-based prototype open to the public at lpss.me that offers personal rather than personalized learning [24]. The prototype lpss.me was active from Fall 2014 to Fall 2016. Currently it is being redesigned as a set of tools to address the needs of NRC clients.

While the competency management functionality developed within the lpss.me platform is based on a set of preloaded competencies and competency profiles, the user can also define their own competencies and what they want to achieve such as “carpentry” or “creative writing”. The user can choose a competency from the preloaded set or add user-defined competencies and receive recommendations for a set of learning resources for a chosen competency (or a set of competencies). The user also has the option to self-assess the level of skills acquired. A screenshot of the LPSS user interface for the competency management functionality is presented in Fig. 1.

Fig. 1.
figure 1

User interface for personal competency management in LPSS

Full size image

Feedback from the users on the lpss.me prototype is being incorporated in the NRC’s suite of learning and performance technologies and its showcase platform Techquity (see Fig. 2). The Techquity showcase platform provides the medium by which the benefits of individual LPSS research tools can be demonstrated to clients and allows for tool integration to create broader service offerings. The showcase site itself, named “Techquity” combines “technology” and “equity” and serves as a landing page where all LPSS tools (showcases) can be accessed. Some showcases may demonstrate single technologies, while others demonstrate how these technologies could work together.

Fig. 2.
figure 2

The NRC Techquity showcase platform

Full size image

4 User Feedback on LPSS Competency Functionality

User feedback on functionalities within LPSS, including competency management and competency and learning resource matching, was elicited via an online survey of lpss.me users and through users’ responses to questions regarding LPSS functionality in the course of the remote usability testing of lpss.me conducted in February of 2016.

4.1 Online Survey

An invitation to participate in the online survey was sent to 299 users who were registered with the lpss.me personal learning platform. Between November 2014 and October 2016, 57 users responded to the survey, a response rate of 19%. The majority of respondents were male, from 40 to 69 years old, involved in online learning and highly educated. As a result, the findings from the survey cannot be seen as representative of the general population, but are sample viewpoints from power users within the online learning community.

A total of 24 questions were asked, ranging from demographics questions (e.g. age, gender, education, and familiarity with online learning, etc.) to more specific questions about the LPSS system, including “Which of the LPSS pages or features have you looked at so far?” [24, 25]. LPSS questions that elicited responses concerning user competencies within the system included the open ended text questions: “What would you like a learning and performance management system (like LPSS) to do for you? How would you like to use it?”

4.2 User Feedback on Competency Functionality

The competency management functionality within LPSS attracted significant interest, with roughly 80% of the respondents indicating familiarity with the “My competency profile” functionality and 96% indicating familiarity with the “Browse competency” functionality.

When asked to give feedback on LPSS platform features, the participants provided feedback and suggestions for improving competency features, including:

  • Making competency profiles less general and more specific, tied to a particular task.

  • Making a better connection between competencies and resource recommendations.

  • Adding social connections for people with similar competencies providing a way to network around competencies and learn from what others interested in similar competencies are studying.

  • Providing the ability to browse competencies with the users profile associated, to optimize the social network effect.

Examples of user responses related to the LPSS competency functionality are listed below:

“…If I am correct, LPSS is about lifelong learning through the perspective of your competences. The idea and its features are interesting, but you can see the realization is too much inspired by research and less through how people actually want to experience stuff online. For example, I would suggest that at the start of the LPSS, LPSS already would know what kind of competences I might be looking for based on LinkedIn or something. It should give suggestions. Also, learning - for me at least - should be much more visual. Probably this is on the roadmap, but I would like to stress the importance of visual stuff, also to make competences more readable…”

“…Being linked into a group of other users who share common interests/competencies would help…”

“I like the idea of having competencies listed that I want to achieve as part of a personal learning and development plan. Then to point to resources that will help to achieve competencies. I would like more detail in the competencies section. I could see naming a competency area like Professionalism… and then have several more detailed competency statements… and then have some way to measure progress toward achieving each competency statement”.

4.3 Feedback on the LPSS System and the User Interface

The survey respondents came up with some ideas on what they would like a learning and performance management system (like LPSS) to do for them, how they would like to use the system, and how the system should be designed to help them learn better:

“As a learner I like it to make suggestions for relevant resources within the competencies, I would like to develop. Based on my already achieved competencies it would also be interesting to see, what other people with a similar profile would recommend to study. Basically it has to (be) intuitive in use”.

“I would like to find a way to connect these competencies with an online digital identity tool for employers/conference organizers to scrape so I don’t need a dedicated website”.

“Network me with other learners interested in similar competencies”.

Users’ feedback on the user interface design and suggestions on modifications that might help the usability of the system included comments such as:

“I was confused when trying to create my own competencies. It would be useful to have a support tool that guides you as you create competencies. Also, the format of competencies that I have doesn’t seem to fit into the input boxes”.

“Navigation is clear, clean look. Features that are functioning work well. Perhaps instead of a tick mark to indicate that a competency had been chosen it could change colour”.

“Needs a better personal dashboard, with multiple competencies it will become a long page. When getting started the concept of resources is not clear. Love the concept”.

“…Create an online sharing and learning environment that is transparent, accessible, personal, friendly, inviting…”

4.4 Usability Evaluation

In addition to the LPSS user survey, usability evaluation of the LPSS platform was conducted in February 2016 using remote usability software (Loop11). The users that signed on lpss.me within the period from December 2014 to December 2015 were invited via email to participate in the usability testing. LPSS users are all volunteers who were invited to sign up to use lpss.me as a website in development. An invitation email to participate in the study was sent to a total of 150 users, with the reminder email sent a week later. The remote usability study for lpss.me has involved users performing mandatory tasks within lpss.me and answering task-related and general questions on lpss.me use, as well as several demographic questions. The study contained some regular lpss.me tasks that users were asked to complete, for example: to add a topic to the competency profile. The questions that users were asked to respond to were either within a text box or Likert scale questions with radio buttons.

During the study period a total of 16 participants clicked on the study link in the invitation email; out of these 16 people, three users completed the study (known here as P1, P2 and P3). Two users (P4 and P5) provided some responses to questions but did not complete the study. The task analysis was done for the completed studies only, and responses to questions analysis included both completed and partial user responses.

At the end of the usability study participants were asked a series of general questions about the system, including would they use it, which features they liked the most, and which features are useful for learning or for doing their job. Two of the users who were able to do all the tasks gave some useful responses that point to the potential of LPSS as well as the need to push the system further and add greater functionality.

For example, in response to a question about the most liked features of the system user P3 commented: “Competencies. Easy to add” and added that “… I would like to have the opportunity to be able to showcase my competencies and abilities”; that the tool is useful for learning and job and that “…The tool is great to force you to reflect on your career”.

Users were also asked which aspects of LPSS they found less useful, if they had any suggestions for new features, which additional features they would like to see, and if they had any issues with the lpss.me website. Participant 1’s response was that it would be useful to have: “Link to my publications or similar demonstrations of competencies”.

While providing important insights on the user’s experience with lpss.me, the usability study was limited in that only a small percentage of LPSS users looked at the study, and an even smaller percentage finished the entire study which was comprised of tasks and survey questions. This is not surprising considering that this type of study is more time consuming than a straightforward survey (about 40 min for the study vs less than 10 min for the survey) and users were not compensated for their efforts.

5 Design Considerations for Future LPSS Development

Based on the results of the literature review and the user feedback on desirable LPSS functionality, future LPSS development will focus on greater functionality within LPSS, greater personalization for the user and a better user interface created through the use of multimedia. The development of competency functionality will focus on providing the ability to showcase the competencies, social networking for people with similar competencies and including the assessment of both formal and informal learning.

Competency assessment in most cases is a complex process that involves many actors and software; sometimes it might be too complex to automate. LPSS can provide functionality to access the learning records and activities related to a particular competency for actors such as experts, instructors, and for learning and performance analytics. In addition, the LPSS research team plans to explore ways to connect competency assessment with NOC 2016 (Canadian National Occupational Classification) and with user resumes, similar to what is currently done with ESCO classification in EU [26]. LPSS technology components currently in development for competency management and competency-based training include competency key phrase extraction, matching courses to competencies, characterizing informal learning and recommender technologies to recommend learning resources based on existing and desired competencies.

6 Conclusions

This paper reports on the findings of the literature review on the current trends within competency-based education, training and management, as well as the challenges with competency management frameworks in Canada. Results from user surveys and usability study for the lpss.me learning and performance management platform related to the competency management functionality were presented. We found that in Canada competencies are most commonly used within the medical profession, including medical education, professional training and accreditation and in public service. One major concern is that frequently competencies are applied in a varied and un-uniform manner, and they can be assessed in sometimes unreliable ways. In addition, competencies might be too rigid or inflexible for some organizations, too focused on the cultural contexts of a specific country and have limited international transferability. Contextual factors also limit competencies transferability – for example, in the area referred to as public health human resources, the needs of each community are dependent upon the context, the place, location and its particular needs.

LPSS users want better resource recommendations, social connectivity with others with similar competencies and the ability to showcase the competencies. Future LPSS development of competency functionality will focus on providing capability to capture both formal and informal competencies and will target competency functionalities that address LPSS users’ concerns such as the lack of ability to showcase their competencies, and the connection between their competencies, job opportunities, and social networking opportunities.