Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

“Out of the necessity to find a better way to represent children’s conceptual understanding emerged the idea of representing children’s knowledge in the form of a concept map… One of the powerful uses of concept maps is not only as a learning tool but also as an evaluation tool” [1].

Since the emergence of Concept Mapping, applications of the technique and its artifacts have expanded far beyond representing “children’s knowledge”. Indeed, professional applications of Concept Mapping have extended the technique into realms that stretch the notion of knowledge representation [2]. A primary use with adults has been the elicitation and representation of adults’ knowledge, particularly for the purpose of preservation in the face of an aging workforce [3]. After nearly two decades of knowledge preservation activity, relatively little work has explored the organizational use of Concept Maps elicited from experts. While some have suggested that such products could be used to accelerate the achievement of expertise [4] and organize knowledge for transfer [5], answering the question, “Now what do we use these?”, remains a daunting challenge for professional Concept Mappers [6].

This paper describes an attempt to get back to the roots of Concept Mapping as a means of both representing and evaluating knowledge, in the context of professional work. We describe a pilot project in which we used Concept Mapping to elicit and represent knowledge from domain experts, then demonstrated the use of Concept Maps for assessing the mental models of other professionals. For the latter, we introduced Sero! – a prototype Concept Map-based learning assessment software platform. The project produced a model for a general approach for the organizational use of Concept Mapping and Concept Maps.

2 Project Motivation

Globally, human beings are facing serious challenges from demographic imbalances related to aging. In the United States, for example, the number of people age 65+ will more than double, increasing from 13 % of total population in 2010 to 21 % of total in 2050 [7]. Thus, experienced, proficient workers will populate the workforce as they near, then enter, retirement. Their eventual exit sets up the prospect of widespread knowledge loss. DeLong [8] has chronicled four big costs of this trend for organizations: reduced capacity to innovate, threatened ability to pursue growth strategies, increased incidence of errors, and increased number of efficiency losses—and how they can wreak havoc across a variety of industries. Particular segments of the economy will feel these costs more acutely, for example the utilities and power industries. The sheer size of the problem for the utility industry in the U.S.—i.e., as of 2008, 65 % of senior engineers were eligible for retirement within three years—means that some of the costs will be unavoidable.

ReliabilityFirst is one of the eight Federal Energy Regulatory Commission-approved Regional Entities responsible for ensuring the reliability of the North American Bulk-Power System (BPS). Early in the company’s formation, a classic regulatory scheme was adopted, to enforce adherence to grid reliability and cybersecurity standards and requirements. As time progressed, the regulatory scheme evolved to focus on providing education and assistance to the power industry to move beyond simply compliance into reliability and security “beyond the standards.” This evolution, coupled with the aging of the electric industry workforce, highlighted the need to mitigate the risk of losing internal expert knowledge. Therefore, ReliabilityFirst retained Perigean Technologies in 2015 for a pilot project demonstrating knowledge elicitation (KE), including Applied Concept Mapping, for the preservation of expert knowledge.

2.1 Knowledge Elicitation as a Means of Preservation

Knowledge elicitation (KE) is the process of scaffolding the expression of knowledge, wisdom and knowhow through in-depth interviews and observations about cognitive events, structures, or models [9]. As suggested by the Electric Power Research Institute, “there is no right or wrong knowledge elicitation method or set of methods. The choice depends on a range of considerations, some of which may not come into play until knowledge elicitation is under way” [10]. Applied Concept Mapping has been employed extensively throughout the U.S. utilities industry as a preferred knowledge elicitation method, primarily due to its flexibility and efficiency [11–13].

2.2 Assessment as a Means to Reintroduce Expertise

Learning assessment in adult workplaces is an understudied phenomenon [14]. Several reasons underlie the lack of attention, not the least of which is the organizational tension between learning and getting the job done [15]. It is not entirely clear how prevalent the formal assessment of learning is, though the use of cognitive assessments for the purposes of selection has been growing along with the rise of other psychological assessments of candidate recruits [16]. Such assessments are typically focused on generalized knowledge and capabilities, rather than specific knowledge domains.

From what is known, learning assessment is carried out in formal and informal ways, with the latter comprising methods such as providing on-the-spot feedback following task performances, which may not even be recognized as an assessment activity [15]. Where it is formalized, it is often employed for the purpose of granting some form of certification, reviewing performance, or enforcing accountability. The “view that the purpose of assessment is to support current learning and foster further learning has not tended to be the primary purpose of assessment in relation to industry training” [17]. Despite being a sine qua non to the development of expertise [18], “[a]ssessment is often treated as separate from learning when it should be considered a part of learning” [19].

Yet, the “demand for ongoing learning has implications for workplace assessment” [20]. These include: who creates and proctors the assessment; when and why it might be given; and how results might be used. In workplace settings, supervisors, mentors, and/or trainers may take on the role of assessor and may or may not be responsible for the creation of the assessment. Also, the traditional boundaries between formative and summative assessment are looser than in educational settings, which opens the door for pragmatic learning assessment approaches. “Sometimes summative forms of assessment can be used in formative ways—for example, written comments on tests or discussion of test results. The discussion occurs in order to further the learner’s understanding of what (the test showed that) they do and do not know. Without that discussion, the learner simply receives a judgement with no explanations and is unlikely to learn further from that judgement on its own” [21].

The relative openness of adult workplace learning assessment environment offers an opportunity to use novel assessment practices as a means to reintroduce expert-derived Concept Maps created through knowledge elicitation back into the organization. Such a program can draw on the assessment approaches that have been used with children, and faces many of the same and some additional new challenges.

3 Sero! – Learning Assessment Platform Using Concept Maps

Since 2012, the authors from Perigean Technologies LLC have been developing a software platform to enable learning assessment using Concept Maps. The project has been built upon prior lessons learned in developing a Concept Mapping game (also called Sero!) [22] that stemmed from traditional motivations for game development – i.e., fun. The learning assessment version of Sero! grew out of motivations aligned more closely with traditional applications of Concept Mapping.

3.1 Motivations

Meaningful diagrams have for decades shown to be highly diagnostic of how learners organize and represent knowledge across the novice to expert spectrum [23, 24]. In an early example, Cooke and Schvaneveldt [25] demonstrated how similar meaningful diagrams (Pathfinder networks) revealed differences in memory structures and conceptual representations for naïve, novice, intermediate and advanced computer programmers. The U.S. Department of Education calls for Cmapping as one of the types of Interactive Computer Tasks that is “highly recommended” [26, 27] for inclusion in every National Assessment of Educational Progress (NAEP) Science Assessment at the 8th and 12th grade levels [28].

Adult Learner Assessment using Concept Maps.

Compared to the volumes of research in K-12 and higher education environments, relatively little work has explored Concept Maps for use in adult workplace assessment, learning and training contexts. Stevens [29] used pre-drawn fill-in Cmaps for assessing adult learning in a 40-hour training course called Hazardous Waste Operations and Emergency Response, which was taught to personnel working with hazardous waste and offered in various sites across the U.S. The Cmapping method was shown to have higher reliability than the control (multiple-choice test), and both were positively and moderately correlated in post-test use. In the context of a Counter-Improvised Explosive Device (IED) training solution evaluation, Moon et al. [30] demonstrated that Cmaps provided an efficient means to model levels (novice, intermediate, and expert) of U.S. Marine Corps personnel’s Counter-IED knowledge and skills. Knowledge and skill models were represented in Cmaps, which enabled both quantitative and qualitative analysis of the differences across groups. Once modeled as Cmaps, the same descriptions of expert knowledge were used to assess learning using Multiple-Choice Concept Maps. When presented to learners in a pre/post-intervention assessment context, clear learning performance differences were discovered.

More recently, Knollmann-Ritschel and Durning [31] explored the use of Cmap-based assessment for team-based learning in the context of military medical instruction. To assess understanding of content across a learning fundamentals module at the Uniformed Services University of the Health Sciences, the researchers replaced an individual assessment using multiple-choice questions with Cmaps, and combined the assessment with a group assessment and application exercise wherein teams created Cmaps. Learner performance and feedback from faculty and learners support the use of Cmaps in team-based learning. They also demonstrated the benefit of Cmapping in knowledge acquisition, organization of prior and new knowledge, and synthesis of that knowledge across disciplines.

Assessment Approaches and Challenges.

Strautmane [32] offered a comprehensive review of the variety of assessment tasks. One method involves starting a map from scratch, and asking the learner to generate and organize all of the content (i.e., concepts, links, linking phrases, arrowheads, propositions). This is a highly unconstrained task that is notoriously difficult to analyze. Additional constraints can be placed upon the task, and these constraints also simplify the process of analyzing the learner responses. Learners can be provided a skeleton map with some content provided and some content missing, and then required to fill-in and/or organize the content by creating linking lines. This approach is referred to as a generate-and-fill-in (GAFI), and it can be performed with or without any diagram organizing tasks – i.e., spatial organization, arrowhead additions. A skeleton map can also be provided, along with a set of content for the learner to use to fill-in the rest of the map. This approach is called select-and-fill-in (SAFI), and it also does not require organizing tasks. The SAFI task can be made more complex by presenting distractor content. A Multiple-Choice Cmap (MCCM) provides all of the content in the map but requires learners to select content from provided choices, which can include distractors that can be drawn from within the same Cmap or other sources [30, 33].

There are numerous challenges to implementing a Concept Map-based assessment system. Given the proliferation of techniques available, it is no surprise that assessors and learner understanding of Cmaps and their purpose can also hinder successful implementation, an issue that has been seen repeatedly in partial implementations [34–36]. Guidance for selecting which Cmap-based assessment technique(s) to use for which learning purpose is scarce in the literature. Ruiz-Primo offers some guidance to assessors: “in order to better determine the cognitive demands evoked by different mapping techniques, it is important to consider not only what is provided, but also the extent/amount of what is provided, the significance of what is provided, and what is required from the examinees” [37].

With regard to scoring, a wide range of rubrics and methods has been explored. They range from subjective measures (e.g., number and appropriate inclusion/exclusion of concepts and propositions, structure of the map) that are typically used with the more unconstrained tasks [38], to objective measures of correctness that are feasible with more constrained tasks. McClure and colleagues [35] compared holistic, relational and structural scoring methods without and with the use of a master map unveiling a high reliability for the latter. While challenges in scoring reliability were cited early in the research [39], it has been broadly demonstrated that methods that compare the learners’ performance against a master/criterion/reference/expert map are the most valid and reliable method for scoring Cmap tests [40].

Technology Gap.

The time and labor required to manually set-up an assessment, analyze the maps, or transcribe their content for analysis, can be considerable [41]. Above all else, this is the challenge that has bedeviled broadscale deployment of Concept Map-based assessment. Indeed, this challenge is only magnified in adult workplace learning environments. Several prototype software tools have demonstrated rudimentary, prototype capabilities to compare and assess Cmaps for the purposes of learning assessment [42]. Yet, as Strautmane has stated, “there is still a need for a [Cmap]-based knowledge assessment system that could perform the assessment automatically with a little intervention by the [instructor]” (emphasis added, [43]). Our aim with Sero! is to attempt to meet the challenges described above with a software tool that enables learning assessment using Concept Maps derived from domain experts.

3.2 Software Description

Sero! is a browser-based application hosted in the cloud that currently enables basic functionality for designing, distributing, and analyzing two Cmap-based assessment tasks (i.e., GAFI and MCCM). A notional view of the current Sero! architecture is shown in Fig. 1.

Fig. 1.
figure 1

Sero! notional architecture.

Full size image

The Sero! research and development database (DB) utilizes a CouchDB cloud repository to store structured JSON documents that represent users, assessments, and Cmaps. The client-side browser application renders the app as a single HTML page using Angular.js and the D3.js rendering library to create force-directed graphs based on Cmap data. The DB creates aggregate views and indexes that provide metrics about assessment data to the application. CouchDB provides a RESTful API to add/edit/render documents and views.

In Sero! the two user types are Assessor and Learner; Assessors create assessments then proctor them to a class of Learners. Every user regardless of type will interact with Cmaps in different stages of completion by, for example, manipulating the position of the nodes or changing the words inside the nodes to create different propositions. This functionality is the heart of the application, first used by the Assessor to create new assessments, then by the Learners to complete the assessments, and again by the Assessor to provide feedback on finished assessments. Manipulation of the graph in the graphical user interface (GUI) updates the local JSON representation of the Cmap. This document is loaded to CouchDB via web services when the assessment is first created by the Assessor and then updated when completed by a Learner.

To date, our internal software engineering efforts have focused primarily on architecture development and basic GUI. Sero! currently provides basic mechanisms for conducting two types of Cmap-based assessment. Figure 2 shows the Sero! GUI from the learner and assessor views. In the assessor view (left side of the figure), assessors can select which content will be assessed, assign an assessment type (GAFI or MCCM), and generate distractors (for the MCCM). In the learner view (right side of the figure), the concepts, linking phrases, and propositions in yellow are items that need to be addressed by the learner, either by filling in content (i.e., GAFI method)—which requires recall—or selecting from alternatives (i.e., MCCM)—which requires recognition.

Fig. 2.
figure 2

Sero! GUI.

Full size image

Having described Sero! and the motivations underlying its development, we now turn to our demonstration project, which attempted an end-to-end implementation of a model for the organizational use of Concept Mapping and Concept Maps.

4 Demonstration: Knowledge Elicitation Activity

The focus of the knowledge elicitation activities was to preserve any potential high level knowledge from the three chosen experts, and to assess the organizational use of knowledge elicitation and transfer methods.

4.1 Process

Our KE process followed a modified version of the recommended Cmapping technique. In the described technique, which is shown in Table 1:

Table 1. Comparing recommended and modified techniques for Concept Mapping knowledge elicitation
Full size table

one researcher acts as a facilitator and provides support in the form of suggestions and probe questions, while the other acts as the mapper and captures the participant’s statements in the Concept Map, which is projected on a screen for all to see. The mapper needs to be proficient at quickly and accurately conducting the mapping work on the fly. This includes a facility for glancing to and from the computer monitor and the projector screen to follow the facilitator’s guidance and the participant’s statements [44].

In our demonstration, as in many KE sessions the lead author has executed, the roles of facilitator and mapper are executed by a single facilitator/mapper – i.e., elicitor. The most common reason for this modification is funding, as one knowledge elicitor is cheaper than two. Having a single elicitor introduces many challenges and advantages. The main challenge involves maintaining continuous engagement with the SME while also building the map. The main advantage is the removal of coordination activities between facilitator and mapper.

The main challenge is very difficult to reconcile and is aggravated by other factors. In the experience of the lead author, many SMEs prefer engaging directly with the facilitator/mapper, ignoring the projected, emerging Concept Map altogether. Concept Mapping is often new to them, and given that they are sharing their intimate knowledge, having a conversation may feel much more comfortable than building a diagram. In some cases, projecting the Concept Map for all to see is not feasible.

For these reasons, the lead author’s technique for Concept Mapping KE as a single facilitator/mapper typically proceeds as shown in Table 1, which also contrasts the modifications of the recommended technique. Of course, with either technique, a great deal of experience with KE and Concept Mapping is required [45].

The main contrast is the collapsing of steps 3–6 into a single step 3, which is executed by the mapper after the KE session. The mapper uses the parking lot arranged concepts captured during the session, linking them together based on the recorded statements from the SME. The draft Concept Map(s) are then provided back to the SME for review, which can be done in a format most comfortable for the SME.

We have not conducted a formal comparison of the modified technique with the recommended technique. We suspect the modified technique would yield greater volume, as the amount of discussion possible without the burden of map-making during the KE session is likely greater. We also suspect that the modified technique offers some efficiency gains – i.e., measured by propositions per minute [46] – especially if the audio recording is played back at a faster rate than the original recording. Efficiency comparisons would also need to include the time necessary for iterative review. In our demonstration, we received different amounts of feedback from our SMEs. One provided only a few minor changes, while another suggested major restructuring.

4.2 Validation Check

For an additional validation check of the elicited knowledge, we compared the Concept Maps we created to a previously developed report — 2015 RF Regional Risk Assessment (RRA; [47]) — that focused on topics that were relevant to the knowledge we elicited. The validation was fruitful, and helped to cross-check the data-driven, quantitative RRA report with the qualitative results of the experts in the Concept Maps, helping to illuminate to potential areas to research in the 2016 assessment.

5 Demonstration: Concept Map Products

Our KE process yielded two Concept Maps per SME. One Concept Map described the specific domain – i.e., cybersecurity, generation, transmission – and the second described aspects of the SMEs’ jobs and tasks – i.e., how they performed analysis and outreach. In hindsight, it is not surprising that as the SMEs described the outcomes of their work, they quickly got into the macrocognitive challenges involved in producing the outcomes [48]. While some of these aspects were described throughout the KE session, the mapper was able to organize the content into distinct Concept Maps after the sessions. Sample, sanitized Concept Maps are shown in the next sections. They were created using CmapTools [49] Figs. 3, 4 and 5.

Fig. 3.
figure 3

Concept Map describing generation risks to the BPS

Full size image
Fig. 4.
figure 4

Concept Map describing an analysis process for generation risk to the BPS

Full size image
Fig. 5.
figure 5

Concept Map describing an analysis process for transmission risk to the BPS

Full size image

5.1 Risk Maps

Concept Map describing generation risks to the BPS

5.2 Job and Task Maps

Concept Map describing an analysis process for generation risk to the BPS

6 Demonstration: Findings

With our KE content captured, we next turned to our learning assessment demonstration. The scope of the assessment was to ascertain what learners may already know, in the spirit of meaningful learning [50]. Three ReliabilityFirst personnel with various levels of experience in the domains, and one external consultant with related industry experience, were asked to complete three Sero! assessments as proxy learners.

6.1 Assessment Set up

The first step in setting up the assessments was bringing the CmapTools-created Concept Maps into Sero!. The mechanics for doing so were straightforward – export propositions as text, save as a.csv file, import into Sero!. The prototype version of Sero! used for the demonstration rendered the Concept Map, and allowed for basic spatial manipulation. We did not import the Concept Maps in total. While there are certainly view-ability constraints on rendering the Concept Maps given their sizes, there are likely also constraints on size with regard to assessment. We opted to use Concept Maps with approximately 25 to 35 elements (concepts and linking phrases), as these seemed manageable for any given assessment. This often required finding reasonable breaking points within the larger Concept Maps, which in some cases required repeating content across assessments.

The next step involved determining which sections of the Concept Maps were amenable to assessment. There were several considerations for making the determination. Sero! can currently enable GAFI and MCCM assessment items. For more elaborate concepts or linking phrases, it was not reasonable to expect a learner to generate matching content. But while elaborate concepts and linking phrases were appropriate for MCCM items, it was necessary to determine what sorts of distractor content was appropriate in order to make selection challenging. Here we drew on other content within the map where the item required several distractors. For MCCM items at the linking phrases, we also looked for linking phrases whose opposites or scales could be offered for selection – e.g., increases/decreases, every 4/30/60 s. We also considered the relative importance of concepts and linking phrases in the larger context of the Concept Map. Concepts and linking phrases that had multiple outgoing and/or incoming connectors were deemed high priority for assessment. Linking phrases that were central to demonstrating understanding of the domain were also deemed valuable for assessment. We frequently used fans [51] as a queue for GAFI items, as these exercise recall skills that can reasonably be thought to also assess conceptual organization.

6.2 Assessment Taking

Our participants were provided access to their three assessments via the Internet. Figure 6 shows the user interaction for a GAFI item (left side of the figure) and for a MCCM item (right side of the figure) for one of the transmission assessment Concept Maps.

Fig. 6.
figure 6

Concept Map describing an analysis process for transmission risk to the BPS

Full size image

6.3 Scoring

The prototype version of Sero! enabled scoring correct answers for the MCCM items, and a simple match for GAFI items. Results were accessible in the assessor role, but not available automatically to the learner role. We manually produced result sets for review by the participants by overlaying the assessment items (in yellow highlight) with the comparative results, green for correct/match and red for incorrect/mismatch, as shown in Fig. 7. In this example, three items did not match, and one did.

Fig. 7.
figure 7

Sero! results

Full size image

6.4 Participant Feedback on Process

Our participants provided feedback on the process. All reported having enjoyed the process and being challenged by the assessment task. Regarding the individual assessment items, they found several points of disagreement, particularly with the GAFI items. Their objections were warranted, given the immaturity of the prototype Sero!’s basic matching algorithm.

Most importantly, all indicated that working through the Concept Maps revealed gaps in their mental models of the domains. Interestingly, because the assessment context was not controlled, they all reported executing information search strategies to seek answers for the assessment – strongly suggesting the capacity for Sero! assessments to prompt self-learning activities.

7 Future Directions

Our project will continue along two pathways. We will continue to conduct KE with at-risk ReliabilityFirst personnel. As of this writing, target topics and personnel are being selected.

We will also advance our development of Sero! through involvement in the U.S. Government’s Advanced Distributed Learning (ADL) Initiative (www.adlnet.gov). ADL is developing a Training and Learning Architecture (TLA), a set of standardized Web service specifications and Open Source Software designed to create a rich environment for connected training and learning – i.e., an intelligent learning ecosystem. The authors from Perigean Technologies LLC will be advancing Sero!’s design and development to provide the TLA with a deep learning assessment capability to inform on learner progress and to prime learning content, toward enabling a personalized learning environment for adults. Advancements will include maturing Sero!’s GUI, comparison algorithms, mechanisms for cumulating and aggregating results, and integrating with the TLA.

8 Summary

This paper has demonstrated a general approach for organizational use of Concept Mapping and Concept Maps. Beginning with knowledge elicitation and continuing through learning assessment, our approach holds great opportunity to reduce the risks of organizational knowledge loss, and to create new opportunities for leveraging this knowledge for improved risk management and employee engagement. Achieving such benefits requires advances in techniques and software, which we have offered here.