Keywords

1 Modeling Metacognitive Activities in Medical Problem-Solving with BioWorld

Medical diagnostic reasoning is complex and ill-defined in that there is no single problem solving sequence for obtaining the correct answer. There are many routes to solving the problem and one medical problem may lead to a new set of medical issues that need to be resolved. Well-defined problems, on the other hand, often have clear procedures and outcomes. Ill-defined problems are more difficult to solve since there is no set of rules that will lead to the right answer [1].

Consequently, novice physicians must learn to monitor and control their problem-solving efforts by executing actions that will help them progress towards a correct diagnosis. Self-regulation is critical for effective medical problem-solving in that physicians must orient their actions and evaluate the consequences of such actions before planning new ones.

BioWorld is a technology-rich learning environment (TRE) designed to help medical students regulate their learning about medical reasoning by providing feedback in the context of learning to solve virtual patient cases [2, 3]. BioWorld provides tools that scaffold the learner’s requisite cognitive and metacognitive activities.

Novices attain higher levels of competence through deliberate practice [4] as the system dynamically assesses their performance against expert solution paths and provides the necessary feedback.

Dynamic assessment in this system relies on a novice-expert overlay. This overlay is used to develop feedback when novices request help. When help-seeking occurs, help is provided by the tutoring module which applies a set of pre-defined rules based on the context of the learner’s activity. The system also provides cumulative feedback by comparing the novice solution with an expert solution following the completion of the case. Our contention is that each individual has a different learning trajectory within specific problem solving contexts [5]. This trajectory can be identified by designing a learner model within a computer-based learning environment that captures the learner’s competence and performance within a domain of study.

BioWorld assesses the learning model against an expert model of competence and performance and provides scaffolding that fosters cognitive and metacognitive activities within the domain of medical problem-solving. This allows the system to assess novice performance along the path towards competence and enables the system to deliver support and feedback tailored to the individual needs of different novices; a key factor in successfully fostering the development of metacognitive skills and knowledge. In this way BioWorld captures and assesses learners’ trajectories towards expertise in medical reasoning. However, there are several challenges involved in fostering metacognitive activities while solving problems in BioWorld using an expert model.

This chapter covers the essential design guidelines for scaffolding metacognitive activities in problem-solving within the domain of medical education. Specifically, we review recent advances in modeling metacognition by outlining analytical techniques to design, evaluate, and develop expert models by capturing metacognitive activities in problem-solving. We demonstrate the use of on-line measures, including concurrent think-aloud protocols as well as video-screen captures, and log-file traces of user interactions. Educational data mining techniques are outlined with the goal of capturing metacognitive activities as they unfold throughout problem solving.

These trace methodologies are used to model self-regulatory processes along the trajectory towards competency in diagnostic reasoning. We summarize three studies that examine help-seeking activities in the context of BioWorld. These findings lead to insights with respect to designing appropriate scaffolding tools in order to promote higher levels of competence in novices. Future directions for expert-driven models of metacognition are outlined. We commence our chapter with a detailed discussion of how BioWorld is designed to scaffold medical problem solving and metacognition. Some of the principles for designing metacognitive scaffolding tools for BioWorld can be generalized to designing metacognitive scaffolding tools for other computer-based learning environments.

2 A Model of Metacognitive Activities in Problem-Solving

The study of self-regulation within domains requires consideration of the task that is performed by a learner as well as the strategic processing demands that are inherent to the domain [6]. In the medical field, self-regulation has been studied in numerous ways but from different lenses, such as: self-assessment in the context of professional development [7]; examining the interaction between personal attributes and environmental affordances [8]; and examining the developmental phases that occur through clinical practice [9].

For the purposes of this chapter, we outline a model that synthesizes existing accounts of self-regulation in problem-solving and situates the underlying activities in diagnosing patient cases in the medical domain [10–12]. We model cognitive and metacognitive activities in the context of BioWorld, a TRE that serves as a platform to support novices in solving problems within the medical domain.

We conceptualize self-regulation as a super-ordinate construct that encompasses metacognition, namely the ability to orient oneself in the problem space, plan and execute actions, monitor outcomes, as well as evaluate and elaborate a solution [3]. Solving such problems requires more than clinical experience, it requires the ability to regulate problem-solving by adapting one’s approach to solving the problem.

Social cognitive models of self-regulation characterize metacognitive activities as occurring as part of a recursive and iterative process involving forethought, performance and reflection, where adjustments to the solution are made on the basis of progressively refining the problem space [13, 14].

In the forethought phase, novices orient themselves in the problem space, at the same time, formulating a plan to solve the problem. The performance phase is characterized by the novice’s efforts to solve the problem by executing the planned steps and monitoring the outcomes. The self-reflection phase involves the novice’s evaluations of the overall progress and elaborations about the problem space, resulting in conclusions about the case. The problem-solving process is recursive in that the outcomes of prior steps inform the next ones that are taken to solve the problem (as shown in Fig. 13.1).

Fig. 13.1
figure 1

Model of metacognitive activities in problem-solving

Full size image

Metacognitive activities determine a learner’s progress through each phase of diagnostic reasoning. For example, diagnostic hypotheses are refined progressively by engaging in strategic processing until a final diagnosis is reached. We distinguish amongst several types of metacognitive activities; namely, orienting, planning, executing, monitoring, evaluating, and elaborating.

Self-regulated learners orient themselves to a problem space by identifying or recalling information that is relevant to their efforts to outline tentative hypotheses (diagnoses) for the patient condition. In an effort to test these hypotheses, learners formulate plans that involve ordering lab tests, searching for specific symptoms or information about the disease, and asking for consults.

Once the plan is executed, learners make efforts to monitor the outcomes by evaluating their own progress or noticing unexpected or conflicting information. This appraisal may lead learners to revisit their own diagnosis by reviewing all the available evidence and hypotheses, leading to changes in the final diagnosis.

On the other hand, learners might elaborate further their own diagnosis by making conclusions on the basis of the evidence. As an example, a proficient learner may notice pertinent vital signs, such as the patient heart rate exceeds the normal range, which could be caused by a tumour of the adrenal glands. To test this assumption the learner formulates a plan to test for pheochromocytoma by verifying the serum levels of the catecholamines, adrenalin and noradrenalin.

The results indicate that the serum levels are elevated, which is pertinent to a diagnosis of pheochromocytoma. As such, the learner order a series of tests to rule out known alternatives to the diagnosis, while concluding that pheochromocytoma is a likely explanation for the patient condition.

Although the above example indicates a linear solution path, learners experience challenges to regulating their progress toward reaching a solution. Learners may fail to identify patient symptoms and vital signs that are indicative of the correct disease due in part to the low prevalence of this disease in the broader population.

Successful problem-solving requires that the learner makes an appropriate judgement that the serum-levels are in fact elevated, which implies that the learner is knowledgeable of the normal range of serum-level values.

Finally, learners may vary considerably in their levels of confidence in the final solution, depending on their ability to rule out their other hypotheses and to provide evidence that their final diagnosis is correct. BioWorld allows learners to practice their medical diagnostic reasoning by providing them with virtual patient cases.

BioWorld provides learners with feedback that highlights the similarities and differences between learners’ approaches to solving the problem and the expert solution path. In the next section, we review the design guidelines of BioWorld, and how it supports learners in regulating their own progress in solving patient cases.

3 BioWorld: A Deliberate Practice Environment for Diagnostic Reasoning with Virtual Patients

BioWorld is designed using the principles of a cognitive apprenticeship approach to instruction [15, 16]. Cognitive apprenticeships provide learners with opportunities to link abstract knowledge to real world experiences. In this case students apply their knowledge to medical patient cases. When diagnosing virtual patient cases is too difficult for learners to do alone, they can still appropriate knowledge from skilled experts and mentors.

BioWorld provides mentorship in the form of computer coaching and cognitive tools [17] embedded in the learning environment, which help structure the learning experience for novice learners. Medical students test their knowledge by formulating a diagnostic plan to solve a case, gather pertinent information about the patient, and obtain feedback in relation to the diagnostic process, as shown in Fig. 13.2.

Fig. 13.2
figure 2

BioWorld main user interface

Full size image

In doing so, the software serves as a training platform for novices to practice regulating the diagnostic process and become more proficient. The path taken by expert physicians in solving the problem is modeled and made explicit to novices, thereby supporting them in diagnosing rare diseases that would be otherwise beyond their reach [2]. Forethought processes include framing the patient’s problem prior to formulating a diagnosis about a patient case.

Novices begin by gathering evidence (i.e., patient symptoms, history, etc.) from the case description and highlighting the evidence they consider important to the patient case. They send their evidence to an Evidence Table that is visible throughout their problem solving activity. Medical students use the Hypothesis Manager dropdown menu to indicate their differential hypotheses.

The menu displays a comprehensive list of diseases organized by the biological system that is afflicted. Physicians formulate differential diagnoses where they consider more than one disease at the same time. They can pick up to 10 diseases at the same time; however, they must indicate the level of confidence they have in each hypothesis by using the Belief Meter and select which disease they believe to be the most likely.

During the performance phase, novices search for evidence pertaining to a disease, conduct diagnostic tests to confirm or disconfirm their hypotheses, and search the online Library for additional information regarding the typical symptoms and transmission routes of a disease, definitions of standard medical terminology, and range of normal values for a specific laboratory procedure. Learners monitor their progress in solving the problem by evaluating their diagnostic test results and evaluating the patient’s vital signs in the context of the Chart interface.

The Consult allows novices to request hints that are delivered in increasing order of specificity. After learners have submitted their final diagnosis, BioWorld supports them with additional tools that foster reflection on the diagnostic process.

The Categorization panel allows them to categorize their own evidence items stating which items confirm, refute, or are irrelevant to their final diagnosis.

Following categorization students use the Prioritization panel to rank the evidence items in terms of their relative importance to the final diagnosis. In the Case Summary panel, novices write a brief justification for their final diagnosis on the basis of the evidence items that were gathered throughout the case.

This summary is written for the next hypothetical physician who would see the patient. Finally, the Student Report provides formative feedback to each student by highlighting the similarities and differences between their solution steps and that of a validated expert.

The expert solution path also provides a case summary, written by an expert, which outlines in detail the steps that were taken to solve the case, and how each step contributed to formulating the final diagnosis. In the next section we focus on the importance of metacognitive scaffolding within BioWorld using expert models.

4 Metacognitive Scaffolding with Expert Models: Novice-Expert Overlay Component of the Expert Model

Developing expertise requires practice with appropriate levels of scaffolding. BioWorld [2] provides opportunities for deliberate practice [4] by making “expert models of performance and competency more visible to learners in the context of the problem solving” [18: 805]. BioWorld was developed using a cognitive apprenticeship framework [15] where medical students learn clinical diagnostic reasoning by practicing realistic diagnostic tasks and are scaffolded in the context of their learning with expert models.

In BioWorld diagnostic reasoning is assessed using a novice-expert overlay system [19, 20]. An overlay model highlights differences between the solution paths of learners and experts, often revealing learner misconceptions. It is important to identify differences between novices and experts to support the novice along a learning trajectory that will lead to more expert levels of performance.

Variations of novice-expert models have been used in TREs to serve different purposes. In BioWorld, novice-expert models have been used to examine clinical reasoning dynamically, with a particular focus on process models as well as outcome comparisons [21]. The BioWorld user-model detects relevant patient symptoms and patient history, diagnostic laboratory tests ordered, and library information accessed.

Evidence items characterize the learner’s path to solve the problem, which are analysed to identify similarities and differences with the expert’s approach. Visualizations of the novice and expert models provide learners with the opportunity to review a representation of their problem solving steps and become cognizant of where their solution path differed from an expert’s solution path.

In addition, this comparison can help learners attend to evidence they missed or evidence that is not part of the expert solution. Identifying these differences in the reasoning path or the decision-making procedure is particularly useful for revealing learners’ misconceptions and incorrect reasoning strategies.

BioWorld captures user interactions and compares them to an expert model for the purpose of adapting instruction to the specific needs of the learner. The expert model fosters metacognition as learners are supported in formulating plans, monitoring their progress, and adaptively engaging in strategic actions while solving problems. The novice-expert overlay model individualizes feedback by highlighting similarities and differences between their respective solution paths.

Another example of this methodology is demonstrated by [22] who uses learning trajectories to model metacognition as learners plan, monitor, and evaluate their cognitive behaviors in a problem space.

The Interactive MultiMedia Exercises platform allows learners to practice solving simulated chemical analyses problems. Learning trajectories are computed based on problem outcomes, using item response theory estimates of solution frequencies.

The solution process is then analyzed in terms of strategic activities using Artificial Neural Networks (ANNs) and Hidden Markov Models (HMMs) [23]. BioWorld analyzes strategic activities while solving problems by providing learners with an external representation of the trajectory towards competency in diagnostic reasoning.

This representation is composed of evidentiary items that justify the case solution obtained through a cognitive task analysis of several experts, wherein the symptoms and laboratory tests pertinent in solving a case are identified. The feedback provided in BioWorld highlights similarities and differences between the expert’s and novice’s approaches. Expert-driven modeling in BioWorld assists in supporting the help-seeking behaviors of learners through the use of the novice-expert overlay.

5 Developing an Expert Model to Trace Metacognitive Activities in Problem-Solving

Developing an expert model requires knowledge extraction from experts. We developed CaseBuilder (see Fig. 13.3), a case authoring tool for expert physicians, to create virtual patient cases for medical students to solve in BioWorld [24]. The experts that we work with are also medical instructors.

Fig. 13.3
figure 3

CaseBuilder

Full size image

The expert must externalize his or her knowledge about a case by filling in all of the requisite elements needed to solve the case and thereby defining the problem space by identifying the elements used in the novice-expert overlay model that are needed to individualize feedback. CaseBuilder is also used to create context-specific hints that are delivered through the help-seeking model.

The physician enters each case through the fields that are provided. Currently, patient information is linked to vital signs, disease categories, diagnostic tests, and expert arguments for solving the cases. Once the expert enters the data, the case is stored and it is usable by the BioWorld engine to present the new case to medical students. The CaseBuilder is an excellent tool for extracting expertise from the medical instructor.

In the section below we examine the use of the expert model to foster metacognition. In particular, we examine help-seeking behavior to determine when novices are at a learning impasse. The expert overlay model is used in the context of help-seeking to foster metacognitive and cognitive activities.

6 Examining Help-Seeking as an Indicator of Metacognition

Help-seeking is considered a metacognitive process since it indicates that learners monitor their problem solving and identify when they lack prior knowledge or the competency to continue the task independently. Help-seeking is particularly important in the context of solving problems since it is an indicator of obstacles and learning impasses [25].

To overcome these obstacles the learner can ask for help from a more knowledgeable other or from the TRE’s help system. Once the learner has received help they must evaluate if the help was useful and if it was sufficient to continue solving the problem. If the help provided was not sufficient the learner can repeat the help-seeking process until they are provided with enough information to continue the task.

Help-seeking is a cyclical process which involves every step of self-regulation: planning, monitoring and reflection [26]. An obstacle in BioWorld might occur when the laboratory tests ordered do not support a hypothesis or when the evidence provided does not fit a specific hypothesis.

Karabenick [27] distinguishes between three types of help-seeking behaviors, avoidance, executive, and instrumental help-seeking. Help-seeking avoidance refers to instances when a learner fails to request help, despite the fact that help is in fact required.

Executive help-seeking involves a request for help when it is in fact needed; however, the learner requests final answers and makes no attempt to solve the problem independently. Instrumental help-seeking consists of a help-request where a learner asks for only the amount of information that would be sufficient to solve the problem independently.

As such, the distinction between each help-seeking type is associated with the necessity of the relevant information [28]. These three types of help-seeking behaviors can be further categorized as maladaptive or adaptive.

Help-seeking is considered maladaptive when a learner fails to use help functions effectively or ignores them entirely (see [29] for a review). In the medical domain, learners avoid the use of help functions such as glossaries, hyperlinked lectures, and expert advice, despite acknowledging that they lack the prior knowledge necessary for diagnosing the case on their own. Thus metacognitive awareness of what is known or not known does not lead to the execution of appropriate actions. In one study, fourth year medical students avoided or ignored help more than adaptively incorporating the help provided into to their diagnosis [30]. Maladaptive help-seeking behaviors were significantly correlated with poor quality solutions. This example demonstrates that even advanced medical students avoid or ignore help and this is problematic because maladaptive help behaviors are associated with poor quality diagnoses.

Help-seeking is considered to be adaptive when it enables students to continue the learning task independently [31] and this process is described as an exchange between self-regulation and other-regulation [32]. The support provided by more knowledgeable others scaffold learners to continue the task independently enabling them to complete a task that they would otherwise be unable to complete [33]. This exchange between self-regulation and other-regulation is necessary in order for the individual to become an autonomous learner.

Learners can obtain help from TREs as well, and scaffolding is provided to learners based on their current level of performance. The hints provided by the system supply the learner with enough information to continue the learning task independently without providing the final solution. In the same study mentioned above, adaptive help-seeking, although less prevalent than maladaptive help-seeking, was significantly correlated with better quality diagnoses [30].

It was suggested that for medical students to use help functions effectively, the help provided must be contextualized so that learners can apply the additional information offered by the system directly to the problem [30]. In BioWorld, learners have access to two help-seeking tools: the Consult Tool and the Library Tool. The Consult Tool provides context specific on demand hints that are delivered in increasing order of specificity and the Library Tool provides a glossary of medical terminology, diagnostic testing procedures, and typical symptoms and transmission routes of a disease [2].

Both of these tools have been designed with the purpose of scaffolding learners throughout diagnostic reasoning to foster adaptive help-seeking behaviors. Rule-based approaches to learner modeling have been used to study help-seeking episodes that result from self-monitoring while solving problems [34].

A series of decision rules determine how help-seeking behaviors are classified as effective or ineffective with the aim of providing appropriate feedback [35, 36]. BioWorld uses a help-seeking model to determine the type and level of hint to deliver a learner when they request a consult based on the expert model. Learners are supported through the analysis of previous help-seeking and problem-solving behaviors.

A series of rules allow the system to analyse user interactions and deliver hints in increasing order of specificity with the aim of gradually supporting learners to engage in the correct path to solving the case. Another aspect of the help-seeking model is to provide learners with supplementary knowledge in relation to diseases, lab-test procedures, and so on. The model encompasses search behaviors in terms of topics searched and pages viewed. We provide an overview of our research findings on help-seeking below.

7 Overview of Empirical Evidence of BioWorld’s Role in Fostering Help-Seeking

Adaptive help-seeking behaviors are conducive for learning and lead to accurate diagnoses. Learners are able to ask for help at any time during their performance using BioWorld to solve patient cases. The following empirical studies address how and when learners ask for help to facilitate their diagnostic reasoning. We determine how students help-seek with the goal of encouraging learners to use help options more effectively.

7.1 Study 1: Using Process Data to Examine Self-Regulatory Behaviors During Clinical Problem Solving Using Technology

This study consisted of 30 students (28 medical and 2 dental students) who were registered at a Canadian University. All students had passed the same basic science course. Sequential pattern mining techniques were used to describe participant self-regulatory processes in BioWorld.

Self-regulation, for the purpose of this study, was defined as help-seeking behavior that was indicated by using the Consult Tool to receive help in the context of solving a case. Sequential pattern mining is a data mining technique that can be used to identify regularly occurring patterns in learning activities and behaviors [37].

In this study the sequential pattern mining technique classifies help requests according to groups (or clusters) of help requests whose sequence of activities occur prior to asking for help. We used this method to interpret the reasons why novices request help by identifying patterns in how they regulate diagnostic reasoning before asking for help from the Consult Tool.

A consult request was defined as clicking on the Consult Tool button with the goal of receiving a hint. For the purposes of this analysis no hints were provided when a student asked for help to observe how students naturally regulate their learning before and after requesting a consult.

Log-files were use to identify the behaviors that occurred before and after requesting help and these behaviors served as the boundaries of our unit of analysis for transcribing and coding the concurrent think-aloud protocols.

Data analyses suggest that students ask for help during the later stages of solving the problem and the amount of consult requests varied across patient cases. On average 83 % of the time taken to solve the case had elapsed (SD = 18.0 %) prior to asking for help. More consult requests made while diagnosing a rare disease such as Pheochromocytoma (52 %) and less requests were made while solving more common diseases, such as Diabetes mellitus Type 1 and Grave’s disease (i.e., 28 and 21 %, respectively).

Consult requests were often preceded by ordering a lab test (72 %) and were followed by either: (a) submitting the final diagnosis (28 %), (b) changing their conviction in regards to their hypotheses (21 %), or (c) reading a topic in the library (14 %).

The results of the sequential pattern mining technique supported 5 distinct categories in the taxonomy of self-regulatory processes. However, the most interesting help-seeking pattern occurred before and after conducting diagnostic tests where the test results were unexpected. Help-seeking also occurred more frequently when reasoning about a rare disease rather than a common one.

These results have important implications for creating more effective forms of adaptive instruction by anticipating when students experience difficulty during reasoning and how to promote adaptive help-seeking in these instances.

For example, targeted prompts can be designed to encourage the appropriate use of help functions. This will help circumvent help-seeking avoidance behaviors. In study 2 below we examine another form of help-seeking, which pertains to knowledge acquisition that is gained by looking up information in the on-line-library.

7.2 Study 2: Supporting Diagnostic Reasoning by Modeling Help-Seeking in BioWorld

It is expected that if learners effectively monitor their learning then they will identify gaps in their knowledge and ask for help to improve performance. In addition to the Consult Tool in BioWorld, learners can address knowledge gaps by visiting the on-line library.

In study 2 we explored search behaviors in the Library Tool in relation to final solution accuracy using data from the same sample described in study 1. We hypothesized that if participants recognised a lacked of prior knowledge, then he or she would conduct a library search, leading to an improved final diagnosis.

In order to analyse learners’ library search behaviors the RapidMiner C4.5 decision tree algorithm [38, 39] was used to split the data set of search behaviors into a tree-like network (Fig. 13.4). The nodes of the decision tree depict topics searched in the library derived from student log file data while solving three patient cases, Amy, Cynthia and Susan. The topics stated in the model are comprised of topics critical for solving the case and topics that highlight learner misconceptions about the nature of the unknown disease.

Fig. 13.4
figure 4

BioWorld library search behavior model

Full size image

The results indicate which search behaviors were predictive of selecting a correct diagnosis. For example, in solving the case of Susan, learners who read about hyperthyroidism had a 100 % (5/5) chance of selecting the correct diagnosis: hyperthyroidism. However, learners who did not read about hyperthyroidism had only a 39 % (25/64) chance of selecting the correct diagnosis.

In other cases, topics searched by learners lead to diagnostic errors, which indicate learner misconceptions. In solving the case of Amy, learners who read about pheochromocytoma had a 0 % (0/5) chance of selecting the correct diagnosis: pheochromocytoma. But, learners who did not read about pheochromocytoma had a 56 % (38/68) chance of selecting the correct diagnosis. Thus reading about pheochromocytoma decreased the likelihood that the student would arrive at the correct diagnosis. These results suggests that learners engage in both effective and ineffective help-seeking by the Library Tool and this can lead to correct or incorrect diagnoses.

The results indicate which search behaviors were predictive of selecting a correct diagnosis. For example, in solving the case of Susan Taylor, learners who read about hyperthyroidism had a 100 % (5/5) chance of selecting the correct diagnosis: hyperthyroidism. However, learners who did not read about hyperthyroidism had only a 39 % (25/64) chance of selecting the correct diagnosis.

In other cases, topics searched by learners lead to diagnostic errors, which indicate learner misconceptions. In solving the case of Amy, learners who read about pheochromocytoma had a 0 % (0/5) chance of selecting the correct diagnosis: pheochromocytoma. However, learners who did not read about pheochromocytoma had a 56 % (38/68) chance of selecting the correct diagnosis.

Thus reading about pheochromocytoma decreased the likelihood that the student would arrive at the correct diagnosis. These results suggests that learners engage in both effective and ineffective help-seeking by the Library Tool and this can lead to correct or incorrect diagnoses, respectively.

Taken together, studies 1 and 2 indicate that learners engage in help-seeking and the nature of this behavior influences diagnostic reasoning and task performance.

Analyzing the patterns of behaviors that precede and proceed help-seeking suggest that help-seeking during diagnostic reasoning can be adaptive or maladaptive such that some of the search behaviors are more likely to yield a correct diagnosis while others are more likely to yield an incorrect diagnosis. The pattern of these behaviors and their respective outcomes are predictable, making it possible to develop individualised support to prompt adaptive help-seeking behaviors and to correct for maladaptive help-seeking thus promoting positive learning outcomes.

Studies 1 and 2 were based on comparing learner models that collected dynamically while problem solving, consisting of process data, and comparing to an expert overlay. Study 3 examines the expert case summary, which can be considered the final outcome of the problem solving, where learners summarize how they diagnosed the patient case.

7.3 Study 3: Case Summary Data-Diagnostic Learning Outcomes

In this study we compare how novice case summaries compare to written case summaries of experts who solve the case in BioWorld. Kellogg [40] highlights the importance of effective written communication in both academic and professional settings. As such, the written case summaries constitute an important exercise for learners. In BioWorld, a typical written case summary highlights the vital signs, relevant symptoms, and lab-tests that were germane to solving the case.

The written case summary is a written justification of a learner’s solution and it is a unique organization of thoughts, actions, and plans that reflect the learner’s knowledge after completing a particular case.

An important challenge towards developing the novice-expert overlay model in BioWorld is the analysis of unstructured data, such as the written case summaries. The unstructured text-based data collected from the written case summaries can provide novel insights into learners’ reasoning after problem-solving that complement user interactions that are analysed in the current version of the novice-expert overlay model.

Thus, the case summaries provide an additional comparative model to guide learners. We have taken first steps for developing a robust novice-expert overlay model using the case summaries written by learners and experts.

Toward this end, we examine the accuracy of commonly used text-mining algorithms in terms of differentiating case summaries written by novices and experts. The resulting text classification model will highlight the key linguistic features that characterize expert knowledge and performance on this task.

The findings stand to inform design guidelines of the revised user model that would be capable of assessing novice case summaries in order to guide instruction necessary to support expertise development in clinical reasoning.

7.4 Study 3: Text Mining Algorithms

Text classification is used in a number of domains [41], including news filtering and organization, document organization and retrieval, opinion mining, and email classification and spam filtering [42]. The advantage of automatic text classification is that it can significantly reduce the cost and time involved in manual categorization.

Text classification algorithms are commonly used in intelligent tutoring systems for assessment purposes [43]. The linguistic features that characterize highly proficient texts are of particular interest to instructional designers [44].

These quality indices can ascertain the aspects of text that are most germane to problem solving, and can provide designers with guidance on a response mechanism when the system detects their deficiency or absence. The text classification problem addressed in this study is the recognition (classification) of case summaries written by novices and experts in BioWorld.

For the purposes of classifying novice and expert written case summaries, the following algorithms were selected on the basis of their frequent use in text classification problems and the ease of implementation in the revised user model [45, 46]: Naïve Bayes (NB), Multinomial Naïve Bayes (MNB), and Sequential Minimal Optimization (SMO). The WEKA [47] toolkit was used to train and compare the text classification algorithms.

WEKA is a comprehensive workbench for machine learning algorithms for data mining tasks, comprising of a myriad of tools for data pre-processing, classification, clustering, and feature selection.

7.5 Study 3: Dataset

The dataset used to train the text classifiers included a total of 74 case summaries written by both novices and experts. The case summaries were labeled as written by either a novice (n = 60) or an expert (n = 14). A sample of a case summary written by a novice is given next: “16 year old girl, previously active and with no significant family history with onset of extreme fatigue, polyurea, polydipsea, difficulty concentrating and 6 lbs weight loss. Lab showed +FBS, and glucose in urine.”

A sample of a case summary written by an expert is shown as follows: “This previously well 16 year old female presents to the ER with abdominal pain and nausea that started today. She complains of 6 months of fatigue limiting her activities of daily life, frequent urination, thirstiness, blurred vision, and weight loss. On exam she was in shock with a blood pressure of 95 systolic and a tachycardia of 100/min, and was mildly tachypneic at 22/min. Her temperature was normal. Investigation revealed a random blood glucose of 18.2, ketones in her blood, serum K of 5.8 with a normal ECG, elevated anion gap and a WBC count of 12. Further investigations for a precipitant of the ketoacidosis (CXR, Urine leukoyte esterase, and abdominal ultrasound) were negative.”

In order to evaluate the efficacy of the classifiers, five datasets were generated to compare several text pre-processing approaches. We manipulated the way in which case summaries were indexed (IDFT, TFTT) and transformed (Lower case, Stemmer, Stopwords) for training the text classifiers (see Table 13.1). In doing so, our evaluation will take into account several alternative approaches to addressing the classification problem.

Table 13.1 Dataset pre-processing steps
Full size table

7.6 Study 3: Results

After pre-processing the data for experimentation, we performed the classifications using the WEKA toolkit. A 10-fold cross-validation was used to evaluate the performance (classification accuracy) of the NB, MNB, and SMO algorithms for each dataset. The results (overall comparison between the classifiers and the breakdown of the classification accuracies) obtained are shown in Table 13.2.

Table 13.2 Text classification accuracy
Full size table

The three classifiers: Naïve Bayes, Multinomial Naïve Bayes, and Sequential Minimal Optimization that we employed in our experiments displayed high accuracies in the classification tasks. The SMO algorithm, which provided the best accuracy of 93.42 %, was found to be the most accurate in terms of distinguishing between the novice and expert case summaries. These findings warrant the use of SMO to revise the current version of the novice-expert overlay system used in BioWorld.

The findings from study 3 suggests that the classifiers employed perform well in the context of differentiating novice-expert written case summaries and supports the idea of leveraging text classification in developing a novice-expert overlay model.

The findings in this study highlight that the SMO algorithm is the most accurate classifier (with the highest accuracy reaching 93.42 %) of novice-expert differences in written case summaries.

The findings could be improved in the future with the collection of more case summaries. Future research will explore the key linguistic features in written case summaries that should serve as quality indices by identifying similar and different terms mentioned in novice and expert case summaries.

In doing so, the findings will apprise domain expert and instructional designers by specifying the remedial steps in the response mechanism that should be taken by the intelligent tutoring system when case summaries are found to be deficient of or lacking the identified quality indices.

This study represents a first step towards developing a novice-expert overlay model component of the expert model, which will help to promote a more expert-like approach to diagnostic reasoning amongst learners.

8 Conclusion

A key dimension of expertise is metacognition, knowing what one knows and does not know [5]. Consequently, our expectation is that competent physicians know what they know and do not know. BioWorld is a TRE designed to foster metacognitive activities of physicians-in-training (novice medical students).

Novices need to know when to ask for help and we need to identify when they reach a learning impasse, when they need more information, and when they have misconceptions. BioWorld was designed to support help-seeking activities in the context of solving virtual patient cases. We presented an underlying model of the cognitive and metacognitive activities that occur in the context of diagnostic/clinical reasoning.

In particular, we documented the three phases of metacognition, forethought, performance and reflection that occur during clinical reasoning with BioWorld.

Theoretical definitions were provided of these important constructs in the context of medical reasoning. Furthermore, we discussed the importance of designing computer-based learning environments that provide cognitive tools to support metacognitive activities throughout the problem solving process.

BioWorld provides a cognitive apprenticeship for novice physicians to deliberately practice their diagnostic reasoning skills with scaffolding. The use of expert overlay models were described in terms of how they support adaptive help seeking and how they support the analysis of help-seeking while learning to diagnose a patient’s disease.

The expert model also was used to examine the process and products of the diagnostic process. Specific advances were described in terms of modeling metacognition through analytical techniques to design, evaluate, and develop expert models by capturing metacognitive activities in problem-solving.

Educational data mining techniques are outlined with the aims of capturing metacognitive activities as they unfold throughout problem solving. These trace methodologies were used to model self-regulatory processes along the trajectory towards competency in diagnostic reasoning. In particular, sequential data mining techniques were used to identify patterns in help seeking behavior in the context of solving cases.

This method revealed that the most comment antecedent to help seeking was receiving a diagnostic test result that did not support a hypothesis about a disease. An analyses of help-seeking in the form of library usage indicated that those students that looked up a disease in the library were more likely to solve the case. Finally, text mining techniques were used to compare expert and novice case summaries in an attempt to assess differences in case solutions.

These techniques were highly accurate and can be used in future studies to assess learner trajectories. These findings lead to insights with respect to designing appropriate scaffolding tools in order to promote higher levels of competence in novices. The empirical findings will be used to inform our future work in the design and delivery of appropriate feedback. We anticipate that the principles for designing metacognitive scaffolding tools in BioWorld can generalize to designing metacognitive scaffolding tools for other computer-based learning environments.