Introduction

Educators are increasingly employing problem-based learning (PBL) as a way to foster higher-order learning outcomes (Lazonder and Harmsen 2016). In this approach, learners are afforded opportunities to solve contextualized, ill-structured problems that are similar to the types of challenges that practitioners experience (Hung 2011; Wang et al. 2013). According to proponents of this instructional strategy, learners not only encounter the relevant concepts central to the problem space, but the contextual nature of the case allows learners to better retain the material (Hmelo-Silver et al. 2007a, b; Jonassen 2011; Kolodner et al. 2004). Moreover, the problem-solving opportunities embedded within PBL also provides opportunities to generate additional competencies such as question generation (Graesser and Olde 2003), causal reasoning (Eseryel et al. 2013), and decision-making (Jonassen and Hung 2008). This strategy also facilitates the transfer of knowledge to new problems when compared with more didactic approaches to instruction (Engle et al. 2011).

One theory that serves as the foundation for PBL includes case-based reasoning (CBR). According to the theory, indices are generated based on the characteristics of a case, including the outcomes and subsequent lessons learned (Schank 1999). An important aspect of CBR is how learners retrieve and reuse the cases; that is, how individuals assign indices (labels) to that experience and apply this knowledge towards similar problems. In PBL, instructors serve as facilitators during student inquiry, which deepens understanding of the ill-structured case presented to the learner (Hmelo-Silver and Barrows 2006; Salinitri et al. 2015). As learners iteratively solve problems with their peers and receive feedback from their instructor, they refine their indices and better retain the case in memory. These iterations not only helps with the meaning-making of the current case, but the case is also available for reuse when similar problems are encountered (Tawfik and Kolodner 2016).

Despite the theoretical benefits of CBR and PBL, others have argued that instructional strategies that employ ill-structured problems are ineffective for novices. Critics cite empirical studies that suggest novices and experts process complex problems in different ways based on their level of experience (Jacobson 2001; Kirschner et al 2006). As it relates to PBL, some studies suggest that learning the content in conjunction with solving complex, ill-structured problems are beyond the cognitive load of novices (Kirschner and van Merriënboer 2013). Others contend that the underlying issue of PBL relates to the level of scaffolding (Hmelo-Silver et al. 2007a, b; Kim et al. 2017). In light of these criticisms, CBR provides strategies for scaffolding students as they solve problems in classroom settings. One method includes case-libraries learning environments (Jonassen 2011), whereby learners access narratives of how experts encountered similar problems. These problematized scaffolds serve as a form of ‘vicarious memory’ for novices to bridge the expertise gap and allow the learner to transfer the lessons learned towards the new problem (Kolodner et al. 2004).

Recent studies have examined the use of case libraries as scaffolds during ill-structured problem-solving; to date, those studies find that cases supports problem presentation (Bennett 2010) and solution generation (Luo et al. 2018) as they are applied to the extant problem. At the same time, research has found significant challenges to case library implementations. In line with expert-novice studies, research finds that learners struggle to identify the relevant aspects of the case that are important (Boshuizen et al. 2012; Schenke and Richland 2017; Tawfik et al. 2019). If learners are unable to retain and reuse the experiences based on important indices, case libraries are ineffective in supporting problem-solving.

The aforementioned studies suggest that learners need scaffolds beyond as-is (static) case libraries to ensure they understand the pertinent aspects of the case. By providing index retrieval supports, learners' problem-solving may improve when utilizing case library learning environments. In other domains, recommendations systems have been used as a way to proactively present users important information for consideration and provide content they might have otherwise overlooked (Musen et al. 2014; Schafer et al. 2007). When applied to case library learning environments, this approach could aide retrieval of relevant cases and thus better provide optimal resources that scaffold learners. However, very few studies explore how to proactively support retrieval and reuse in case library learning environments. Additional research in this area would provide insight into (a) how knowledge structures develop in CBR, (b) how learners access cases, and (c) patterns of use in their iterative problem-solving. To address this gap, we first survey the theory and research of PBL. We then focus our discussion towards CBR theory and the empirical literature regarding case library learning environments. Finally, we present a study about how a case library using a recommendation systems supported learners as they solved ill-structured problems.

Literature Review

Problem-Based Learning

Educational theorists argue that learning is best achieved when situated within authentic contexts (Jonassen 1997; Loyens et al. 2015). Rather than encountering concepts in isolation, a learner instead understands its relevance within a broader phenomenon. Learners must therefore understand the problem and iteratively generate a solution in light of its structuredness, dynamicity, and domain specificity (Jonassen and Hung 2008). By self-directing their learning, learners garner additional competencies such as causal reasoning (Eseryel et al. 2013), question generation (Graesser and Olde 2003), decision-making, and argumentation (Crowell and Kuhn 2014; Ju and Choi 2017).

Classroom approaches to problem-solving have been systematized in the form of problem-based learning (PBL), most notably by one of its early adopters, McMaster’s University (Woods et al. 1997). Because of the unsatisfactory clinical performance of their medical students, educators sought out instructional strategies that went beyond rote memorization and fragmented domain knowledge. Educators thus migrated towards PBL, which required medical students to collaboratively solve ill-structured clinical cases (Barrows and Tamblyn 1980; Jeong and Hmelo-Silver 2016). As part of this process, learners justified their proposed solutions to their peers given the unique perspectives (Ju and Choi 2017). Finally, the instructor facilitated student learning by providing scaffolding and engaging in collective reflection to repair errors in understanding (Ertmer 2005; Hmelo-Silver and Barrows 2006). Although originating in medical education, PBL has since seen expanded implementation in other domains where practitioners encounter ill-structured problems, such as engineering (Bédard et al. 2012), law (Wijnen et al. 2017), and teacher education (Ertmer et al. 2009). Research finds that learners are able to better apply their learning (Kim et al., 2017), understand principles (Belland et al. 2017), and acquire conceptual knowledge (Walker and Leary 2009) in PBL when compared with more didactic forms of learning.

Developing Knowledge Structures through Problem-Based Learning and Case-Based Reasoning

The intended learning benefits of PBL can be described through the lens of case-based reasoning (CBR). CBR argues that experiences serve as the foundation for knowledge structures (KS); that is, the manner in which an individual organizes the relationships of information units/concepts in memory is influenced by one's experiences (Clariana et al. 2014; Jonassen et al. 1993). When an individual is required to solve a new problem, s/he will access a set of cases (“case library”) that s/he deems as relevant based on her/his similarity assessment (Xiong 2011). If the case is germane to the extant problem, the individual will reuse the prior experience as a template. This is used both to understand the scope of the problem space and proffer ideas to resolve the issue (Kolodner et al. 2004; Schank 1999). If the reasoner does not identify a case as relevant, s/he will then attempt to resolve the case using alternative means and then retain the new experience as an additional case within their internal case library (Tawfik and Kolodner 2016). As an individual gains additional expertise, “they tend to organize their knowledge around encountered cases and experiences, which may result in more elaborated and coherently organized knowledge structures” (Lachner et al. 2016). Individuals construct knowledge structures based on their understanding of multiple cases, which are organized within a case library over time (Boshuizen et al. 2012).

Since knowledge is semantically organized in a relational manner (Ausubel 1963), researchers believe that the organization of an individual’s knowledge reflects their depth of understanding. The organization also has implications for the ease and flexibility with which they are able to retrieve and reuse their knowledge (Hmelo-Silver et al. 2007a, b; Kolodner 1991). Riesbeck and Schank (2013) further assert that “human memory depends upon good methods of labeling cases so that they can be retrieved when needed” (p. 7). That said, additional research finds that experts and novices organize their KSs and understand their experiences in markedly different ways, which impact retrieval. Experts may establish well-organized KS that is easily retrievable, while novices establish loosely linked KS in their mental representation that is more difficult to access from long-term memory (Herr 2008). In a comparison of how expert and novice teachers assessed classroom management situations, Wolf et al. (2016) found that expert teachers were more likely to focus on important contextual issues and connect them to broader classroom management principles. Alternatively, novice teachers’ attention was more fragmented and tended to focus on superficial features. The propensity of novices to focus on superficial features has been documented across other domains. For example, Hmelo-Silver et al. (2007a, b) similarly found that novices focused on the less salient features of a context. Beyond just recognition of superficial characteristics, the study also showed that experts were able to engage in more causal reasoning between the identified concepts (Hmelo-Silver et al. 2007a, b). Others find that experts’ KSs include systems-level thinking and a robust set of cases, whereas novice KSs are ill-defined and less connected (Snapir et al. 2017).

Supporting CBR and PBL Through Case Library Learning Environments

Studies regarding how individuals organize their KS and case libraries entail multiple implications for PBL. When learners attend to the relevant features of a problem posed in PBL, they better index the memory and later retrieve it when similar experience are encountered. Specifically, learners are better able to diagnose a problem and generate a solution with more advanced KS and refined indices. However, as noted earlier, CBR suggests that experts and novices possess different KS based on their understanding of the problem space and varying levels of experience (Schank 1999). Critics cite similar studies of cognitive load as evidence for why PBL and case curricula are problematic for novices within classroom settings (Kirschner et al. 2006). Along those lines, Hung (2011) argued that there is a theory vs. reality problem in terms of PBL implementation.

In terms of supporting PBL, Kolodner (1991) suggested that there are two specific approaches for CBR to support PBL: problem-solving style and interpretive style. In terms of the former, an individual will reference the prior set of cases and use them to generate ideas about how to solve the existing problem. In this approach, cases provide “almost-right” solutions (p. 55) and serve as a mechanism to diagnose the situation and elucidate potential outcomes for the selected solution. In terms of interpretative style, the reasoner uses cases for situation classification, argumentation, and justification. CBR further suggests that these approaches can be leveraged to reconcile experience gaps between experts and novices. Specifically, one way to address the lack of expertise is through case library learning environments, which serves as a form of vicarious memory (Jonassen 2011; Kolodner et al. 2004). As learners engage in problem-solving, a representative database of cases allows learners to see how experts encountered similar problems (Boshuizen et al. 2012; Schenke and Richland 2017). A well-curated case library scaffolds inquiry into the latent and salient variables that should be considered during problem-solving. Through these narratives, learners engage in meaning-making about the important indices and also reflect on ways in which they are transferred to the primary lesson being solved.

To date, there have been various studies regarding how case libraries scaffold ill-structured problem-solving. In an early study, Jonassen and Hernandez-Serrano (2003) found that participants in a case library condition outperformed the control condition on measurements of prediction, inferences, and explanations. Case libraries as exemplars have also been documented to expose learners to best practices and subsequently refine learner’s understanding of a phenomenon during reflection (Kim and Hannafin 2011). Similarly, Bennett (2010) found that case libraries presented to learners were beneficial in terms of reflection and increased awareness of domain principles within the problem space. She further reasoned “that having access to more than one case allowed the learners to understand how issues can manifest differently in different situations and thereby compare and contrast issues to gain a deeper understanding.” (p. 472). Additional studies have found that the contextualized nature of case libraries helped to (a) connect concepts with practice and (b) conceptualize the problem space in various ways (Luo et al. 2018). In doing so, related research suggests that the case library affords opportunities to understand contextual variants (Boshuizen et al. 2012; Wolff et al. 2016) and promote cognitive flexibility (Valentine and Kopcha 2016).

Despite the positive results of case libraries, the literature also suggests that CBR also presents other challenges to learners during problem-solving. Not only must they engage in meaning-making about their experience, but they must be able to retrieve and reuse cases when new problems are presented. While it is generally agreed that case library learning environments serve as a way to frame the problem, more recent research has explored the degree to which the design of a case library plays a role in supporting transfer and problem-solving. For example, a variety of studies have compared the effects of text-based and video-based cases on problem-solving (Gartmeier et al. 2015; Lajoie et al. 2014). When the design of case libraries explicitly encourages contrasts across the narratives, learners improve learning outcomes (Balslev et al. 2005; Gartmeier et al. 2015) and better engage in collaborative inquiry (Lajoie et al. 2014). More recently, studies have also explored the effects of failure cases and found that learners were able to better engage in conceptual understanding (Asterhan and Dotan 2018; Lin-Siegler et al. 2016), development of alternative perspective (Tawfik and Jonassen 2013), and reflection (Cattaneo and Boldrini 2017). Collectively, the research suggests that the design of the case impacts learning from the narrative and ultimate transfer.

Research Questions

Case-based reasoning theory suggests that case libraries can be used as a problematized scaffold that helps learners understand the complexities of the problem space and generate KS for problem-solving. Researchers have identified, “the development of such problem-oriented knowledge structures is an important reason for teaching with clinical cases, as is done in problem-based curricula” (Boshuizen et al. 2012, p. 758). To date, research studies find that case library learning environment scaffold students’ conceptualization of the case and frame the solution generation during PBL. At the same time, studies regularly show that experts and novices perceive cases in markedly different ways and focus on different aspects of the experience; that is, novices fail to identify pertinent aspects of a case (Boshuizen et al. 2012; Schenke and Richland 2017; Tawfik et al. 2019) or may misappropriate case relevancy during problem-solving (Alfieri et al. 2013; Tawfik et al. 2019). Similar research shows that learners often fail to fully employ the scaffolds embedded within the case library (Gartmeier et al. 2015; Schenke and Richland 2017). If students do not adequately recognize or utilize the critical indices within the case, they will not be able to retain the memory and later retrieve it when needed.

Given the literature on how expertise develops, one could argue that case libraries require retrieval supports to better scaffold problem-solving. One way to do this is through a recommendation system that suggests optimal cases for learners to leverage during PBL on a set of pre-defined criteria. In doing so, learners are not only able to better retrieve the optimal case, but they are better able to reuse the experience to resolve the primary problem presented in PBL. While the literature has looked at post-hoc outcomes of using cases (e.g., effects on argumentation), there is very little data about the retrieval and reuse patterns when learning with case libraries. To address this gap, this study analyzed argumentation scores in conjunction with learning analytics to obtain a comprehensive picture of how participants employed varying designs of case libraries. Specifically, we pose the following research question to understand students’ fine-grained behaviors and interaction patterns that play a role in their problem-solving process:

  1. 1.

    To what degree are the knowledge structures impacted when students are scaffolded in PBL only, scaffolded with static case libraries, or scaffolded with recommendation-based case library systems?

  2. 2.

    What retrieval patterns emerge as learners engage in a recommendation-based case library system?

Methods

Participants

Participants for the current study consisted of students enrolled in an undergraduate marketing class entitled “Sales Management”; topics included sales strategies, workforce development, and sales/client relationships. The selection criteria for this study was based on enrollment within the section of the course offered in the Spring 2018 semester; therefore, 100% of the sample was derived from this class. The specific course was offered in a large, research-focused institution located in the Midwest portion of the United States. In general, the class was offered bi-weekly in a face-to-face session with an experienced marketing instructor with over ten years of teaching with PBL. Participants were undergraduate junior level (ages 19–22 years old) and enrolled in the Sales Management class as part of their marketing majors. A total of 104 students took part in the study and all completed the institutional review board (IRB) consent form.

Procedures

Although students were asked to complete the final assignment individually, they were allowed to work in groups of 4–5 individuals, as prescribed by PBL (Barrows and Tamblyn 1980). Prior to the study, the Microsoft Excel randomization function was used to assign participants into (a) conditions and then (b) groups. In total, there were 25 groups. The conditions were as follows: PBL only (N = 31), PBL with static case library (N = 32), and PBL with recommendation system case library (N = 41). Groups were given their own dedicated space to access the materials (PBL problem to solve and case library) on the learning management system to avoid potential confounds from other conditions. Only the lead researcher had access to the condition assignments. As such, the course instructor was unaware of the participant treatment assignments.

As participants of the conditions logged in to their learning management system, they were posed with an ill-structured, decision-making problem (‘Nick’s Dilemma’) and offered a dedicated discussion board thread on the topic. Participants were given two days to read through the problem individually and, if applicable, their assigned case library. The case library was embedded directly within the main problem-to-solve; therefore, all participants only needed to access the link assigned to the condition (see Figs. 1 and 4 below).

Fig. 1
figure 1

PBL problem with static case library

Full size image

Over the course of the next two weeks, participants were required to discuss with their peers about how to resolve the main PBL case. After the initial two days of reflection, each group member was expected to daily post at least a 100-word response (roughly one paragraph) for a total of ten days. In terms of the two day lead time, this decision was made so that participants would not feel rushed to post; rather, they would have time to reflect on the problem, read the related resources, and generate ideas about how to resolve the issue. In terms of the activity, participants were advised that regular posts would help garner new questions and address emergent issues among their peers. Participants were further instructed to avoid simple phrases (“I agree” or “Good Idea”), but instead justify ideas and elaborate on their peers’ posts as they added to the discussion. It was suggested that each member’s goal was to present issues, dilemmas, and answers related to the Nick's Dilemma case so that the group could examine as many issues as possible. As it relates to leveraging the scaffolds, those in the case library conditions (PBL with static case library, PBL with recommendation system case library) were told that the cases provided relevant strategies to solve the main problem to solve. Alternatively, participants in the PBL-only condition were encouraged to engage in their own inquiry about how to address the case.

At the end of the activity, participants individually submitted a multifaceted argumentation about how they would resolve the issue. Specifically, the argumentation text was required to detail the following aspects: initial ideas to solve the problem, counterarguments, and rebuttals. Argumentation was selected because it is reflective of how individuals articulate the problem space (Hmelo-Silver 2013), generate causality (Ju and Choi 2017), assess evidence (Crowell and Kuhn 2014), and justify their solutions (Jonassen and Cho 2011). The elements for the argumentation essay was based on Nussbaum and Schraw’s (2007) assertion that different aspects of argumentation reflect distinct aspects of understanding. The initial argument was defined as the participant’s initial stance on how to solve the problem. The counterargument required learners to articulate alternative perspectives that others might have about the issue. Finally, rebuttals could refute the counterclaim (refutation strategy), present a compromise between the initial argument and counterargument (synthesize), or suggest one side was more valid than the other approach (weighing).

Materials

Ill-Structured Problem/Main Problem to Solve

Participants were tasked with solving an ill-structured problem entitled “Nick’s Dilemma.” In the problem, participants read about how Nick and his boss, Sheila, needed to reconstruct their medical device sales management team in light of recent turnover and ever-increasing market competition within the region. The scenario presents the following solutions to Nick and Sheila: hire an external candidate, hire an internal candidate, and revisit the advertisement for the position. The internal candidate, Terry, is a loyal employee who has been with the company for over ten years and holds a position within their internal help desk. While she has a track record of loyalty within the company and high marks from her supervisors, it is unclear if her skillset would translate into one that must proactively manage client/customer relations. The lack of direct experience would also require additional training costs, which are problematic given the rapid rise of competitors and subsequent market share losses.

The scenario also describes an alternative candidate, Lewis, who has direct prior experience with external sales, which would potentially reduce the training cost and thus better address the competition concerns. Although he has positive reviews from previous employers, it is revealed that he failed to disclose a driving under the influence citation from years ago. Finally, the participants read about how a competitor firm has closed within the metropolitan area, which results in an influx of available candidates. However, the advertisement requires additional time and money that the company may not be able to afford.

Case Library Learning Environments

Participants were randomly assigned to three different conditions. For each of the different conditions, participants were given a main ill-structured problem (“Nick’s Dilemma”) to solve while the instructor facilitated peer work. In line with PBL (Ertmer 2005; Hmelo-Silver and Barrows 2006), the instructor was focused on the facilitation of knowledge construction among the students (e.g.—probing questions) rather than explicit instructions about the specific concepts or ways to solve the problem. As noted earlier, those randomly assigned to the first condition (PBL only) only had access to the main problem to solve, while the instructor provided support as needed. In the case library conditions (PBL with static case library; PBL with recommendation system case library), participants were able to access five cases that detailed how other experts encountered similar issues. To design the narratives with the case library, the lead research used the CBR instructional design protocol (Jonassen and Hernandez-Serrano 2002) to facilitate the interview with the SME (see Tawfik et al. 2012 for further detail).

For the PBL with static case library condition, the five cases were embedded at strategic points based on the primary lessons learned from the case; hence, the the hyperlink text was designed to represent an initial index associated with the case. The placement of the links was largely driven by the SME’s response to the following question within the CBR instructional design protocol: “Which features of the problem situation were most important and what was the relationship between its parts?” (Jonassen and Hernandez-Serrano 2002, p. 79). As such, there is one hyperlink that relates to each of the five related cases (see Fig. 1). For example, clicking on the link “technical expertise and social skills” navigates to a case entitled “Chris' Choice”.  This case details how Chris prioritizes his decision-making based on a set of core competencies needed for a specific job and opportunities for mentorship. At the end of each case, the participants were encouraged to think about (a) the main points and (b) how the presented case related back to the main problem to solve. To further support learning experience design, learners are also able to access each of the five cases in a header that appears at the top of the screen.

The third condition (PBL with recommendation system case library) similarly presents the links embedded within the main problem to solve, but also utilizes a search feature combined with a recommendation algorithm. Specifically, the system is designed such that (a) learners are recommended specific cases based on how experts weighed the relative importance of an index (search term) in solving the main problem to solve and (b) the degree to which those same terms are weighted in the related cases. As noted earlier, the participants were only told that cases included relevant information to solve the problem. As to not bias their interaction, the participants were not prescribed how to leverage the recommendation system or detailed information about how to interpret the scores.

To develop the retrieval algorithm, domain experts (N = 5) were asked to read the original business main problem to solve (“Nick’s Dilemma”) and rank their perceived relevant indices for each of the relate cases (e.g., market share losses = 5, employee morale = 4). Each expert had over ten years of experience with sales management. To create the recommendation system, s/he met with a member of the research team and documented their rankings in a think-aloud manner. The researcher would only intervene if the domain expert had questions related to navigation of the site or the task, but offered no advisement on the rankings. Each interview lasted for approximately one hour. The rankings were recorded in a spreadsheet and later member-checked by the experts.

The rankings were incorporated into a hybrid approach that uses expert filtering and content-based filtering (Schafer et al. 2007) to generate an initial set of ratings that drive the recommendation algorithm. For example, the case “Chris’ Choice” details how a well-educated, external employee with experience might be able to mentor some junior-level sales associates. To guide the learner’s decision-making, the case weights indexes or labels with themes such as “external hire” and “mentorship” as highly relevant. At the same time, other indices such as “MBA” are prominent within the narrative, but not deemed relevant to solving the main problem. This expert-driven approach helps avoid the common ‘cold start’ problem in the early stage of these systems (Schein et al. 2002) when learners do not have enough information for the initial set of recommendations (Balabanović and Shoham 1997).

To use the system, the learner searches a term and, based on the set of related indices germane to each case and its computed relevance to the main problem, the algorithm recommends the optimal case based on the expert rankings. Specifically, the expert scores an index (j), which provides a j-by-k matrix for each narrative in the case library (see Fig. 2). The recommendation system uses the mean of expert scores to condense each index on a single value. The main problem to solve (b) is then compared with the case (c). As such, the similarity of the main problem (b) and any case (c) is the difference between the mean expert scores (j) for each case normalized as the range of values in the scoring system. The algorithm is then based on the distance of an index from the problem to solve (b) to each one in the other cases and then returned and ranked to the user (see Fig. 3). Therefore, the recommendations are based not only on the indices within the case or the prevalence of the search term, but the expert’s calculated perceived importance for each index and case.

Fig. 2
figure 2

Example of expert indices and expert rankings

Full size image
Fig. 3
figure 3

Retrieval algorithm

Full size image

The intended interaction with the recommendation system is largely driven in two ways: the top links and the search terms (see Fig. 4). The top links are similar to the static case library in that they are readily accessible on the top menu, which is designed to support self-directed learning and fluid navigation within the systems. However, participants in the recommendation system condition are also able to access a search box that allows them to input search terms at any point during their problem-solving process. In addition to free text search, a set of pre-populated search terms is also provided, which allows the participants to select the terms they deem are relevant. Based on the algorithm, the participant sees an output and relevancy score of the associated cases and the degree to which experts deemed that case index as important to the main problem to solve. For example, they see that the term “Internal Recruitment is 75% relevant to Janice’s Transition”, “Internal Recruitment is 70% relevant to Holly’s Choice”, and so on (see Fig. 5). Given that the algorithm helps drive retrieval, the results are designed to cue the learner about the degree to which they might reuse the cases towards the main problem to solve.

Fig. 4
figure 4

Search based on an index

Full size image
Fig. 5
figure 5

Search retrieval based on index

Full size image

Measurements

Analytics for Investigating Students’ Knowledge Structures in Argumentation

To reveal the KS difference between conditions (RQ1), a text network visualization system (Graphical Interface of Knowledge Structure; GIKS) was employed to derive and measure KS from participants’ argumentation essays. The GIKS is designed to capture important concepts in a text and then visually represent the relationships between the selected key concepts as graph-theoretic psychometric network graphs (Pathfinder networks), which are hypothesized to represent the salient KS related to the text content (Kim 2017). The GIKS has been deemed valid and reliable through various investigations in diverse domains to measure KS from texts, such as extracting text structures from narrative and expository texts (Clariana et al. 2014), identifying knowledge transfers from first language to second language writing (Kim and Clariana 2015), and capturing different knowledge patterns in texts across languages (Kim 2017).

Using the GIKS, this investigation converted all of the participants’ essays (initial, counter, rebuttal) and experts’ comparison essays into KS network graphs (see Fig. 6).The similarity of the participant KS with the expert KS were measured as percent overlap—calculated as the “links in common” for two KSs divided by the average number of links in the two KSs. For example, the percent overlap of the expert and PBL with static case library patricipant, as shown in Fig. 6, is calculated as 4/((11 + 5)/2) = 0.50 (50% overlap) where four is the number of links in common (see for detail, Clariana et al. 2014). Note that the GIKS system automatically generates the similarity value (Table 1).

Table 1 Experiment condition and activities
Full size table
Fig. 6
figure 6

Example knowledge structure (KS) network graphs derived from the rebuttal argumentation from an expert (top left), PBL only condition (top right), PBL with static case library condition (bottom right), and PBL with recommendation system case library condition (bottom left). Note: A student’s KS consists of a highlighted network graph showing the similarity and difference compared to the expert KS; yellow indicates ‘missing’ links/nodes and red indicates ‘incorrect’ links/nodes (Color figure online)

Full size image

Analytics for Investigating Students’ System Usage Patterns

To better understand how learners engaged the recommendation system scaffold during their problem-solving (RQ2), the study employed a hierarchical cluster and a frequent sequential pattern analysis using the log data recorded within the recommendation system. The purpose of these analyses was to explore how the participants assigned to the recommendations condition interacted with the system and their iterative retrieval strategies. This research employed hierarchical clustering because it is especially suited for high dimensional and small sample size problems (Farrelly et al. 2017). The log dataset analyzed in this study contained information regarding whether and when each participant viewed each case. The hierarchical clustering approach links each sample incrementally, creating a dendrogram, which is similar to a tree-like structure (Langfelder et al. 2008). As opposed to other clustering methods that classify adjacent samples using a designated centroid or medoids, hierarchical clustering begins with each sample representing its own cluster and merges the two most similar clusters. Prior to clustering, this study pre-processed the log data to compute access frequency per each case and calculated the Euclidean distance between samples (i.e., participants) in the frequency space. We then compared the coefficients obtained from four different methods that could be used to measure cluster distance (see Table 2). Based on the comparison result, the study employed Ward’s method that links two clusters in a way that minimizes the increased sum of squares after each iteration (Kuiper and Fisher 1975).

Table 2 Results of clustering methods comparison
Full size table

Frequent Sequential Pattern Analysis

In addition to the hierarchical clustering analysis, this study employed a sequential pattern analysis using the cSPADE algorithm (Exarchos et al. 2008) to better understand learner behavior as they solved the problem using the recommendation system. The algorithm scans a vertical data format consisting of individual IDs, time-stamps, and a list of objects entered in a sequential manner and then identifies important sequences through recursive decompositions of subsequences into smaller subsequences. We employed this algorithm to identify differences in frequent sequential patterns between students classified into different clusters. We set the minimum support threshold to 0.2; as a result, we only detected subsequences that appeared 20% of possible sequences. A support indicates how frequently a particular sequence appears out of all sequences existing in a given dataset. A support is used to find prominent subsequences, thus limiting the number of unimportant sequences.

Results

Analytics for Investigating Students’ Knowledge Structures in Argumentation

To answer the first research question, we compared the three conditions on various elements of argumentation. Specifically, all student essays and expert essays were converted to KS network graphs using GIKS; then the student and expert KS network graphs were compared using the network similarity. A one-way MANOVA was run to determine the difference of the conditions (PBL only, PBL with static case library, PBL with recommendation system case library) on the argumentation performance (initial, counter, rebuttal). Preliminary assumption checking revealed that data was normally distributed as assessed by the Shapiro–Wilk test (p > 0.05); there were no univariate or multivariate outliers, as assessed by boxplot and Mahalanobis distance (p > 0.05) respectively; there were linear relationships, as assessed by scatterplot; no multicollinearity was identified (r = 0.381, p = 0.002); and there was homogeneity of variance–covariance matrices, as assessed by Box's M test (p = 0.009).

The differences between the condition on the combined dependent variables was statistically significant: F(4, 112) = 17.675, p < 0.001; Wilks' Λ = 0.376; partial η2 = 0.387. Follow-up univariate ANOVAs showed that initial (F(2, 57) = 30.875, p < 0.001; partial η2 = 0.520), counter (F(2, 57) = 14.295, p < 0.001; partial η2 = 0.334.), and rebuttal scores (F(2, 57) = 17.283, p < 0.001, partial η2 = 0.390) were significantly different between the condition when using a Bonferroni adjusted α level of 0.025. Tukey post-hoc tests showed that, for initial scores, participants from both case library conditions had significantly higher similarity scores with the expert argumentation essay (0.45 to 0.52) compared to the PBL condition. However, for counterargument scores, the recommendation-based case library group had significantly lower similarity with the expert (0.30) compared to other conditions. For rebuttal scores, PBL with recommendation system case library condition scored the highest (0.63), followed by the PBL with static case library condition (0.47), and lastly PBL-only condition (0.38). Collectively, the results suggest that the case library conditions outperformed the other conditions on the simpler task. As it relates to the recommendation system, those with access to this scaffold outperformed participants on rebuttal scores, but not counterargument scores. The results are further summarized in Table 3.

Table 3 The average similarity of participants’ KS network graphs in each group with the expert KS network graphs (as % overlap)
Full size table

Clustering Analysis

This study adopted multiple methods to determine the number of clusters, including a silhouette analysis (Reynolds et al. 2006) and cluster dendrogram (Forina et al. 2002). In addition, we used NbClust, an R package that uses 30 indices to determine the number of clusters. All methods proposed two as the best number of clusters.

The profiles of students classified into the two clusters can be summarized in multiple ways (see Fig. 7). The first cluster (N = 22) was characterized by students’ infrequent view of the cases; we labeled the cluster as inactive retrieval students. In contrast, the second cluster (N = 19) showed a much more frequent view of all the cases, which was labeled as iterative retrieval students.

Fig. 7
figure 7

Case library cluster patterns for inactive retrieval students (bottom) and iterative retrieval students (top)

Full size image

Frequent Sequential Pattern Analysis

The total number of sequences was 1,171; therefore, this number was used as a denominator to compute a support for each sequence. By setting the minimum support to 0.2, we only included sequences that were observed more than 235 times in the dataset. According to the result, the number of identified rules revealed that the iterative retrieval students interacted with the cases more actively than inactive students. We found that the supports of all cases were 1.00, which indicates every iterative retrieval student viewed all the cases. Out of eight subsequences that had a support greater than 0.2, five were not adjacent cases on the menu. Therefore we can infer that those students explored cases on the recommendation system rather than simply viewing the cases as they were ordered. Table 4 shows the identified sequences.

Table 4 Subsequences identified from active retrieval students
Full size table

In contrast, any inactive retrieval student did not view all cases as indicated in Table 5, showing that the supports (i.e., the fraction of all sequences) of all cases except “Nick” were less than 1. It should also be noted that no sequence was found with the minimum support of 0.2, but only single cases. This result indicates that inactive students tended to view fewer cases than the iterative retrieval students for each access. Furthermore, every inactive retrieval student viewed the main problem to solve (“Nick”), which was the first one in the menu, and some of them seemed to have left the website after viewing the case.

Table 5 Frequent cases identified from inactive retrieval students
Full size table

Discussion

According to CBR, learning is driven by how an individual understands an experience, assigns relevant indices, and later organizes that knowledge within memory (Jonassen 2011; Kolodner 1997; Schank 1999). However, critics contend that students' simultaneous responsibility for inquiry and knowledge acquisition in PBL precludes the development of robust knowledge structures (Kirschner et al. 2006). Given the literature regarding how knowledge structures are organized within memory based on experience (Tang and Clariana 2017), researchers have attempted to design scaffolds that address the gap between novices and experts. In contrast with other scaffolds, CBR theorists argue that case libraries uniquely serves as a form of vicarious memory, broadening a learner’s set of experiences from which to retrieve when solving the problems posed in PBL (Jonassen 2011; Tawfik and Kolodner 2016). While research has shown positive effects of case libraries to scaffold PBL, additional studies show that novices fail to identify the essential aspects of the case (Hmelo-Silver et al. 2007a, b; Jacobson 2001; Wolff et al. 2016), which complicates transfer and knowledge structure development. This study thus adds to the existing literature about PBL, CBR, and scaffolds in terms of the following: (a) how case libraries scaffold PBL, (b) what, if any, supports are needed by novices to retrieve the optimal case, and (c) retrieval strategies employed when supported with a recommendation system.

Discussion of Knowledge Structures and Design of Case Library Learning Environments

The first research question focused on the degree to which knowledge structures are impacted when scaffolded through case libraries. Both case library conditions (static case library and recommendation system case library) outperformed the PBL-only condition on initial arguments. Potentially, the narrative format within the case library helped the learner to better contextualize the variables within the problem space in contrast with learners who received no scaffolds. Since research shows that the opening stance is the least challenging (Crowell and Kuhn 2014; Jonassen and Cho 2011), it may be that the contextualization in both case library conditions equally supported their ability to generate an initial understanding and ensuing justification for their solution. From a cognitive flexibility theory perspective (Jacobson and Spiro 1995), it is possible that the diverse narratives allowed learners to see the indices in various contexts, which expanded learners’ conceptualization of the problem space and better prepared students for transfer. The results adds to prior literature which finds that scaffolds are an essential element of PBL and that cases-as-scaffolds play a key role in subsequent learning outcomes (Asterhan and Dotan 2018; Bennett 2010; Ertmer and Koehler 2018; Hernandez-Serrano and Jonassen 2003; Schenke and Richland 2017), especially for early stages of problem representation and solution generation.

Another noteworthy finding is that the PBL with recommendation system case library performed the lowest on measures of counterargument. Research suggests that counterarguments are one of the most challenging forms of reasoning for novices because of ‘my-side bias’ tendencies (Hemberger et al. 2017; Kuhn 1993; Newell et al. 2011). To overcome this siloed perspective, case libraries can provide multiple situations whereby learners can understand how the concepts play out within a context. However, it is up to the learner to transfer the principles to the main problem to solve after they have compared and contrasted the stories within the case library. It may be that this abstraction across cases is challenging when developing an alternative perspectives, which resulted in the lower scores from the PBL with recommendation system case library condition. Along those lines, the ‘my-side bias’ tendency may have also been subject to the design of the recommendation system. In this condition, participants were focused on comparing and contrasting different opinions within the higher-ranked cases, thus possibly limiting the scope of narratives they accessed. By focusing on the higher ranked cases, they may have failed to consider additional cases that presented alternative perspectives. While the recommendation system presumably introduces an array of resources an individual might not otherwise consider, it may be that the ratings caused the students to narrow their search. In doing so, students may have inadvertently limited their retrieval and reuse to a few cases with high ratings, and possibly limited inquiry and iterative problem-solving.

The results may also be a byproduct of how the PBL with static case library was designed. In the early stages of the main problem to solve, the characters lay out a case for why they favor an internal candidate and learners are linked to related cases that talk about the benefits of different variables, such as employee morale and mentorship. As learners read the narrative, they are introduced to another candidate and provided hyperlinks to related cases about the benefits of external hires and ways in which the right individual can decrease training costs. By being exposed to one candidate and then an alternative candidate in the main problem to solve, this may have served as an inadvertent structure for their argumentation. Learners potentially aligned their argumentation essays to this initial and counterargument flow, which may explain the differing results between the two conditions.

This study also found that learners in the PBL with recommendation system case library outperformed learners on measurements of rebuttal scores, which is defined as their ability to evaluate and synthesize evidence from multiple perspectives (Crowell and Kuhn 2014). These results have both theoretical and practical implications for CBR and PBL. In terms of scaffolding strategies, Reiser (2004) asserts that “problematizing is a process of focusing attention along productive dimensions, but naturally, it does not guarantee that this focus of attention will lead to productive results” (p. 299). In terms of CBR design, the findings between the different case library conditions suggest it may be overly simplistic to assume that cases in isolation can serve as vicarious memory and bridge the experiential gap. In the context of this study, the recommendation system scored the degree to which experts deemed the case as beneficial (e.g., Case 1 has 86% relevancy score on the main problem to solve; Case 2 has 74% relevancy score on the main problem to solve) in solving the main problem. Indeed, the presentation of the indices in the recommendation system condition may serve as an additional scaffold to understand the essential aspects of the case. As shown by the learning analytics, it is possible the recommendation system’s search functionality alerted the learners about which indices to prioritize during their problem-solving. By encountering high-scoring cases with each search, the various narratives may have encouraged a more balanced understanding of the relevant problem space, thus leading to higher rebuttal scores. These results thus add to the growing body of literature that describes how the design of the case library impacts learning outcomes (Gartmeier et al. 2015; Lajoie et al. 2014; Tawfik 2017).

Discussion of Analytics for Investigating Students’ System Usage Patterns

In addition to knowledge structure development, this study also explored how learners iteratively retrieved the cases when scaffolded with a recommendation system (Research Question 2). In prior years, measurements of case libraries consisted of qualitative reflection or analysis of post-hoc learning artifacts such as concept maps (Fitzgerald et al. 2011; Hmelo-Silver et al. 2007a, b; Lajoie et al. 2014) or essays (Lin-Siegler et al. 2015; Tawfik and Jonassen 2013). More recently, Wang et al. (2013) highlighted a gap in that “many existing studies in the field have tackled problem-solving and knowledge construction separately, failing to see them as an integrated two-way process” (p. 294). Methodologies to address this gap often included think-aloud analysis of a case (Ertmer et al. 2008; Jacobson 2001) and eye-tracking (Wolff et al. 2016). This study uniquely described how learning analytics could be used to better understand patterns in problem-solving; namely, retrieval patterns that novices employ when using case libary recommendation systems during their problem-solving. Although participants who used the system yielded better learning outcomes in rebuttals than the other two groups, they were not equally engaged in studying cases. From a theoretical perspective, the gap in case-based reasoning patterns between experts and novices may not be fully addressed by using recommendation systems alone. Analytics of participants’ usage patterns in the system thus provides direction for improving the case recommendation system. This finding suggests a need for additional instructional supports to encourage students’ iterative use of the recommendation systems and sustained retrieval of cases during problem-solving. Examples might include utilizing students’ learning traces (e.g., log data) to alert inactive students as attempted in studies on early warning learning management systems (Jokhan et al. 2018) or learning analytics dashboards (Kim et al. 2016).

Conclusion and Future Studies

Studies show that PBL approaches engender higher-order learning outcomes when compared with more didactic, lecture-based approaches to education (Hartling et al. 2010; Lazonder and Harmsen 2016; Walker and Leary 2009). A key part of these results is the degree to which learners are properly scaffolded to overcome their gaps in understanding (Kim et al. 2017). Theorists argue that cases libraries may serve as a source of vicarious memory and uniquely scaffold the expert-novice gap often cited by critics of PBL (Jonassen 2011; Tawfik and Kolodner 2016). However, studies find that experts and novices tend to identify indices and aspects of the case in vastly different ways (Hmelo-Silver et al. 2007a, b; Jacobson 2001; Wolff et al. 2016). To address this gap, this study explored the impact of index retrieval scaffolds during problem-solving (RQ1) and the patterns of inquiry when accessing a recommendation system case library (RQ 2). The results of the rebuttal scores suggest that the recommendation may support a more balanced understanding of the problem space. However, the lower counterargument scores lead to questions about whether the recommendation system was too prescriptive and limited exploration of alternative perspectives. Finally, the learning analytics provided interesting insight regarding the different profiles of CBR retrieval pattern as learners employed recommended case libraries.

While the study provides some understanding regarding scaffolding for index generation and retrieval, there are opportunities to address the limitations and contextual nature of the research. Although this research indicates that access to different case library designs impacted learning outcomes, additional studies could explore to what degree novices explicitly identify indices within the case as they solve the problem. To further under the expert/novice gap, this student data could then be compared with how experts solve similar types of problems. Given that indices are dynamic and based on additional experience, it would be also be interesting to measure indices longitudinally or based on various case sequences. This will provide further insight into the role of case libraries and design guidelines that impact effective index generation, retrieval patterns, knowledge structure development, and transfer. Related to this point, one might argue that the design is germane to the type of domain since fields vary in the types of problems they encounter (Jonassen and Hung 2008). Other studies could thus explore the degree to which these results are maintained and the role of design across domain areas, including engineering, medicine, and social studies. It is possible that the results are a byproduct of the problems situated within different contexts.

Future research could further explore how the design of the case was presented to the learner. Past research has argued case libraries bridge the experience gap often cited by critics of PBL (Fitzgerald et al. 2011; Rong and Choi 2018; Tawfik and Kolodner 2016). This study tested that hypothesis by exploring the degree to which the experiences depicted in static case libraries or recommendation-based case libraries supported learners when compared with no scaffolds. However, it may not be feasible for all educators to generate algorithms and recommendation systems; therefore, future studies could explore the effectiveness of lower-cost options at conveying case aspects and facilitated problem-solving transfer. In this study, the retrieval and reuse of the case was supported through the use of a pull-down menu and search functionality. As alternatives to managing retrieval are developed, it may be that there are others ways to present the indices that might better support index generation and transfer. For example, a knowledge structure diagram might impact how learners identify and select the relevant indices to new problems. Related studies could also build on the retrieval algorithm and explore its effects on learning transfer. As noted earlier, the algorithm was structured to recommend a case based on the experts’ ranking and their scored relevance towards the main problem to solve. Additional retrieval approaches could yield differential outcomes if the algorithm was based on other metrics, such as the number of views or peer-relevancy scores. Exploration across different mediums could provide important insight into how the design impacts case retrieval and reuse during problem-solving.

Another opportunity for future research includes exploration of retrieval patterns across  various case library conditions. While retrieval analytics were limited to the recommendation system condition, future studies could compare analytics across different case library designs. Given the growing body of literature that suggests the design of a case library impacts learning outcomes (Gartmeier et al. 2015; Lajoie et al. 2014; Tawfik 2017), this type of research could provide additional insight into how novices engage in the case-based reasoning cycles and problem solving. In the current study, two distinct clusters emerged that could be attributed to various things. For instance, the results could be influenced by achievement levels, motivation during problem-solving, or even the overall user-experience. These studies could provide additional design scholarship and further explicate how learners engage in iterative problem-solving using case library learning environment.

The cluster analysis provided insight into how learners iterate their problem solving using case libraries. However, this was only present for the recommendation system case library. A follow-up study could compare similar analytics across various case library designs. Along these lines, further research could explore how the learners’ retrieval mechanism adapts based on learners’ emerging knowledge gaps. To account for individual differences in domain knowledge or prior PBL experience, a follow-up study could also employ a delayed treatment design. We also suggest that future research conduct qualitative analyses to future investigate student experience with the recommendation system. For example, focus group interviews could help to identify challenges students faced as they engaged in case-based learning in the system. Such studies would provide further information about the role of requisite scaffolds needed to better support novices during their knowledge structure development.