The effects of successful versus failure-based cases on argumentation while solving decision-making problems

Abstract

Solving complex, ill-structured problems may be effectively supported by case-based reasoning through case libraries that provide just-in-time domain-specific principles in the form of stories. The cases not only articulate previous experiences of practitioners, but also serve as problem-solving narratives from which learners can acquire meaning. The current study investigated the effects of different case-types (success, failures) on analogical transfer to similar problems. In the first week, undergraduate sales management students (N = 36) were assigned to different case library treatments (success, failure) and asked to construct a multifaceted argument (initial argument, counterargument, rebuttal) to resolve an ill-structured, decision-making hiring problem. In the following week, students constructed an argument to solve a novel case without the support of the case library. Data analysis revealed the failure-based case library condition produced significantly higher scores on measurements of counterarguments and holistic argumentation scores on both tasks. A discussion of the implications for pedagogy and instructional design are also presented.

Introduction

Researchers have noted the importance of problem-solving to support higher order learning (Jonassen 2011; Kim and Hannafin 2008). However, problem-solving competencies, such as argumentation and causal reasoning, are often difficult for learners to attain as they transition to ill-structured tasks. One way for instructional designers to facilitate problem-solving is by providing learners access to a case library learning environment that details the experiences of practitioners using case-based reasoning (CBR) theory. However, very little empirical research currently exists as to the best instructional designs that support higher order learning through case library learning environments.

In practice, individuals not only engage in successful problem-solving, but experience and learn from failures as they resolve ill-structured problems. A key issue, therefore, is how exposure to case library learning environments that consist of failure or success narratives might impact problem-solving skills of students, such as argumentation. Whereas success-based cases document best practices, failure problem-solving narratives may help elucidate latent pitfalls that a learner might not have otherwise recognized. If case library learning environments are to be used as a strategy to engender problem-solving, further research is needed to ascertain how success or failure might be employed construct these supports.

Literature review

Problem solving is the most ubiquitous intellectual activity in our everyday lives. As Popper (1987) notes: “All life is problem solving”. Learning to solve problems should therefore constitute a major focus of education. However, learning to solve problems represents a significant challenge for educators and instructional designers because problem solving is not a unitary activity. Problems vary, so learning to solve different kinds of problems must also vary, calling on different cognitive and social learning processes. Furthermore, problems vary along a continuum between well-structured and ill-structured problems (Jonassen 2000). To date, most problems encountered in formal education are well-structured problems. These problem types typically present all elements of the problem; engage a limited number of rules and principles that are organized in a predictive and prescriptive arrangement; possess correct, convergent answers; and have a preferred, prescribed solution process.

Ill-structured problems, on the other hand, are the kinds of problems that are encountered in everyday practice. Ill-structured problems are more difficult to solve because they have many alternative solution paths; vaguely defined or unclear goals and constraints; and multiple criteria for evaluating solutions. As is emphasized by advocates of problem-based learning (Barrows 1980, 1996; Barrows and Tamblyn 1980; Dunlap and Grabinger 1996; Gijbels et al. 2005; Norman and Schmidt 1992; Savery and Duffy 1996) and the authors in this volume, classroom-based problem solving should also require students to solve complex problems.

Everyday ill-structured problems tend to be more complex than well-structured school problems. The complexity of a problem is a function of the breadth of knowledge required to solve the problem, the level of prior knowledge, the intricacy for the problem-solutions procedures, relational complexity within the problem, and problem structuredness including intransparency, heterogeneity of interpretations, interdisciplinarity, dynamicity, or competing alternatives (Jonassen and Hung 2008). Complex problems have uncertain methods and competing alternative solutions, requiring learners to make decisions among competing solutions under conditions of uncertainty. With complex problems absolute certainty in reasoning is rare. Certainty in problem solving tends to occur with the repeated solution of well-structured problems that invoke routinized behaviors.

CBR

Another critical characteristic of complex problems is that they are often case-based. That is, complex problems tend to occur in rich contexts, establishing each as a unique experience. Interpretation of those experiences relies on understanding the multiple, interacting perspectives that each problem case has taught us. Among the prominent theories for describing the development of case knowledge required for the acquisition of complex problem-solving skills is CBR. According to CBR, learning to solve complex problems results from experiencing similar problems. These experiences, stored in the form of cases, represent the interpretation of previous problem-solving experiences and the subsequent lessons learned. An encountered problem (the new case) prompts the reasoner to retrieve cases from memory and apply them to the present circumstance to suggest a solution (Aamodt and Plaza 1996; see Fig. 1). If the suggested solution will not work, then the solution must be revised. When an effective solution is confirmed, the learned case is then retained for later use. Over time, individuals build a case library of problem-solving experiences within memory to reference.

Fig. 1

CBR cycle, Aamodt and Plaza 1996

Case-based reasoning is a form of analogical reasoning that contends individuals naturally reason through previous problem-solving experiences situated within authentic contexts (Kolodner et al. 2004). As such, previous experience becomes the basis for learning as an individual reasons through ill-structured problems. As knowledge and expertise increases, practitioners rely more on reusing previous cases stored in episodic memory that are relevant to the current problem rather than linear problem-solving processes (Kolodner 1992). According to CBR, over time individuals are better able to assess the current problem, find previous cases that are relevant to the current problem, leverage that case to inform a solution, assess the potential solution, and update internal memory as one learns from the experience (Aamodt and Plaza 1996).

In order to support problem-solving instruction, CBR can be applied through case library learning environments as a way to supplement episodic memory. A case library learning environment is a database containing stories of how experienced practitioners have solved similar or relevant problems. The case library database functions similar to the case library within the memory of practitioners. Since novices often lack experience, the stories provide conceptual and strategic information that can be applied when facing new problems (Jonassen and Hung 2006). Reflection upon the narratives contained in the database serves as a proxy for the experiential knowledge the learners do not yet possess (Jonassen and Hernandez-Serrano 2002). As multiple stories are encountered, the narratives from the case library learning environment becomes indexed within the learner’s episodic memory (Kolodner et al. 2005).

Despite the potential of CBR and case library learning environments, the empirical literature is very limited. In one study regarding the efficacy of CBR, Hernandez-Serrano and Jonassen (2003) found that access to a case library learning environment while solving food product development problems resulted in higher scores on predictions, inferences, and explanations related to those problems. The case-library format helped to contextualize the content and allowed the learners to understand how to use the domain-concepts for problem-solving. However, the literature base on CBR is almost entirely descriptive of various implementations and additional research is needed to understand the effects case libraries have upon problem-solving (Jonassen 2011).

Failure-driven memory

If case library learning environments represent a better means generate comprehensive mental models, further studies are required to ascertain how best to construct these supports (Jonassen 2011). Mental models are important to problem-solving because they consist of the cognitive representations that individuals use to make sense of their experience based on the interrelated nature of domain-specific knowledge (Ifenthaler 2010; Seel 2008). One issue is how exposure to a case library learning environment that consists of failures or successes might impact an individual’s mental model. One could argue that case libraries built around successful stories constructs stronger mental models because learners are able to see how various strategies can be utilized for effective problem solving. Alternatively, others might contend that failure exposes faulty assumptions and latent variables, allowing the learner to better understand the complexity of the problem.

This study represents a small step in that direction by focusing on the comparative efficacy of success-based versus failure-based cases in support of problem-solving. Failure cases are contrasted with “best practices” cases that emphasize successful efforts of problem solving. It is reasonable to assume that because learners lack experiential knowledge, a model of successful problem-solving would best instruct novices in appropriate domain practice. However, Schank’s (1999) theory of failure-driven memory suggests that errors are a more effective means of learning in CBR. When encountering a failure, CBR proposes that one will first attempt to find the relevant higher-level knowledge structure to employ for processing. If no counterpart is found within memory to accommodate the failure, a new path from the experience is generated that serves as a new foundation and experience for future problem-solving. That is, if the experience is “classified initially as being interestingly deviant, [the experience] serves as embellishments to the scripts of which they are a part and thus serve as a source of predictions” (p. 53). Along with the prediction an alternative belief is generated to explain the goals or behaviors of the situation that is not produced during case successes (Schank 1999). Finally, this serves as a new memory along with an associated index.

Schank’s theory of failure driven memory (1999) also suggests that failure is important because it requires an individual to amend erroneous assumptions about previous decisions, assessments, and actions. When analyzing experiences of failure, individuals contemplate and reflect upon future transfer cases (Sitkin 1992), better understand the conditions of success (Shepherd 2003), focus on the subtle key factors that may be latent within the experience (Kolodner et al. 2004), and thus generate a more complete mental model (Mathan and Koedinger 2005). Failure cases may also promote opportunities for creativity of new solutions and techniques not afforded by cases of successful problem-solving (Hoeve and Nieuwenhuis 2006; Parviainen and Eriksson 2006). As such, “dynamic memory is always changing as each processing failure causes learning to take place” (Schank 1999, p. 47).

To date, much of the learning through failure research is situated within non-CBR workplace learning literature (Gartmeier et al. 2008, 2010). For instance, Bauer and Mulder (2007) note that investigation of medical failure serves to identify deficient interpretation of the problem, misapplication of new method practices, and application of outdated methods. The medical field also employs morbidity and mortality conferences to present cases of failure as opportunities for learning (Gore 2006; Rosenfeld 2005). Similarly, research of military defense preparation found that those that reflected upon failure experiences were able to generate a more comprehensive and thorough causal map of the incident (Ellis and Davidi 2005; Ellis et al. 2006). The authors reasoned that failure best helped to identify flaws in one’s mental model and stimulate further inquiry to explain the phenomenon.

Purpose of the study

The purpose of this study was to examine the efficacy of success or failure-based case library learning environments to support argumentation. A common assumption is that a case library learning environment built around successful business stories (best practices) would allow the learner to see how various domain strategies can be appropriately utilized by practitioners. Alternatively, presenting cases that document failed problem-solving experiences may uncover variables that a learner might not have otherwise recognized. Therefore, we pose the following research questions:

• 1. To what extent does learning performance on an argumentation task outcome differ if the learning experience is based on cases of success or failure?

  • a. To what extent does learning performance on initial arguments differ if the learning experience is based on cases of success or failure?

  • b. To what extent does learning performance on counterarguments differ if the learning experience is based on cases of success or failure?

  • c. To what extent does learning performance on rebuttals differ if the learning experience is based on cases of success or failure?

  • d. To what extent does learning performance on holistic argumentation scores differ if the learning experience is based on cases of success or failure?

Methodology

Participants

Participants were drawn from undergraduate business students enrolling in a Fall 2011 class entitled Sales Management, an upper-division course offered in the College of Business at a Midwestern university located in the United States. A total of 36 students were enrolled in the course (Male = 19, Female = 17) across two sections. This particular course generally enrolled junior level marketing students. All participants voluntarily elected to participate in the study.

Procedure

The study was completed during the Fall 2011 semester. During the second week of the semester, participants completed a pretest to assess any differences between treatment groups in previous knowledge. Participants were assigned to two treatment groups (failure case library, success case library) by section. During the eighth week of the semester, participants were required to access the web-based learning environment to complete the initial task over 1 week. After reading the initial problem to solve, participants were prompted to click on the resources portion of the screen to view a case library learning environment (see Fig. 2). In the ninth week (transfer problem), the same procedure was followed except for the inclusion of the case library learning environment.

Fig. 2

Learning environment

Materials

Instructional materials included two sales management decision-making problems. Each of the problems to solve (initial, transfer) were derived from an authentic experience encountered by the course instructor, who had over 20 years of business experience prior to teaching. The researcher interviewed the instructor and asked him to detail a hiring and selection problem he encountered and that his students would be expected to solve upon entry into the workforce. Moreover, the researcher asked the instructor to describe an ill-structured context where he was conflicted whether to hire a particular individual and the reasons for his indecision. This experience was then translated into a problem for the participants to solve accessible through the web based learning environment (see Fig. 2).

The initial problem to solve presented a potential interviewee who had an excellent work history and strong character references, but also a criminal history. As such, it was unclear whether the hire was justifiable. In the initial task the applicant offers to pay any additional insurance costs associated with his criminal history. The researcher and instructor modified the case to be further ill-structured. For instance, the case was constructed such that the individual intentionally removed his criminal history from his application. In addition, the protagonist of the case was compelled to quickly make a correct hire because of recent turnover within the position and market pressure. The risk and pressure further required the participants to question the hiring decision.

Argumentation was used as the primary data source because it is among the strongest means for assessing the ability to solve ill-structured problem problems (Jonassen 2011). As learners engage in argumentation, they generate multiple relationships with prior knowledge that create additional routes by which to recall information (Nussbaum 2008). Kuhn (1993) further argued the following serve as evidences of knowledge construction in argumentation: generation of causal theories that support claims, evidence proffered to support theories, generation of alternative theories, strategies that serve to weaken the supportive theory, and rebuttal of alternative theories. Decision-making problem types, in particular, are often contextualized and possess multiple solutions, unclear goals, and require the learner to make inferences supported by evidence (Jonassen 1997). As part of the argument, students were required to construct an initial argument, counterargument, and rebuttal (Kuhn and Udell 2007; Nussbaum and Schraw 2007). Participants were allowed 1 week to complete the initial task. Argumentation also served as an appropriate assessment of problem-solving because the activity assessed the beliefs, judgments, evidence, and subsequent conclusions germane to the decision-making problem (Kuhn 1993; Kuhn and Udell 2007).

As participants solved the initial task, the learning environment displayed five cases to support their problem-solving. In order to construct the case libraries, the researcher asked the practitioners to recall previous hiring and selection experiences similar to the presented problem using a protocol for successful (see Table 1) and failure cases (see Table 2), adapted from Jonassen and Hernandez-Serrano (2002). Tables 1 and 2 include not only the questions, but the indices that connected the stories. A total of 10 cases (five successful and five failure) were composed as a result of the interview.

Table 1 Interview protocol for successful stories

Table 2 Interview protocol for failure stories

The independent variable in this study was the type of cases provided in support of problem solving: success or failure cases. While solving the initial task, the success condition participants referenced five similar cases of practitioners’ effective problem-solving for hiring and selection in business. Alternatively, the failure condition participants referenced five cases of hiring and selection failed outcomes. The learning environment interfaces appeared completely the same to each treatment group, with the only difference being the content of the case libraries. Rather than randomly assign individuals to sections at the beginning, participants were assigned the case libraries based on their course section. Participants were assigned by section because participants often employed group work in the class. By assigning the case library using intact groups we hoped to minimize the risk of a participant sharing the learning environments across treatments. The course sections were assigned to different case libraries using a Microsoft Excel randomization macro.

The cases were constructed to evoke similar concepts. Surface factors of the case such as character names, gender, lessons, and contexts remained the same to avoid potential confounds. The following framework was also developed as an effort to maintain structural characteristics across the cases: setting (e.g., steel mill, Fortune 500 company), general problem description (e.g., changing jobs within industries, new business struggles), assumptions (e.g., ambiguity in verbiage; different roles require different skillsets), constraints (e.g., time demands, glass ceiling), social issues (social fit, employee morale), overarching sales management lesson (e.g., hiring qualified workforce, job placement alignment), and outcome (e.g., loss of customers, increased turnover). This framework helped to reduce variability between the cases and maintain consistency across the narratives. The only difference was whether the case depicted a success or failure narrative (see Appendices 2, 3).

In the following week, a transfer task required the participants to solve an ill-structured hiring problem. In the transfer task, the protagonist was uncertain whether to nominate an individual for a promotion because of a recent driving under the influence record and poor credit history that was accumulated prior to employment at the firm. The protagonist is also provided with an alternative candidate that had a strong sales background, but lacked the technical expertise desired for the position. As in the case of the initial task, participants were once again required to construct a multifaceted argument that justified the decision. In contrast to the initial task, no cases were provided to assist the participants in the transfer task. As such, the second task assessed participant’s transfer ability.

Instruments

A 20-item, five-option pretest was administered early in the semester. The pretest consisted of both general marketing and specific selection concepts. The purpose of the pretest was to identify any differences in domain knowledge between the treatment groups prior to assigning the case library learning environment.

The primary data source consisted of the arguments that participants constructed in support of their decisions. Participants were prompted to construct an argument that included three different components: initial argument, counterargument, and rebuttal. For the purposes of this research study, the initial argument was defined as the student’s initial stance as to whether a particular individual should be hired (Kuhn and Udell 2003). Initial claims were scored using the rubric in Table 3. Participants were scored a 0 if the argument factor (initial claim, counterclaim, and rebuttal) lacked consistency or coherence; 1 point for a position that was poorly developed; 2 points for a position that was clear, but lacked a quality explanation; 3 points for arguments that were well developed, but only partially explained; and 4 points for arguments that were well constructed and included evidence.

Table 3 Initial argument scoring rubric

The second dependent variable, participant counterargument, was defined as the participant’s ability to offer an alternative solution or challenge to the initial argument. This variable required a deeper level of problem-solving because it necessitated a participant to generate alternative positions or proffer evidence that contradicted the initial position (Nussbaum and Schraw 2007).

The third dependent variable consisted of a rebuttal. Nussbaum and Schraw (2007) argued that a rebuttal may include various strategies. For instance, a participant could choose a strategy that challenged the validity of the counterclaim (refutation strategy). Alternatively, an individual could proffer a solution that offered a compromise between the argument and counterargument (synthesizing). Lastly, the participant could argue that one side was stronger and more influential to the argument (weighing). In each strategy, the rebuttal required the student to integrate the evidence from both the initial position and counterarguments. Counterclaims and rebuttals were similarly coded using the rubrics in Tables 4 and 5. These coding schemes were adapted from Jonassen and Cho (2011). However, whereas the Jonassen and Cho study included the rubric for a holistic, line item, and rebuttals, this study amended the rubrics to include additional argumentation variables such as initial argument and counterargument. The final variable, holistic score, was calculated by aggregating the initial, counterclaim, and rebuttal scores (0–12 points).

Table 4 Counterargument scoring rubric

Table 5 Rebuttal scoring rubric

The unit of analysis for the quantitative data analysis was the argumentation variables for both the initial and transfer case of the participants. After the initial and transfer argumentation activities were submitted to the researchers, all identifying information was removed from the responses by an individual not involved in the study.

Prior to coding, two researchers were given rubrics that explicitly defined the categories for each variable (Tables 3, 4, 5). Initial inter-rater reliability strived for at least 80 % as prescribed by other literature (Rourke et al. 2001). Raters met after coding the first 5 argumentation essays to verify that the rating was reliable for the study. However, no problems were identified and the arguments were independently rated (Pearson’s r = 0.98). Inter-rater reliability reached 100 % after negotiating the final assigned scores among the coders (Pearson’s r = 1.00). Inter-rater reliability therefore met the 80 % recommendation as prescribed in previous literature (Rourke et al. 2001).

Results

Prior to introduction to the learning environment, a pretest was completed to ascertain potential differences in terms of domain knowledge between the participants in the two treatments groups. The Shapiro–Wilk test was conducted to ascertain the normality of the data, however no significant differences were identified between the pretest scores success (p = .44) and failure conditions (p = .32). Once normality was established, the results of the Levene’s Test for Equality of Variance indicated that no significant differences were identified between the mean success (8.61) and failure (9.11) conditions. As such, the study proceeded assuming equal prior knowledge of the subject matter between groups.

The first research question attempted to ascertain to what extent success cases and failure cases influenced the initial arguments scores. Because the data was not normalized, the experiment employed an ordered logistic regression with task as a repeated factor (Fullerton 2009). However, the study revealed no significant differences between tasks on the initial claims (dependent variable) for the success and failure conditions (X² = 3.13, df = 1; p = .077; d = 0.42) (see Tables 6, 7).

Table 6 Argumentation initial scores by task

Table 7 Argumentation initial scores by treatment

The second research question investigated whether significant differences existed between mean counterclaim scores. Because the data was not normalized, the experiment once again employed an ordered logistic regression with task as a repeated factor to identify significant differences between the counterclaim scores (Fullerton 2009). In contrast to initial claims, the failure counterclaim scores showed significant differences when compared to success counterclaim scores across success and failure conditions (X² = 4.66, df = 1, p = .031; d = 0.28) (Tables 8, 9).

Table 8 Argumentation counterclaim scores by task

Table 9 Argumentation counterclaim scores by treatment

The third research question investigated whether significant differences existed between mean rebuttal scores. Because the data was not normalized, the experiment once again employed an ordered logistic regression with task as a repeated factor (Fullerton 2009). However, the study revealed no significant differences over tasks between rebuttal scores mean sores across success and failure conditions (X² = .39, df = 1, p = .529; d = 0.22) (Tables 10, 11).

Table 10 Argumentation rebuttal scores by task

Table 11 Argumentation rebuttal scores by treatment

The final research question assessed differences in overall mean argumentation scores. To quantify overall argumentation scores the initial argumentation, counterclaim, and rebuttal scores were summed to create a final score (0–12; see Tables 12, 13). As in the counterargument variable, significance differences were found between mean total scores for the success condition and failure condition (F = 4.534, df = 1, p = .04; d = 0.56) .

Table 12 Argumentation overall scores by task

Table 13 Argumentation overall scores by treatment

Discussion

Practitioners in most professions encounter and solve ill-structured problems using implicit and explicit argumentation to justify their solutions. This study focused on two unique properties of problem-based learning environments: the use of case libraries to support problem solving and the use of argumentation to assess problem-solving abilities. In order to add to the limited empirical research on CBR, this study sought to investigate the efficacy of case libraries to support problem solving, specifically the degree to which cases libraries depicting failure or success support argumentation. In order to assess problem solving ability, participants constructed argumentation texts in support of their decisions, including initial claims, counterclaims, and rebuttals.

When making initial claims, results revealed the participants’ ability to construct a preliminary stance and strategy regarding the hiring decision did not significantly differ between treatment groups that accessed cases of success or failure. This suggests that success or failure cases aide commensurately for an individual to construct an initial conceptualization of the problem during an argumentation task. According to CBR, indices are generated at various levels for a case (Schank 1999; Kolodner et al. 2004). During problem solving, CBR proposes individuals retrieve, revise, reuse, and retain the case using the indices within memory (Aamodt and Plaza 1996). The findings suggest that both success and failure-based cases helped to generate similar indices that introduced the problem and provide a high-level problem conceptualization during the argumentation task. Participants were later able to retrieve and reuse similar high-level indices for problem-solving.

When participants were asked to identify and describe counterarguments, the hallmark of argumentation (Kuhn and Udell 2003), those that received failure cases produced significantly higher counterclaim scores when compared with the success condition for both tasks. This suggests that students who accessed failure cases were able to better understand and construct arguments using alternative perspectives. There may be multiple explanations for these results. According to Schank (1999), cases are useful for script construction within memory. Although success may help to solidify concepts, success based cases generate less indices because the variables are not always salient during processing of the experience (Schank 1999). An individual thus generates relatively fewer indices by which s/he can reference a narrative for future problem-solving.

Alternatively, CBR argues that individuals are naturally compelled by the mental discomfort caused by unsatisfied expectations to create an explanation for the failure. Failure-based cases therefore may expose latent nuances within a narrative, aide in repair strategies of invalid assumptions, and subsequent script revision. In contrast to success-based cases, Schank (1999) argued an alternative belief is generated to explain the goals or behaviors of the situation. Associated indices related to the explanation failure are subsequently stored within memory. It is thus possible that the higher scores were found because the failure treatment forced the participant to reflect upon breakdowns in causality and internalize a more comprehensive mental model of the problem with the additional indices. Participants that accessed failure cases were also possibly able to reference the additional indices within memory related to the alternative belief when constructing the counterarguments.

When asked to construct rebuttals, no significant differences were found between case types for rebuttals. One potential reason for this result is that the construction of rebuttals is a different cognitive activity than articulating an initial stance or alternative viewpoint. Rebuttals instead require a comparison, refutation, or a synthesis of the initial and counterarguments proffered. Although the case library learning environment may have generated indices associated with the cases, the success or failure case libraries may not have adequately supported participants as they reconciled and weighed the indices referenced in the earlier arguments.

Finally, when argumentation scores were combined to reflect the overall quality of student arguments (holistic score), the results of the data analysis revealed significant differences between case types. Similar to the counterclaim results, the failure case library treatment resulted in significantly higher holistic scores when compared with the success case library treatment. This suggests that while the failure case library learning environment may have not have resulted in higher scores for initial claim and rebuttal scores, failure cases helped to promote overall better skills throughout the entirety of the argumentation task. Once again, the self-explanation derived from an investigation of failure cases may also have led to more indices within memory the participant could reference throughout the argumentation tasks. According to CBR theory, this self-explanation is less likely to occur when processing a success case. The increased number of indices derived from failure cases also may have engendered additional generalizations that were applied throughout the all aspects of the argumentation task.

Implications

According to the CBR model, an individual naturally seeks to reference previous experiences when solving new problems. That is, solutions are derived from similar cases. To date, the potential of CBR in learning has been largely descriptive of various design prototypes. This study intended to take a small step toward empirically validating instructional design principles for constructing case library learning environments that support complex problem solving. The research presented in this study suggests that case library learning environments based upon the CBR model may be one way to support analogical transfer in problem-solving, especially for ill-structured problems as the cases function as a just-in-time learning aide (Kolodner et al. 2004). Moreover, the study provides some empirical support for the inclusion of failures in cases libraries to alert problem solvers of potential pitfalls in various solution options.

Kolodner et al. (2004) suggested that a learner can engage in meaning making and causal reasoning as s/he reflects upon stories of failure using a case library-learning environment. Although not entirely authentic, proponents of CBR argue that reflection upon the experiences of others serves as vicarious experience that can be indexed within the episodic memory of the learner (Jonassen 2011; Kolodner et al. 2004). This study extends the workplace failure-based research by suggesting that one does not only have to experience a failure firsthand, but one can gain vicarious experience through accessing narratives within a case-library learning environment. Although it is possible that the generated indices are different between the practitioner and learner, the results suggest the learner individuals still acquire meaning from the meta-failure of the narrative as they reflect upon the conditions, factors, and lessons learned of the case.

One potential reason for the efficacy of the failure case library in the current study is that errors may make the causality more overt as an individual reflects upon the case. Failure-driven memory theory (Schank 1999) suggests that when a failure occurs, the scripts that resulted from prior experiences are more readily available for revision. A case library that depicts failures thus may have required the student to pause and investigate the ‘causal linkages’ of the case when compared with simpler script-based interpretations employed by successful cases. As such, a breakdown in the causal linkages helps to more overtly express to novices how domain specific principles are connected. It is possible the explanations of failure better indexed the cases within memory and therefore served as a better aide during the problem solving activities.

This study is also important as it relates to argumentation for ill-structured problem-solving. Argumentation is one means by which to support and assess learning in terms of reasoning, backing of claims, contemplation of alternative reasoning (Nussbaum and Schraw 2007), synthesis of perspectives (Nussbaum 2008), and conceptual change (Asterhan and Schwarz 2009). In previous studies, researchers have documented difficulties in the facilitation of argumentation skills (Kuhn and Udell 2003; Lin and Anderson 2008). Current instructional design strategies have attempted to support argumentation through directions (Nussbaum et al. 2005), scripts (Jeong and Joung 2007; Stegmann et al. 2007; Weinberger et al. 2010) and graphic organizers (Nussbaum and Schraw 2007). The results of the present study builds upon previous research by suggesting that case library learning environments may be another means by which to facilitate argumentation skills.

The current study indicates that access to a case library learning environment may specifically support the learners as they construct facilitate counterarguments. Empirical studies have shown that a particular difficulty in argumentation is the ability to weigh evidence and take into account the evidence of different perspectives (Kuhn and Udell 2007; Nussbaum 2008). Jonassen (2011) suggested that one reason for the difficulty of counterarguments is that learners are reluctant to search for disconfirming evidence. One potential reason is that an argumentation activity inherently involves articulation and refutation of errant reasoning (Mason et al. 2008). Investigations of failures using a CBR approach appear to better emphasize and highlight misconceptions in reasoning that may have been otherwise overlooked in cases of success.

Limitations and future research

The general finding was that case library learning environments that depict failure are an effective means for supporting problem-solving skills such as analogical transfer and argumentation. However, there are multiple limitations to consider when interpreting the results of the current study. One limitation may be the actual design of narratives found in the case library. Although efforts were made to design the cases in a similar way, it is possible that the higher scores were a result of the different structural features between success or failure cases.

A related study could investigate other factors related to success or failure-based cases to address the limitations of the study. In particular, one limitation is that the current study did not investigate time-on-task data. In the current study, participants were assigned each task over the course of 2 weeks (Task 1 = 1 week, Task 2 = 1 week). However, analysis of differences in time on task may further explain the statistically significant results found in the counterargument and holistic argumentation scores of the failure treatment. Others could observe the individuals throughout the week or employ an analytics software within the learning environment to obtain important usage metrics, such as time on task. This could provide additional data related to motivation and engagement that would further enhance the understanding of case library learning environments.

Another logical follow-up study could include a third treatment group that accessed narratives of both success and failure within a single case library learning environment. A mixture of case types might support cognitive flexibility (Schrader et al. 2003; Spiro et al. 1988) and thus result in a more comprehensive mental model of the problem.

An additional study could allow students to research and submit their own cases through investigation of news stories or history. For instance, industrial engineering students learning to improve nuclear power plant safety could research and submit articles about the Chernobyl or Fukushima power plants disasters. Because popular news stories may have been encountered more frequently and indexed within memory, these cases might also allow the learner to further elicit meaning. In doing so, CBR may become more active as learners are encouraged to assign meaning to the case, construct individual knowledge, and share case interpretations with peers (Jeong and Lee 2008).

Another potential study could combine CBR with the benefits of collaborative learning. In the present study participants were required to individually read the case and then construct an argument. However, research has shown that collaborative argumentation may engender more critical discourse and higher order learning (Jeong and Joung 2007; Lin and Anderson 2008). Collaborative learning may allow the students to explicitly negotiate and socially construct indices related to each encountered case. An instructional strategy that applies collaboration might afford additional perspectives, explanations, and challenge of assumptions between peers that may not have been identified at an individual level (Nussbaum 2008).

An additional study could also investigate how cases of failure or success are impacted by problem types. Whereas this study examined decision-making problems, Jonassen (2011) has suggested that various domains encounter different problem types such as rule-induction, troubleshooting, diagnosis-solution, and strategic performance problems among others. The field of education would benefit from an empirical investigation as to whether differences in failure stories remain across problem types and domains.

Appendices

Appendix 1: Initial task

Nick stepped into work Monday morning with his boss, Sheila. She scheduled this meeting to discuss a series of applicants that were being considered to fill a medical device sales position left open after someone recently left to pursue another opportunity at another company.

“Nick”, she begins, “we need to stop having to fill this position. It is killing us in terms of time and money to have to hire and train a new person every 6 months. We’ve had a lot of turnover in this medical sales position that needs to be stopped. As you know, we’ve missed on some of the previous hires. The three people we have had come in and out have cost us $90,000 over the last year in terms of revenue and training. That’s $30,000 per person! The last individual hired for the position seemed pretty good in terms of technical expertise, but it was pretty clear that the sales aspect of the job wasn’t a great fit. Let’s go through some of these together and see if we can find someone with that right mix between technical expertise and social skills”.

After going through the applicants, it becomes evident that it was difficult to find a great deal of qualified applicants.

“Oh man,” Nick exclaims. “I didn’t realize it would be this hard to find one person to fill a position. A lot of these people look really good on paper, but they just don’t have the sales experience needed. They have decent schooling, but I want to make sure we bring in the right people. We could try to retry posting a job ad in the St. Louis newspaper, but that costs us about $1,500 per month. It’s a risk shelling out all that money, but I think it’s worth it if we get the right person rather than continuing to lose market share and have to constantly train new people. How about that list you have in front of you? Do you see any resumes that you like in particular?”

Sheila thumbs through some applicants. “Actually, here is one that seems pretty interesting. This individual, Lewis, has a decent GPA. It is about a 3.1 overall, but a 3.8 in classes related to his major. He also has somewhat related experience when he worked as a marketing intern for a children’s hospital. Another option is try to try to promote from within. That might only cost us $15,000 to train a new person. I’ve heard great things about one employee in particular. This one employee, Terry, gets great telemarketing numbers in one of the worst territories for selling smaller medical devices. Plus, I know the supervisor in that department raves about Terry’s character and leadership in that role. Although the experience isn’t totally equivalent, it sounds like Terry has a chance to connecting with customers face-to-face.”

Nick quickly peruses the resume. “Terry is interesting, but I really like Lewis’ resume. I see there’s a lot of activity in some service fraternities and showed some leadership qualities there. I just love the experience in the medical field. If the references check out, why don’t we setup an interview? If not, a better bet might be to just draft a posting for a newspaper advertisement.”

Nick takes the time to contact Lewis’ references. Each are previous supervisors for whom he has worked within the last 5 years. They all speak positively about his experience and determination in the workforce. They also raved about how Lewis fit in so well within various work cultures. Nick decides to move forward with the interview at headquarters. However, in doing the background check, Nick discovers that Lewis has had an incident with an underage DUI. This is problematic because the company has a policy that does not allow individuals with moving violations to be eligible for sales positions that require a company car. However, Nick decides to still pursue the interview.

During the interview, Nick is impressed with Lewis’ demeanor and knowledge of the field. As the interview is wrapping up, Nick says “I enjoyed our meeting. I have to be honest with you. I did a background check and your DUI came up. Our policy says that we are not allowed to hire you except for special exceptions. We have a lot of expensive medical equipment and this a big burden for our insurance. If we were to hire you, our insurance would go up even more. What do you think I should do?”

Lewis says, “I know that. I probably should have detailed that on the online application, but I didn’t want to be excluded right away. To be honest, that was something that happened a long time ago. I realize it was a learning experience for me and I did all that the court required of me. Now that I’m 25, I feel as though I have taken those life lessons and will be a better employee for you because I appreciate the opportunity more. This is a dream job for me and I will be willing to do whatever it takes. You could start me off at a smaller job to demonstrate my trustworthiness. In fact, you can even take out any additional liability cost out of my paycheck until it is no longer an issue”.

Nick pauses for a minute. “I am going to have to reflect on this a bit more. I will call you in a week to tell you our decision”.

Assignment

Construct an argument (2 pages double spaced) detailing why Nick should or should not hire Lewis. The argument should include three different components: initial argument, counterargument, and rebuttal.

1. The initial argument provides a stance and supporting evidence for an initial position.2. The counterargument consists of a position and evidence that others might suggest that undermine your theory made in the initial argument.3. Finally, propose a rebuttal will answer and/or respond to the evidence proposed by the counterargument. Rather that restate the initial argument, the rebuttal should reconcile the arguments made in the argument and counterargument. For instance, the rebuttal might (a) try to challenge the validity of the stance made in the counterclaim (b) suggest a solution that offers a compromise between the initial argument and counterargument or (c) argue that one side is stronger and more influential to the argument.

Appendix 2: Sample success story

Case 1: Holly’s chance

After looking for 2 years, Jason finally found the right position that would allow him to transition from the factory floor to steel sales. In fact, he had always dreamed of working in steel sales after 10 years of making steel pipe. The move would require him to move from Pittsburgh to St. Louis, but he was ready to begin his sales career. After many years on the job market and frantic searches for new sales positions, this was finally the new opportunity Jason had been looking for. Although many of his friends had tried and failed to move from the blue-collar culture of manufacturing to the social-oriented sales career, Jason felt up for the task.

Across the country, the St. Louis distributor human resources were reviewing applicants and came across Jason’s resume. Initially, Holly discarded the resume. She had been around long enough to know that individuals did not transition well from moving to the sales side of the steel industry after so many years on the factory floor. Her colleagues did not know how to explain it. They called that missing trait the “it factor”. Holly knew that it was much more than that. She knew that those that failed did not have the social skills necessary to adapt to the new culture. She reasoned this was particularly important sales people that needed to relate with the blue-collar contractors. Finding an individual that had the technical aptitude to speak meaningfully about steel, but know how to communicate in an effective way was difficult to come by. This lack of social skills was unfortunate because the current sales personnel did not have the technical aptitude that could compare with those who actually made steel day-in and day-out.

As Holly looked more closely at Jason’s resume, she wondered if there was some additional work experience that may be applicable. Since Jason did not have many employers, he listed as many as he could on his application even if the positions did not always seem relevant. Despite the advice from his friends who insisted listing a minimum wage position at Ace Hardware was beneath the position he was applying for, Jason included the experience. Holly reasoned that maybe something as simple as listening and relating to customers everyday in this context would be provide Jason the experience needed to build and gain important social skills. Holly knew it wasn’t much, but she strongly believed it showed the aptitude to connect with customers.

Holly decided to take a chance on Jason and hire him. To the surprise of her colleagues, Jason was able to pick up the social aspect of the sale quickly. After 6 months on the job, Jason and Holly sat down to discuss lifting the initial probationary period for all new employees and possible addition of more customer territories. Whereas other individuals required additional training on relating to others or technical aspects of the steel pipe, Jason and Holly began discussing the next steps for his career.

Appendix 3: Sample failure story

Case 1: Holly’s chance

After looking for 2 years, Jason finally found the right position that would allow him to transition from the factory floor to steel sales. In fact, he had always dreamed of working in steel sales after 10 years of making steel pipe. The move would require him to move from Pittsburgh to St. Louis, but he was ready to begin his sales career. After many years on the job market and frantic searches for new sales positions, this was finally the new opportunity Jason had been looking for. Although many of his friends had tried and failed to move from the blue-collar culture of manufacturing to the social-oriented sales career, Jason felt up for the task.

The day of the interview came and the hiring staff really liked Jason’s experience. Although some questioned Holly as to whether Jason had applicable experience that would translate to the new position, she was confident in his skills to become a successful salesman. The job was offered and Jason needed little time to mull over the opportunity. In fact, he had reason to feel confident about his decision. He had a strong college degree in sales and later worked in sales for a mill for another 4 years.

Although Jason had 4 years experience working sales for a mill, he believed it would be a seamless transition as he learned how to be a salesman for a distributor. He soon realized that this position needed a different set of skillsets.

Jason came in on Monday to a meeting with his boss.

“Jason, please sit down” Holly began. “I just heard that we lost another big client last week. That is your third one in 7 months! What’s worse is that one of their executives just said that you yelled at her on a phone conference? What is going on?”

“Well, it’s not my fault. They were complaining about the product and it was clear they just weren’t understanding what it all entails. That’s on them, not me! They come to me complaining about the quality of the product. When I was selling at the mill, we were on a month-to-month commission to keep us moving. We didn’t focus up on all those tedious follow-up issues because we were focused on bringing in money. The way I see it, my job is to sell a product, not have to jump at every little thing they ask me for. You should see some of these things they send to me everyday. They are so trivial and they just drive me crazy all day. I focus on the big things of what I sell to your business, but it’s unreasonable for them to ask me to always hold their hand along the way. How am I making the company money if I’m always responding to every little complaint?”

“Well, that attitude has cost us a lot of money. You need to understand that this is a new culture and it’s not always the same between different industries. We focus on long-term relationship building, not a frenetic selling pace promoted by the mill. You have been so focused on the short-term sales because of your previous experience and now you have compromised our long-term sales strategies. I don’t know if you skillsets truly transferred. As a result of this oversight, I’m wondering if we need to reevaluate your future in this position.”

Appendix 4: Sample counterargument

“While the underage DUI on Lewis’ record can be attributed to childhood mistakes, the repercussions of such a mistake will impact whether he can be hired. With the recent trend of failed hires, costing the company about $30,000 per hire, selecting a candidate with a known risk that will increase the company’s liability jeopardizes the success of the sales team and potentially Nick’s job as a recruiter. With the masses of resumes piling in, Nick’s time is better spent locating a candidate with similar qualifications and no red flags that hiring Lewis and spending time trying to circumvent the costs associated with Lewis’ DUI. A thorough review of possible candidates could possibly save costs in the future associated with lost sales, extra training, etc.

“Promoting from within also boasts several benefits to the company as opposed to hiring outside the firm. Promoting an existing employee is less expensive (increase of $15,000), bypassing questions regarding fit with company culture. Terry, the existing employee, is also better understood by management, perhaps allowing the transition to the new position to be designed more efficiently, without overlap in training, growing pains from becoming acquainted with new people, etc. Management knows Terry’s strengths (face-to-face connections, character, leadership) and could adjust Terry’s responsibilities to her strengths and avoid her weaknesses. The act also serves as incentive for other employees to grow their own careers inside the company.”