Designing and implementing a case-based learning environment for enhancing ill-structured problem solving: classroom management problems for prospective teachers

Abstract

This design-based research study is aimed at two goals: (1) developing a feasible case-based instructional model that could enhance college students’ ill-structured problem solving abilities, while (2) implementing the model to improve teacher education students’ real-world problem solving abilities to deal with dilemmas faced by practicing teachers in elementary classrooms. To achieve these goals, an online case-based learning environment for classroom management problem solving (CBL-CMPS) was developed based on Jonassen’s (in: Reigeluth (ed.) Instructional-Design Theories and Models: A New Paradigm of Instructional Theory, 1999) constructivist learning environment model and the general process of ill-structured problem solving (1997). Two successive studies, in which the effectiveness of the CBL-CMPS was tested while the CBL-CMPS was revised, showed that the individual components of the CBL-CMPS promoted ill-structured problem solving abilities respectively, and that the CBL-CMPS as a whole learning environment was effective to a degree for the transfer of learning in ill-structured problem solving. The potential, challenge, and implications of the CBL-CMPS are discussed.

Introduction

During the last decade, the complex, uncertain, and dynamic natures of real-world problem solving have interested cognitive psychologists (e.g., Sinnott 1989; Voss et al. 1991), applied psychologists (e.g., Zsambok and Klein 1997), educational psychologists (e.g., Spiro et al. 1996), and instructional technologists (e.g., Jonassen 1997; Shin et al. 2003). This stream of research allows us to understand the differences in nature between well-structured problems, often used in classrooms (Spiro et al. 1992), and ill-structured problems, faced in the real world. One of the essential findings from previous studies is that well- and ill-structured problem solving activities require different kinds of skills and abilities (e.g., Schraw et al. 1995). Thus, a successful well-structured problem solving performance in class does not guarantee success in solving ill-structured real-world problems.

Helping college students develop as professionals who are able to deal with real-world problems in complex and dynamic situations, and who can make reasoned and reflective decisions, is one of the essential goals for higher education. In spite of the importance of promoting students’ ill-structured problem solving abilities in higher education, creating such a learning environment is a challenging task for many instructors. One reason why this may be so challenging could be the lack of research-based information and feasible resources available to instructors as they redesign their teaching methods in ways that will enhance their students’ ill-structured problem solving abilities. Only a few conceptual or research-based instructional design models (e.g., Jonassen 1997, 1999; Schwartz et al. 1999; van Merrienboer et al. 2002) or specific instructional strategies, such as question prompting (e.g., Ge and Land 2003), for facilitating ill-structured problem solving are available. More research on instructional design models and implementation models for enhancing ill-structured problem solving skills in the college context is needed to address these issues. In this study, scaffolding of ill-structured problem solving processes was explicitly integrated into an adaptation of Jonassen’s (1999) constructivist learning environment model to develop a case-based instructional model that is aimed at enhancing ill-structured problem solving skills.

Our project was initiated by incorporating the needs of both (a) filling a gap between college classroom learning and real-world problem solving, and (b) creating feasible design and implementation models for improving college students’ real-world problem solving abilities. Two specific goals of this study were (1) to develop and refine a feasible case-based instructional model that could enhance college students’ ill-structured problem solving abilities, and (2) to apply the model to improve prospective teachers’ real-world problem solving abilities in dealing with dilemmas faced by practicing teachers in elementary classrooms. By aiming at theoretical and pragmatic goals and employing iterative cycles of design and implementation of the intervention in real settings, we consider our overall approach to be design-based research (The Design-Based Research Collective 2003; Wang and Hannafin 2005). To highlight the ongoing process of the evolution of our intervention, in this particular paper we would like to focus exclusively on the quantitative data from the learning outcomes, which has been one of the primary references in making our design decisions.

First, we discuss the nature of ill-structured problem solving within the context of classroom management. Second, we explain the instructional design framework that has guided us in developing an online case-based learning environment for classroom management problem solving (CBL-CMPS) and its interface. Third, we report on two successive studies implemented in teacher education courses. Finally, we conclude with a general discussion and the implications of the project.

Instructional design framework

Solving ill-structured problems

Many problems encountered every day in our professional lives pose uncertainties in various ways, including the complexity of the problem context; multiple and, often conflicting, perspectives among different stakeholders; diverse solutions or no solution; and multiple criteria for solution evaluation. These are the general features of ill-structured problems (Jonassen 1997, 2000; Kitchener 1983; Shin et al. 2003; Wood 1983). The complexity of the problems, the existence of conflicting perspectives, and the potential for multiple solutions do not merely make the problems more sophisticated. Rather, these features change the nature of the problems. Thus, ill-structured problem solving demands different sets of skills and attitudes that may not be necessary conditions for solving well-structured problems that have clear goals and known rules to apply to solve the problems (Jonassen 1997; Schraw et al. 1995; Shin et al. 2003).

Domain knowledge and justification skills are known as the major factors that predict success in solving not only well-structured problems but also ill-structured problems (Schraw et al. 1995; Shin et al. 2003). However, there are other factors that may not play critical roles in solving well-structured problems but will in solving ill-structured problems. For example, problem solvers’ epistemic beliefs or interpersonal communication skills may not be related to solving well-structured problems, but these skills play critical roles in the process of solving ill-structured problems (Harrington et al. 1996; Meacham and Emont 1989; Perry 1968/1999; Schraw et al. 1995). Although not many empirical studies have been conducted in this area, problem-solving researchers agree on several essential skills or factors which influence the general performance of solving ill-structured problems. These skills or factors include the following:

• epistemological beliefs–respecting and incorporating multiple perspectives and questioning one’s beliefs and knowledge (Harrington et al. 1996; Perry 1968/1999; Schraw et al. 1995),

• metacognition–planning and monitoring solutions and processes (Brown et al. 1983; Shin et al. 2003),

• justification/argumentation skills–reconciling conflicting interpretations and solutions with sound arguments (Cho and Jonassen 2002; Jonassen 1997; Shin et al. 2003; Voss et al. 1991), and

• domain knowledge (Bransford 1993; Chi et al. 1988; Shin, et al. 2003).

Different models demonstrating essential activities and general processes for solving ill-structured problems have been proposed (see Fig. 1). For example, Sinnott (1989) identified five main activities involved in the dialectic processes of ill-structured problem solving: “processes to construct problem space; processes to choose and generate solutions; monitors; memories; and noncognitive elements” (p. 80). Voss (1988) highlighted recursive interactions between a problem solving structure and a reasoning structure. He discovered three main components–problem representation, solution, and evaluation–under the problem solving structure. Problem representation, along with the development of an argument to support it, was emphasized in his model (Voss, et al. 1991). Jonassen (1997) articulated seven general steps for solving ill-structured problems: articulating problem space and contextual constraints; identifying and clarifying alternative opinions, positions, and perspectives of stakeholders; generating possible problem solutions; assessing the viability of alternative solutions by constructing arguments; articulating personal beliefs; monitoring the problem space and solution options; implementing and monitoring the solution; and adapting the solution.

Fig. 1

Ill-structured problem solving models and the CBL-CMPS model

To guide further development of a feasible instructional model for case-based learning, we adapted Jonassen’s model (1997) for ill-structured problem solving into five essential steps with the accompanying abilities necessary for solving ill-structured problems: (1) understanding situations and contexts where multiple problems may exist, (2) identifying problems by considering multiple perspectives held by different stakeholders, (3) generating possible solutions, (4) choosing appropriate solutions along with a rationale, and (5) implementing and evaluating the solutions. The five steps become the foundation for designing the CBL-CMPS, which is described in the CBL-CMPS model section. We believe that considering multiple perspectives from different stakeholders and generating solutions based on a sound justification while reconciling conflicting perspectives from different stakeholders may be essential skills and abilities that students need to learn in order to solve ill-structured problems. It is important to note that although the aforementioned models for ill-structured problem solving seem to be represented here as a linear process to a certain degree (see Fig. 1), all of these models–including ours–consider ill-structured problem solving as a dialectic and recursive process.

Classroom management as ill-structured problems

Traditional discourse on classroom management tends to focus on “the smooth flow of learning activities as well as daily routines” (Grant and Gillette 2006, p. 99). In order to achieve this goal, classroom management is often reduced to a set of techniques for disciplining individual children’s misbehavior (Doyle 1986). Ayers (2001) argued that this limited view of classroom management leads teachers to see teaching as both a linear process (i.e., classroom management should precede teaching) and a discrete act (i.e., classroom management is separated from the whole of teaching). He pointed out that the commonsense discourse on classroom management centers on manipulating and controlling children rather than on seeing teaching as “intellectual and ethical work” (p. xii). He claimed: “There are a lot of quiet, passive classrooms where not much learning is taking place, and others where children’s hearts, souls, and minds are being silently destroyed in the name of good management” (p. 11).

In addition, the focus on a set of techniques for disciplining children’s behavior as if they are panaceas may prevent teachers from understanding how teaching is a “contextual, local, and situated” act that demands “subtle judgments and agonizing decisions” (Shulman 1992, p. 28). The techniques-oriented discourse and approach to classroom management oversimplifies the issue by assuming that everything about classroom management is a well-structured problem.

However, what appears to be a simple and straightforward problem can be also seen as a complex challenge that requires thoughtful consideration for various stakeholders and for the different aspects involved in the problem. For example, consider the case described in Fig. 2. This situation demonstrates how a seemingly simple problem (e.g., how to arrange children’s desks to prevent interruptions during classroom instruction) can indeed be a complex, ill-structured problem when the teacher frames the problem from multiple perspectives and considers various solutions and their consequences.

Fig. 2

A classroom management case as an ill-structured problem

A design model for a case-based learning environment

Case-based learning

Promoting ill-structured problem solving skills is not an easy task. Although direct instruction (e.g., lectures) may be effective in facilitating well-structured problem solving skills (e.g., domain knowledge), decontextualized domain knowledge isolated from authentic situations and experience has been always questioned for facilitating ill-structured problems skills (Spiro et al. 1992; Whitehead 1929/1967). Beyond the domain knowledge about concepts, principles, models, and theories, which could be constructed through direct instruction, successful ill-structured problem solvers should be able to identify and interpret important situational cues, and should be able to apply (or modify) appropriate principles to a particular situation (Johnson 1988) through experience (Ericsson 2003). It also has been suggested that ill-structured problem solving often relies on case-based reasoning by applying one’s prior experience (Hernandez-Serrano and Jonassen 2003; Schank 1999) because ill-structured problems are often context-dependent (Voss 1987). Thus, ill-structured problem solving skills might be constructed through experience in solving authentic, ill-structured problems (Bransford 1993). To this end, case-based instruction seems to be one of the most effective pedagogical approaches to ill-structured problem solving skills because it provides richer contexts for framing problems and facilitates experience-based knowledge construction (Williams 1992).

In teacher education, many researchers and educators have recently advocated case-based instruction as an effective method for helping preservice (and even inservice) teachers develop a sense of social and ethical responsibility, understand the contextually situatedness of teaching, and facilitate epistemological growth, by engaging preservice teachers in critical thinking and analytical reflections (Harrington et al. 1996; Lundeberg et al. 1999; Merseth 1996; Shulman 1992; Tippins et al. 2002). Recent instructional models and technology-enhanced practices provide richer and more meaningful case-based learning environments where students can construct active knowledge and build successful problem solving skills (e.g., Cognition and Technology Group at Vanderbilt 1993; Harris et al. 2005; Hernandez-Serrano and Jonassen 2003; Jonassen and Hernandez-Serrano 2002; Spiro et al. 1992).

The case-base learning for classroom management problem solving (CBL-CMPS) model

While there are a variety of ways to organize learning activities around cases in case-based learning, the CBL-CMPS model was created based on the adapted model of Jonassen’s (1997) ill-structured problem solving process, discussed in the previous section, and a modified model of Jonassen’s (1999) constructive learning environment design. The former guided us to identify what kinds of problem solving activities need to be facilitated, whereas the latter identified what kinds of learning resources need to be arranged and in which ways. To illustrate this modified constructive learning environment model within the framework of the CBL-CMPS, Fig. 1 is provided. As shown at the bottom of Fig. 1, Case Problems are the central component of the CBL-CMPS model. All of the learning activities are anchored to these case problems. Because different stakeholders can identify different problems in the same case, the next meaningful learning resources are Multiple Perspectives where students are exposed to diverse perspectives. In ill-structured problems, multiple solutions exist depending on the problems identified and the perspectives chosen. Thus, the next layer is the Experts’ Multiple Solutions, where students can see different solutions, solution steps, and the possible consequences of the solutions suggested by experts. Finally, the literature (i.e., general theories, principles, and other information) related to the given case problems are provided to students in the Theories and Literature layer so they can apply what they read to their problem solving process. All of these resources are situated within the traditional college classroom where students interact with their peers and instructor for the problem solving activities.

In this modified Jonassen’s (1999) model, the ill-structured problem solving process is applied to articulate the interface design scheme and learning activities. As shown in Fig. 1, the learning resources are provided through each stage in order to facilitate the essential process and the skills of ill-structured problem solving. Although each stage facilitates different skills required for ill-structured problem solving, students experience three cycles of the iterative process of whole problem solving throughout the stages, and this process deepens their problem solving skills.

A sample screen of the actual interface is presented in Fig. 3. In Stage 1, as shown in Figs. 2 and 3, real-world problems are presented and individual students are asked to identify problems and generate solutions. Most case problems provided were collected from practicing teachers and developed as an audio dramatic story. In this first stage, only the problem part of the case is presented. The purpose of this stage is to engage students in the ill-structured problems and to have them try to solve the problems on their own.

Fig. 3

Sample screen of the CBL-CMPS: Stage 1

Stages 2 and 3 are intended to expose students to multiple stakeholders’ perspectives on the problems and to experts’ different ideas for solutions. In Stage 2, Analyzing Problems, multiple opinions about what the problems are in the given case are presented by different stakeholders, including three principals, one parent, and three experienced teachers. Students are then asked to listen to all of the different perspectives and to refine their initial thoughts about the problems in the given case. The purpose of this stage is to encourage students to consider and incorporate diverse perspectives while identifying the problems in the given case. Likewise, in Stage 3, multiple solutions about the given case are provided by the three experienced teachers, and students are asked to refine their initial ideas for solutions. The purpose of this stage is to help students to explore diverse solutions and to justify the best solutions among their choices with a rationale.

In Stage 4, students are provided with readings that are relevant to the given case. Students are asked to finalize the problems they have identified and the solutions they have generated in the given case. The purpose of this stage is to facilitate students in applying what they have read, such as theories and principles, to the process of identifying problems, generating solutions, and justifying their positions.

Finally, in Stage 5, students are provided with the solution part of the case, which contains how the case was originally concluded by the practicing teacher who shared the dilemma. Also, comments on the conclusion of the case provided by the multiple stakeholders are presented to students. After reviewing the comments provided, students are asked to make their own comments on the case conclusion and reflect on the lessons they have learned from the given case.

We believe that this environment will deeply engage students in an ill-structured problem solving experience and will facilitate learning some of the essential skills and processes necessary for solving ill-structured problems. However, we suspect that engaging students in ill-structured problem solving itself is not a sufficient condition for them to build successful ill-structured problem solving skills. In addition, students may engage in learning and problem solving activities differently as they are exposed to the new learning environment that affords different learning activities and interactions (Gibson 1986; Young et al. 1999). We expect that a variety of scaffoldings and more explicit guidance, plus different instructional strategies, need to be embedded within this case-based learning environment in order for students to build successful ill-structured and complex problem solving skills (Ge and Land 2004; Land 2000; van Merrienboer et al. 2003). In order to develop better strategies, we need to have a clear understanding of how students interact with the new case-based learning environment. Thus, we stopped further development at this point, and we implemented this environment in a teacher education course in order to understand how these learning environments influence students’ learning activities and what kind of additional scaffolding or instructional strategies need to be embedded to make the learning environment more effective. In the following sections, we present two successive studies exploring two implementations of the CBL-CMPS. Then we discuss the implications of our studies and suggestions for future research.

Study 1: the first implementation

The purpose of Study 1 was to test the effectiveness of the CBL-CMPS on ill-structured problem solving abilities, to understand how students interact with the initial version of the CBL-CMPS, and to make further revisions to the CBL-CMPS based on the study results. This study was guided by the following two major questions:

• 1. Do the learning activities and the given learning resources in each stage of the CBL-CMPS improve students’ ill-structured problem solving skills during their case activity? By testing the gain effects on students’ problem solving performance with supports provided throughout each stage, we can diagnose each stage of the CBL-CMPS for further development.

• 2. Does the overall learning experience with the CBL-CMPS improve students’ ill-structured problem solving skills in a transfer test? This question tests students’ independent problem solving performance without any guidance. Thus, this provides information about the overall effectiveness of the CBL-CMPS.

Method

Research design

To answer the two research questions, a single-group repeated-measures design was employed. For the first question (a learning gain test in each stage), students’ written answers (problem identification and solution generation) in response to each case problem were collected three times at different stages (Stage 1, Stages 2 & 3, and Stage 4) during the exploration of each case. Students participated in the exploration of two cases during this study (2 cases X 3 times). For the second question (a learning transfer test), a pretest and posttest on a case problem were implemented before and after the review of two cases. The dependent variables consisted of the seven sub-skills of ill-structured problems solving as explained in the “Measures” section.

Participants

The CBL-CMPS was implemented for 3 weeks in a required undergraduate teacher education course taught by the second author for students who were second-semester juniors in the Spring Semester of 2005. The class had a cluster of 30 students enrolled in the early childhood education program at a large state-funded university in the southern United States. All 30 students were females in their early 20s. Except for one Filipino-American, the other 29 students were Caucasian-Americans. Among the 30 students, we received consent letters from 23 students who allowed us to use their case responses for our research (n = 23). All of the CBL-CMPS activities were part of the regular course assignments (15% of their final grade).

Materials and measures

We collected real classroom stories from interviews with local schoolteachers and casebooks (e.g., Shulman and Mesa-Bains 1990). We then selected and dramatized dilemmas related to the issues of a challenging child, conflicts with parents, rewards, homework, and instructional and grouping strategies. In this study, two of the cases, related to a challenging child and to conflicts with parents, were used for students’ case-based learning activities in the online CBL-CMPS environment. Throughout the stages of the CBL-CMPS, students were asked to answer two major questions: (1) what do you think are the problems in the given case situation, and why do you think so? (problem identification); and (2) what are the solutions to the problems, and why do you think so? (solution generation). Students’ written responses to the open-ended questions submitted online were the nature of their problem-solving data used for the gain test. For the transfer test, another dilemma, related to instructional and grouping strategies, with the same two questions (problem identification and solution generation) was used for collecting students’ problem solving data before and after the CBL-CMPS intervention.

In order to analyze the written responses that the students generated for the given ill-structured problems, seven sub-skills were identified as dependent variables (see Table 1).

Table 1 The seven sub-skills of ill-structured problem solving

Rubrics developed by the authors were used to assess the quality of each sub-area (scale: 0–4 points). Two trained raters (doctoral candidates) evaluated the blind data independently. Students’ answers that were rated with gaps (one or greater points) between the two raters were identified and discussed by the raters before they reevaluated the answers independently. The inter-rater reliabilities (Pearson r) for each of the dependent variables for the two different cases and the pretest and posttest ranged between .651 and .878. The average of the correlations calculated based on Fisher’s Z transformation was .764. The average score between the two raters was used for the final data set.

Procedure

Before implementing the CBL-CMPS, the students received an orientation session on how to use the CBL-CMPS for case activities in class. The students then completed an online pre-transfer test as homework. For the next four class sessions, the course focused on the issue of classroom management. The class met twice a week for 1 h and 15 min per class session. Throughout the four class sessions, the students studied two cases, related to a challenging child and to conflicts with parents, by spending two class sessions per case. For each case activity, the students were asked to complete Stages 1, 2, and 3 of the CBL-CMPS online and to submit their responses to the questions (problem identification and solution generation) posed at the end of each stage as a homework assignment. The students then brought their writing with them for face-to-face class discussions conducted in both small groups and the large group. Common procedures used for the small group discussions were, first, to take turns sharing individual writing, and then to discuss similar and different ideas presented by the group members. Each of the small groups reported their discussion results when debriefing in the large group, which focused on expanding the small group discussions by identifying common ideas as well as exploring different perspectives. These small and large group discussions about the first three stages were done in one class session. The students then checked out Stages 4 and/or 5 (one case did not have Stage 5) for another homework assignment in the same way that they did for the previous stages. The students brought their writing with them again to the following class discussion session. After studying two cases in the same manner, the students were then requested to revisit the pretest dilemma problem and revise their pretest essay online as homework. This particular dilemma was not discussed in class at all, and the revised essay was considered to be the posttest data.

Results and findings

Gain test: the effects of each stage in the CBL-CMPS on ill-structured problem solving

To diagnose the effect of each stage of the CBL-CMPS for further revision of the environment, a two-way (2 cases X 3 times) within-subjects MANOVA was conducted focusing on the Time main effect. The within-subjects factors were case activities with two levels (1st and 2nd Case Problems) and time with three levels (Stage 1–baseline, Stages 2 & 3–reviewing multiple perspectives, and Stage 4–reviewing readings). The MANOVA results showed that there is a significant Time main effect [Λ = .03, F(14, 9) = 21.18, p < .001, η ² = .97] meaning that students’ problem solving performances are significantly different among the different stages of the CBL-CMPS across the case activities.

To determine the effect of each stage of the CBL-CMPS on each sub-skill of problem solving, two-way (2 cases X 3 times) repeated-measures ANOVAs on each independent variable were conducted as follow-up tests to the MANOVA (Green and Salkind 2005). To control an inflated Type I error rate by using the Bonferroni method (Keppel and Wickens 2004), each ANOVA test was evaluated at the α level of .0071 (α = .05/7). To be free from the sphericity assumption, we used the multivariate criterion of Wilks’s lambda (Λ) and the multivariate eta-squared (η²) effect size. The descriptive statistics and the follow-up ANOVA results for all seven dependent variables (0–4 scales) are presented in Table 2. As shown in Table 2, the results revealed significant Time effects on all of the seven sub-skills for ill-structured problem solving with large effect sizes based on Cohen’s guidelines (1988).

Table 2 Means and standard deviations for ill-structured problem solving performance at two repeated measures in three successive cases activities (N = 23) and the results of 2 cases by 3 times repeated-measures ANOVA

Finally, follow-up repeated contrasts for the Time main effects were conducted to determine how each stage of the CBL-CMPS affected each problem-solving sub-skill differently across the case activities. To control for the familywise error rate across the two repeated contrasts at the α level of .0071, the Holm’s sequential Bonferroni procedure was applied (Green and Salkind 2005). Thus, the contrast result with the smaller p value between the two contrasts was tested at the .0036 level (α = .0071/2), while the other result was tested at the .0071 level (α = .0071/1). As shown in Table 2, the results of follow-up repeated contrasts showed that a significant improvement in the students’ scores on the seven sub-skills had been made from the baseline (Time 1) to reviewing multiple perspectives (Time 2) and from Time 2 to reviewing the given readings (Time 3) at large effect sizes with one exception. That is, the multiple perspectives scores in problem identification from Time 2 (M = 2.43) to Time 3 (M = 2.5) was not significantly improved [F(1, 22) = 7.97, p = .010, η ² = .27] at the α level of .0071 (α = .0071/1) while statistically significant improvement on multiple perspectives scores in problem identification [F(1, 22) = 50.29, p < .001, η ² = .70] had been made between Time 1 (M = 2.11, baseline) and Time 2 (M = 2.43, after reviewing multiple perspectives resources). This indicates that students’ major improvement in considering multiple stakeholders’ perspectives while identifying problems appeared after they were exposed to multiple stakeholders’ perspectives on problems. In contrast, more salient improvement on linking to theory in both problem identification and solution generation was made after students reviewed the given readings (between Time 2 and Time 3). Other independent variables seemed to be equally improved at each stage. Overall, these results imply that the individual instructional treatments given in each stage facilitated the intended learning outcomes across the case explorations.

Transfer test: the overall effects of the CBL-CMPS on ill-structured problem solving

The pretest and posttest data on a dilemma case were analyzed using a one-way repeated-measures MANOVA in order to examine whether the students’ ill-structured problem solving skills were transferred to a new case problem. The MANOVA results showed that there is a significant Time main effect [Λ = .27, F(7, 13) = 4.99, p = .006, η ² = .73] meaning that students’ problem solving performances are significantly improved from the pretest to the posttest.

To determine whether each of the seven sub-skills of ill-structured problem solving was transferred to a new case problem, one-way repeated-measures ANOVAs on each dependent variable were conducted as follow-up tests to the MANOVA. Each ANOVA was tested at the .0071 level (α = .05/7) by applying the same Bonferroni method used in the previous analysis. The results indicated that the students’ tendency to consider multiple perspectives in problem identification was significantly improved from the pretest (M = 2.07, SD = .30) to the posttest (M = 2.31, SD = .38), F(1, 19) = 13.70, p = .002, η ² = .42. However, justification, critical thinking, and linking to theory in problem identification did not show any statistically significant improvement from the pretest to the posttest at the .0071 level. In contrast, two of the sub-skills solution generation showed a statistically significant improvement: solution and justification [M_Pretest = 2.31, SD_Pretest = .23; M_Posttest = 2.65, SD_Posttest = .28; F(1, 19) = 20.4, p < .001, η ² = .52] and critical thinking [M_Pretest = 2.17, SD_Pretest = .24; M_Posttest = 2.59, SD_Posttest = .33; F(1, 19) = 20.69, p < .001, η ² = .57]. Although linking to theory in solution generation did not show a statistically significant improvement at our α level (M_Pretest = 1.51, SD_Pretest = .52; M_Posttest = 1.89, SD_Posttest = .72; F(1, 19) = 8.16, p = .01, η ² = .57), we noticed that the p-value was close to be significant.

Implications for the CBL-CMPS revision and the limitations of Study 1

Study 1 examined the effects of each instructional strategy provided in each stage of the CBL-CMPS on ill-structured problem solving and the overall effects of the CBL-CMPS on transfer in ill-structured problem solving. The results enabled us to identify four sub-areas that did not show any statistically significant improvement on a transfer test: justification and critical thinking in problem identification, and linking to theory in both problem identification and solution generation. We took these areas into major consideration in our revision (see Table 3 for a sample of the revision).

Table 3 Example of the initial and revised guidelines for scaffolding ill-structured problem solving in the CBL-CMPS

Study 1 results also indicated significant transfer effects in three sub-skills. However, due to the limitation of the single-group pretest-posttest design, it was unclear whether the significant effects were a result of the intervention or simply a result of test practice or maturation effects. In order to address this limitation, we used a quasi-experimental control-group design for the second study.

Study 2: the second implementation

The purpose of Study 2 was to confirm the gain effects of each stage found in Study 1 and to determine the transfer effect of the revised CBL-CMPS using a more rigorous method. The same research questions asked in Study 1 were used again for this study.

Method

Research design

The same single-group repeated design used in Study 1 was employed in order to confirm the gain effects of each stage of the revised CBL-CMPS during the case activities (research question 1). Students participated in the exploration of three cases (3 cases X 3 times) in Study 2. However, in order to determine the overall transfer effects of the revised CBL-CMPS (research question 2) while controlling the effects of test practice and maturation, a quasi-experimental pretest-posttest control-group design was employed.

Participants

The participants were 58 junior students from a total of 59 enrolled in either one of two sections of the same required course in the Fall Semester of 2006. This course was offered in the Early Childhood Education program at a large state-funded university in the southern United States. One section (30 students) of the course taught by the second author used the CBL-CMPS over the 3-week period (the treatment group) as part of regular course activities (20% of the total grade) while the other section (29 students) taught by another instructor did not use the CBL-CMPS (the control group) until we completed our data collection. All 30 students in the treatment group (n = 30) submitted their consent letters for us to use their case responses for our research. Except for one African-American male and one African-American female, the other 28 students were Caucasian-American females and the majority of the students were in their early 20s. Similarly, 28 students out of the 29 in the control group (n = 28) submitted their consent letters. All were Caucasian-Americans (1 male and 27 females) mostly in their early 20s. In order to tease out the effects of maturation and test practice using a control group, both groups should be equivalent in age, gender ratio, and cultural background (e.g., ethnicity and majors). We found that these two groups were similar in these areas.

Treatment: the revised CBL-CMPS

Based on the results of Study 1, more explicit guidelines designed to facilitate the sub-skills of ill-structured problem solving were provided in Stage 2 and Stage 3. Different guidelines were given depending on the issue relevant to each case. Also, the questions asked in each stage were slightly revised and additional guiding questions were added. Table 3 shows examples of the initial and the revised guidelines.

Procedure

Treatment group. Before implementing the CBL-CMPS with the treatment group, the class met at a computer lab for an orientation session. During this session, the authors explained how to access the CBL-CMPS and the assignments for the case activities. Then, unlike Study 1, the class had a 30-minute pretest session in the lab. During the pretest, the class listened to a dilemma posed by a local schoolteacher, and the students individually wrote and submitted their views of the problem(s) and possible solution(s) of the case online. Students were allowed to review the transcript of the dilemma if needed while completing their writing.

In the following six class sessions (1 h and 15 min per session) during a 3-week period, the students studied three cases in the same manner that had been applied in Study 1. Two of them were the same cases that had been used in Study 1, and the other was a newly-added case related to a homework issue in Study 2. The arrangements of homework and class activities for each case study were implemented in similar ways to those in Study 1. After these six class sessions, the whole class met at the same computer lab again for a 30-minute posttest. We asked the students to answer the same questions with the same dilemma used in the pretest. Unlike Study 1, however, the students were not given their pretest essay responses while completing their posttest essay, so the two tests could be more independent in order to control test practice effects. The dilemma presented in the tests was not previously discussed in class.

Control group. The class met in a computer lab for a 30-minute pretest. The same pretest used in the treatment group was conducted in the control group. Three weeks after the pretest, the class met in the same computer lab again for a 30-minute posttest. During the 3 weeks, they were not given the CBL-CMPS treatment. Instead, the class dealt with various topics related to organizing, planning, and developing instruction in early childhood classrooms by using the traditional lecture and discussion format.

Measures

The same seven dependent variables for ill-structured problem solving used in Study 1 were employed. The same two raters from Study 1 reviewed the blind data with the same 0–4 scale rubrics. The inter-rater reliabilities (Pearson r) for each of the dependent variables for the three different cases and the pretest and posttest ranged between .554 and .922. The average of the correlations calculated based on Fisher’s Z transformation was .790. The average scores between the two raters were used for the final data set.

Results and findings

Gain test: the effects of each stage in the CBL-CMPS on ill-structured problem solving

Aligning with Study 1, a two-way (3 cases X 3 times) within-subjects (treatment group) MANOVA was conducted to evaluate the effect of each stage of the CBL-CMPS on ill-structured problem solving during the exploration of the three cases. The MANOVA results showed that there is a significant Time main effect [Λ = .026, F(14, 16) = 49.69, p < .001, η ² = .97] meaning that the students’ problem solving performances are significantly different among the different stages of the CBL-CMPS across the case activities.

To determine the effect of the CBL-CMPS on each sub-skill of problem solving, two-way (3 cases X 3 times) repeated-measures ANOVAs on each independent variable were conducted with the same method used in Study 1. The descriptive statistics and ANOVA results for all of the seven dependent variables (0–4 scales) are presented in Table 4. The results were consistent with Study 1. Significant Time effects on all the seven sub-skills at the .0071 level (α < .05/7) were obtained with large effect sizes (see also Fig. 4).

Table 4 Means and standard deviations for ill-structured problem solving performance at three repeated measures in three successive cases activities (N = 30) and the results of three cases by three times repeated-measures ANOVA

Fig. 4

Problem-solving performance during case activities with the CBL-CMPS

The follow-up repeated contrasts for the Time main effects also confirmed that there was a statistically significant improvement on all seven sub-skills in each stage at the .0036 level (α < .0071/2) with large effect sizes (Cohen 1988). Most of the effect sizes obtained in Study 2 are larger than those in Study 1, indicating the improvement of the revised CBL-CMPS. These results confirmed that the individual instructional treatments given in each stage facilitated all of the seven sub-skills of ill-structured problem solving across the case explorations.

Transfer test: the overall effects of the CBL-CMPS on ill-structured problem solving

A one-way (treatment versus control) MANCOVA was used to test for differences between groups, with the posttest scores as the dependent variables and the pretest scores as the covariates. The results showed that there is a statistically significant difference between the treatment group and the control group in problem solving [Λ = .523, F(7, 38) = 4.95, p < .001, η ² = .48], meaning that problem solving skills were transferred to a new case problem.

To determine the transfer effect of the CBL-CMPs on each of the sub-skills of ill-structured problem solving, follow-up ANCOVAs on each dependent variable were tested at the .0071 (α = .05/7) level. The descriptive statistics are presented in Table 5. The ANCOVA results revealed statistically significant differences (at the .0071 level) between the treatment group and the control group in multiple perspective scores in problem identification [F(1, 50) = 33.81, MSE = .15, p < .001, η_p ² = .40] and critical thinking scores in problem identification [F(1, 50) = 10.27, MSE = .26, p = .002, η_p ² = .17]. As found in Table 5, there were also practically noticeable, but not statistically significant (at the .0071 level), differences between the two groups in justification scores in problem identification [F(1, 50) = 4.82, MSE = .25, p = .033, η_p ² = .09], solution and justification scores in solution generation [F(1, 50) = 5.60, MSE = .07, p = .022, η_p ² = .10], and critical thinking scores in solution generation [F(1, 50) = 7.49, MSE = .07, p = .009, η_p ² = .13]. However, the linking to theory scores in both problem identification and solution generation did not show any noticeable difference between the two groups in spite of the significant gain effects in the linking to theory during case activities. It could be explained partially by the limitations of our rating system. Unlike the case exploration situation where specific readings were given to the students, so that the reviewers clearly knew what the students had read, no reading was given in the transfer test situation, assuming that the students would use all of the literature that they had read previously in this class as well as in other classes. Thus it was difficult to discriminate different levels of students’ ability to apply certain literature to problem solving activities unless the students explicitly indicated specific theories and principles in their responses.

Table 5 Mean pretest and mean and adjusted mean posttest scores for the learning outcomes

Overall, these results indicated that two of the seven sub-skills gained during the CBL-CMPS experience were transferred to a new case problem while three of the other sub-skills showed positive potential for the transfer effects to some degree. To illustrate how students’ answers were improved from the pretest to the posttest, a sample answer from the problem identification written by one student is presented in Appendix A. We found that the CBL-CMPS and its stages facilitated the college students in (a) considering multiple perspectives, (b) critically reviewing situations and solutions in social and historical contexts, (c) developing coherent arguments, and (d) applying theories to their arguments to a certain degree while solving ill-structured problems.

General discussion

The two successive studies have allowed us to refine and improve the CBL-CMPS, to understand the potential and the limitations of the current CBL-CMPS, and to articulate further directions for our research and design. In this section, we explain important features of the CBL-CMPS that account for its effectiveness, and we discuss the results from the notions of the zone of proximal development (ZPD) and scaffolding. Finally, we conclude with future directions for research and design, and the implications of this research for college teaching and instructional design.

The CBL-CMPS model: real-world problems and graduated scaffolding and fading

An adapted model from Jonassen’s (1999) constructivist learning environment and the general process of ill-structured problem solving (Jonassen 1997; Sinnott 1989; Voss et al. 1991) guided us in developing the CBL-CMPS model and its environment. Learning resources and scaffolding in the CBL-CMPS were presented through the five stages. The results indicated that this environment was effective in improving students’ ill-structured problem solving abilities. We highlight two design considerations that may be most accountable for these successful results.

Engaging learners in real-world case problems

Most learning environments designed to facilitate real-world problem solving place questions, challenges, problems, cases, or projects in the center of the learning activities (e.g., Jonassen 1999; Schwartz et al. 1999). These allow learners to have ownership of their learning and to be deeply engaged in learning activities that result in meaningful learning. For the development of the CBL-CMPS, ill-structured, real-world case problems were collected from practicing teachers and placed in the center of the CBL-CMPS environment (Jonassen 1999). These case problems hold complexity, uncertainty, and dilemmas along with richer contexts. Thus, these cases motivate students to be engaged in problem solving activities and to experience real-world situations. The richer contexts of each case might play as a mental home or anchor where students could build and index their newly constructed knowledge in more meaningful ways (Bransford 1993; Schank 1999).

While the case problem itself seemed to play a significant role in the students’ initial motivation, the engagement of deeper thought processes and learning occurred when they were provided with different scaffolds and learning resources in the later stages of the CBL-CMPS environment. This is our next important design consideration.

Increasing the learner’s role with graduated scaffolding and fading

Successful problem solvers deal with problems while accommodating multiple stakeholders’ perspectives, integrating sound theories and principles, and examining taken-for-granted practices and ideas. It is obvious that students are frustrated with significant learning loads if they are asked to learn all the skills and attitudes simultaneously. In order to facilitate meaningful learning of these complex skills with less burdensome learning loads, we believed that students’ roles and responsibilities in solving problems need to be smoothly increased by graduated scaffolding while retaining the complexity of the original problems (Greenfield 1984; Puntambekar and Hübscher 2005; Quintana et al. 2004; Wood et al. 1976).

In the CBL-CMPS, each stage provided different scaffolding to facilitate one or more aspects of problem solving abilities while asking students to perform a whole task at their level. Due to the nature of ill-structured problems, there were neither right nor wrong solutions, nor was there a clear ending point to the problem solving. Throughout each stage, they were able to take on more responsibilities gradually while solving problems by being scaffolded to open their eyes to see deeper levels of complexities and different aspects of problems and problem solving activities. This phenomenon was explained by the patterns of the students’ performances during case activities in both Study 1 and Study 2. As shown in Fig. 4, for example, multiple perspectives, justification, and critical thinking performances began to increase during stages 2 and 3 when relevant scaffoldings and learning resources were provided. Yet more significant improvements in the linking to theory performances occurred during stage 4 when relevant learning resources were provided. Although in each stage we requested slightly different activities (e.g., what was highlighted/dismissed among different perspectives?), which may be considered as a way of changing the task (Quintana et al. 2004) in order to facilitate the development of certain problem solving abilities in a whole task context, their final task in each stage still remained the same (e.g., how would you reframe/solve the problem based on the new insights you gained in this stage?). Through a repeated cycle of activities in each stage with different cases, the students might be able to internalize problem-solving abilities.

It is important to note that in this environment the students began by solving problems without support. Then, different scaffoldings and resources to guide their problem solving activities were provided during each stage. When the students moved to solve a new case, they started again solving the given problem independently, which can be considered fading, gradually removing the support, to some degree (Collins et al. 1989). Although fading should occur gradually when students have internalized abilities to complete tasks independently, this environment repeated pushing and pulling the support regularly. We believe that these repeated cycles of giving and fading scaffolds in each case may also promote students’ internalization of problem solving abilities (McNeill et al. 2006).

Potentials and challenges of the CBL-CMPS model

Limitations of the CBL-CMPS

Although the results of Study 2 indicate the effectiveness of the CBL-CMPS to a certain degree, these results also demonstrate critical limitations of the environment with a 3-week implementation. In the gain test, as shown in Table 4 and Fig. 4, the baseline scores in all of the sub-areas for all three cases ranged between 1.25 and 2.33 at a 0–4 scale. The final scores during case explorations ranged between 2.22 and 2.89. In a transfer test, where no scaffolding was provided, the average scores of the control and treatment groups were 1.84 and 2.08 respectively. These scores indicate that students began with tendencies to simplify the given situation and identify problems from a single perspective, mainly the teacher’s perspective (e.g., Ben is a disruption to the other students in the class, see Appendix A). Their arguments and critical thinking were lacking in both problem identification and solution generation. Throughout the implementation, the students began to understand the complexity of given situations and to acknowledge the possibility of different interpretations of problems from multiple perspectives. However, they were still not successful in critically evaluating those multiple perspectives in relation to particular situations given to them and in integrating the multiple perspectives with different weights into building their own perspectives to approach the problems.

Further interpretation of the results

It would be meaningful to situate our results in two well-grounded epistemic development models: Perry’s (1968/1999) intellectual and ethical development scheme and King and Kitchener’s (1994) reflective judgment model. In fact, these two models guided us to characterize different levels of ill-structured problem solving performances in our evaluation rubric development. Perry’s original nine positions of epistemic development have been categorized as four major stages through refinement (Moore 2002)–dualism (black-and-white types of thinking and its variations), multiplicity (acknowledging uncertainty and accepting multiple opinions), contextual relativism (acknowledging the importance of contexts for meaning-making), and commitment within relativism (adding ethical and moral responsibility and professional commitments to the contextual relativism). In this scheme, our results indicate that the students seemed to begin at the late stage of dualism, where they tended to simplify situations from a single perspective and identify misbehaviors as problems, and then they moved to the early stage of multiplicity, where they tended to acknowledge multiple perspectives.

Likewise, in King and Kitchener’s (1994) reflective judgment model with three major stages including pre-reflective, quasi-reflective, and reflective thinking, we suspect that the students seemed to move from the late stage of pre-reflective to the early stage of quasi-reflective thinking through the intervention. King and Kitchener’s study reported that the traditional second-year college students usually were placed in the later stage of pre-reflective thinking. Considering the fact that we implemented our innovation in the second-year traditional class with a few exceptions, our results seemed to be aligned with their results to some degree.

A framework to conceptualize the challenges of the CBL-CMPS

In order to illuminate the potentials and challenges of the CBL-CMPS, we would like to take Vygotsky’s notion of the ZPD, which is defined as “the distance between the actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance and in collaboration with more capable peers” (Vygotsky 1978, p. 86). The individual’s ZPD is co-constructed with more capable others based on the scaffolding that more capable others can provide (Stone 1993). With the recent views of scaffolding that have been extended from humans to artifacts (Puntambekar and Hübscher 2005), we can possibly think of three levels of performance that determine one’s ZPDs depending on the types of scaffolds: independent level without scaffolds, scaffolded level with artifacts, and scaffolded level with human or a combination of human and artifacts (see Bidel and Fisher 1992; Sherin et al. 2004). To conceptualize our results with this framework, Fig. 5 was constructed based on the average scores across all case activities and all levels of performance measures from Study 2.

Fig. 5

Scaffolded performances with the case-based learning for classroom management problem solving (CBL-CMPS) environment

Points A (P_Pretest) and E (P_Posttest) indicate independent performance levels captured during the transfer test. Point B indicates an initial performance level in the CBL-CMPS Stage 1, Point C is a scaffolded performance level with artifacts (fixed guidelines and resources facilitating multiple perspectives, justification skills, and critical thinking) obtained at Stages 2 and 3, and finally Point D shows a maximum level of scaffolded performance with artifacts (fixed guidelines and resources facilitating the application of literature to problem solving) and human support (large and small group in-class discussions facilitating multiple perspectives and critical thinking after Stages 2 and 3 activities) obtained at Stage 4. While interacting with the CBL-CMPS, the students’ performances had been improved by employing different scaffolds from Point B up to Point D through Point C. Thus, the shadowed box created between Points B and D is the ZPD that was co-constructed between the students and the CBL-CMPS environment with a 3-week implementation. L₁, the distance between P_Pretest and P_Posttest, is the transfer of learning that occurred as a result of the 3-week implementation, which explains the effectiveness of this environment. P_Max (Point D) is the maximum level of performance that appeared during problem solving activities scaffolded with the CBL-CMPS. Then L₂, the distance between P_Posttest and P_Max, is identified as the ZPD constructed with the CBL-CMPS, but those skills are not fully internalized. This zone nicely indicates the potential of the CBL-CMPS. Likewise, P_Target indicates the target performance level we aimed for in our project. L₃, the distance between P_Max and P_Target, thus is the further ZPD that should be extended from the current ZPD, and this indicates the next challenge that the current CBL-CMPS environment is faced with.

Further directions for design and research

This conceptual graph in Fig. 5 immediately leads us to a question–whether L₂ (and even L₃) learning would occur by simply adding more cases to the current environment and implementing it with a longer period, or whether L₃ (and even L₂) learning requires additional scaffolds or significant revision of the current environments. Thus, future studies need to identify the relationship between the time of case learning experiences and the development of students’ problem solving abilities with the current CBL-CMPS. If the limitations of the CBL-CMPS with an extended time frame are identified, follow-up studies should address what significant scaffolding and instructional strategies need to be considered and how the current CBL-CMPS model and environment can be redesigned in ways that will gradually promote students’ problem solving or thinking abilities within a continuum of their development.

Implications for college teaching and instructional design

We conclude this paper by providing three implications. First, this CBL-CMPS is effective for helping teacher education students who are in the dualism stage or pre-reflective stage to move toward the multiplicity stage or quasi-reflective level. It is important to note that the students were promoted from upper dualism to lower multiplicity after only a 3-week implementation. This growth for college students would have required a longer period of time, from several months to a year (King and Kitchener 1994). In addition, our intervention was implemented in a regular classroom setting without requiring that significant changes be made by the instructor; thus, this CBL-CMPS can be considered as an effective tool and a feasible instructional model that can easily be used by other instructors.

Secondly, the CBL-CMPS model can be applied to teaching other disciplines that deal with the dilemma type of ill-structured problems (see Jonassen 2000, for details about the typology of problems) that generate conflicting perspectives, such as pollution dilemmas in environmental science or embryonic stem cells research dilemmas in biomedical engineering. Resolving these types of problems requires problem solvers to acknowledge, reconcile, and incorporate multiple perspectives when building one’s problem space; to develop reasoned arguments for justifying one’s position; and to apply scientific knowledge to the problem solving. In fact, most ill-structured problems have similar characteristics and require the aforementioned abilities. Thus, this model could also provide implications for other instructional design models promoting the abilities to deal with different types of ill-structured problems.

Finally, for instructional design, this study suggests that in order to create appropriate ZPDs with newly designed technology-enhanced learning environments, the iterative design-and-evaluation processes are necessary along with an incremental scaffolding approach. To obtain the maximum benefits from the CBL-CMPS environment, we began with minimal scaffoldings (or guidance) in the initial version of the CBL-CMPS (Study 1) while increasing scaffoldings (or guidance) to construct appropriate ZPDs with the target learners in the next iteration (Study 2). With this efficient approach, we can identify minimum support levels that maximize students’ learning and independent performance at their level. Unlike well-structured problems that have clear answers and the right procedures for solutions, and for which we can identify skills to learn more explicitly, ill-structured problems are more open and unclear in nature. Thus, new scaffolds and learning resources for promoting the students’ ill-structured problem solving abilities will be discovered as new needs emerge while the target audience takes unique, but relatively common, routes in the development of their ability to deal with ill-structured problems within this newly evolving learning environment.

Appendix A

Table 6 A sample of a student’s answer for the pretest and posttest from Study 2