Assessment of Real-World Problem-Solving and Critical Thinking Skills in a Technology Education Classroom

Abstract

In the twenty-first century, Science, Technology, Engineering, and Mathematics (STEM) workers need to be able to utilize their existing knowledge in science and mathematics and solve complex real-world (authentic) problems. Making timely decisions on what disciplinary areas contribute to the creation of a problem and thereby developing a reasonable solution requires critical thinking. Together, problem-solving, and critical thinking are touted as the most important skills (or abilities) needed by employees for tackling the challenges of this century. Also, having the necessary background in science and mathematics, being able to communicate well, and working with diverse teams comprised of people from all walks of life are all essential for those seeking employment. Teaching students to problem-solve in real-world STEM contexts is known to be complex and there are limited assessment instruments appropriate for classroom use. Ad hoc trial and error approach to problem-solving without the use of science and mathematics-based knowledge can be detrimental in the real-world context. Herein lies the challenge: faced with a design problem out of the context of the classroom, students may not readily recognize the STEM domains applicable to solving the problem. Engineering, through its hands-on and design-oriented approach, offers a platform in K-12 grades for integrating content and practices in the STEM fields and provides opportunities for higher-order learning. This is because higher order cognitive demands (as per Blooms Taxonomy, apply, analyze, justify, and create are higher-order thinking abilities) are made when engaged in design-based problem-solving experiences. Assessment of engineering problem-solving skills in the context of technology education or in engineering education in K-12 grades is problematic because it is time-consuming to design the lessons for each aspect of the design process and evaluate problem-solving, as problems encountered may be unique to each team or individual. Frequently, students engage in their own unique and sometimes ad-hoc trajectories in defining a problem and set about developing alternative solutions. Similarly, assessment is also time-consuming and cumbersome because of a multitude of reasons: e.g., teamwork and collaboration require peer assessments and rubrics, creativity and communication are multifaceted and require separate assessments for each facet, and there is no right or wrong solution thereby requiring subjective assessments based on many factors. For assessment in the classroom, while it is possible to prescribe a process to be followed and create benchmarks regarding every aspect of an engineering design process, doing so will eliminate the authenticity of student performance. Furthermore, students being grade-focused, tend to follow instructions closely which then inhibits their creativity and investigation using the iterative process to evaluate and optimize their solution. In this chapter, we describe an assessment instrument with metacognitive questions and a related rubric for scoring student problem-solving skills when faced with an authentic design challenge. Metacognitive questioning directs students’ thinking and responses to specific assessment items measured by the related rubric. This assessment instrument and its related scoring rubric can be used by teachers for delivering instruction and later for evaluating students’ performance by removing some of the subjectivity in evaluation.

1 Research Focus and Questions

Twenty-first-century learning outcomes occur when students can gain a deep understanding of science and math concepts, and use the content and practices of these disciplines with the content and practices of technology and engineering to solve problems situated outside the classroom. For this to occur, integration of the disciplines in the instructional approach is essential (National Research Council (NRC), 2009; Sanders, 2012). Engineering (through its characteristic design-based pedagogical approach) offers a platform in K-12 education for integration of content and practices in the STEM fields and provides opportunities for higher order learning because of higher cognitive demands in critical thinking and design-based problem-solving experiences (National Academy of Engineering (NAE) and NRC, 2014; Katehi et al., 2009; Sheppard et al., 2006; Wells, 2016). In an integrative STEM education program, five of the twenty-first-century skills (Collaboration, Creativity, Communication, critical thinking (CT) and problem-solving (PS) and citizenship) are expected to be the focus of instruction. However, CT and PS skills are not assessed in traditional science and mathematics standardized testing and rarely assessed in technology education. When students are tested for their problem-solving abilities in the traditional classroom the focus is on the extent of the correctness of the end result, and rarely, if ever, on the reasoning or procedures leading to the result (Docktor & Heller, 2009; Shavelson et al., 2003; Steif & Dantzler, 2005). Furthermore, the content knowledge tested is related to what has been recently taught in the classroom, which does not require the solver’s demonstration of metacognitive processes involved in CT that require selecting the discipline-specific content knowledge. Among the reasons behind the lack of focus on CT and PS in assessments is the lack of time to allow students an opportunity to explore various alternatives for designing a solution to the problem and pick the solution that best suits the various factors that affect success. Instead, students are provided instructions and materials for designing the solution which in turn lead students in the direction of a design solution that is the instructors’ prescribed solution. It could also be true that the instructors lack the engineering or science background necessary to provide students the depth of instruction needed to facilitate students’ explorations. Whatever the reasons, the result is that CT and PS skills are often not taught and/or assessed in the classroom which leads to students’ lack of experience in these skills.

This study was intended to address the lack of research to support the benefits of technology/engineering design-based learning (T/E DBL) as a signature pedagogical approach of integrative STEM education, for “conceptual attainment” (Zuga, 1995, p. 67) and “problem solving” (Zuga, 2000, p. 2) skill development as outcomes of technology education (Cajas, 2000; Kolodner, 2000; Zuga, 2000). Furthermore, the development of instruments for assessment was a necessary precursor to discovering the benefits of T/E DBL. A review of published literature in the first fifteen years of the twenty-first century resulted in identifying relevant research studies on students’ problem-solving (PS) skills in physics and its sub-discipline of mechanics, which contributed to the development of the instruments and the data collection in this study. In this exploratory and descriptive study, we attempted to add to the research base regarding the benefits of a T/E design-based pedagogical approach in developing problem-solving and critical thinking skills among students in a program that emphasizes STEM education.

The specific questions in the study were:

• 1. To what extent are students successful in using content and practices of engineering, science, and mathematics for solving an authentic design-based problem outside the confines of the classroom where the subjects were originally taught?

• 2. How are the key student abilities in CT and PS correlated to overall student success in solving the problem? Specifically, what is the correlational strength of the relationship of the key student abilities with their overall success in solving the authentic design-based problem?

2 Research Methods and Design

For the problem-solving activity in this study, a design-based problem was chosen. Design-no-make (DNM) was introduced by David Barlex in 1999 through the Young Foresight initiative (Barlex, 2003). At the time it was introduced, it was used to help focus students’ learning, and teachers’ instruction toward the design phase instead of making the designed product. To this day, Technology education classrooms tend to be more hands-on in their approach and students in those classrooms tend to be naturally more engaged in the making aspect of their design solutions. Research has shown that the DNM approach is valuable in helping students explore a wide range of design criteria, helps develop more understanding of the technological concepts, and that students enjoyed the experience as well (Barlex & Trebell, 2008). From an instructional perspective, the distractions of making the prototype were removed from the learning experience, and thus allowed students to explore various ideas and concepts in greater depth. Students spend more time on the best possible design solution through examination of various alternatives that could yield a solution and which of those would best satisfy the criteria for a successful outcome. The resulting design solution when implemented by the students in the “making” aspect also becomes a better process where students’ deeper understanding of their design results in increased efficiency and engagement in troubleshooting or problem-solving during manufacture. In the current study, this type of DNM challenge was suitable and instrumental in revealing students’ use of schematic and strategic knowledge domains (previously described) correlated to CT and PS skills.

This study examined student responses to a design-no-make challenge (DNMC) as a means for assessing their higher order thinking skills evidenced by their selection and utilization of science and math content to solve the problem described in the DNMC. The students were from a specialized school (the Academy) that has as its core objective, the cultivation of engineering ways of thinking and acting so that students would be better prepared for their college education and future careers in STEM disciplines. This school will be referred to as the Academy in this chapter. In K-12, especially in the middle school (6th–8th) and high school (9th–10th) grades, often tech-ed courses are offered for students to gain experience with technological and engineering (T/E) design, though not all students take these classes and there is not enough continuity of exposure to T/E design over the years. The DNMC with prompts was developed using a physics-based authentic problem typical of the types of problems encountered by humanitarian workers of Virginia Tech, engineering students in their work in Malawi (http://www.beyondboundaries.vt.edu/team-malawi.php). In discussions with the co-founder, Dr. Andre Muelenaer, and physics educators in secondary education, the design challenge was first developed.

The metacognitive question prompts were developed to elicit responses to demonstrate key student abilities (SAs) as identified in this study. The scoring rubric for the DNMC response was adapted from the rubric developed by Docktor (2009) to measure the key SAs identified as indicators of students’ abilities to solve authentic problems outside the classroom where the related subjects were first learned. The domain of the problem was situated in the physics and mathematics content areas. Both physics and mathematics were components of the curriculum in the Academy where the study was conducted. Researchers have found that lack of literacy in these two content areas (physics and mathematics) as contributing to the challenges faced by undergraduate students in engineering programs (Budney et al., 1998; Steif & Dantzler, 2005). One of the reasons students drop out or transfer out of engineering programs is that they are inadequately prepared to apply the foundational knowledge in these subjects (ibid). The DNM was therefore developed with a combination of physics (work and power) concepts and algebraic manipulation of formulae with attention to applied unit conversions.

Data were collected on students’ utilization of the design-based approach and acquired science and mathematics concepts in solving an authentic problem (DNMC) using a rubric, developed as part of this study, to score students’ responses to the presented challenge problem. The description and the challenge provided to students is shown in Fig. 10.1.

Fig. 10.1

Design-no-make challenge

The following questions asked of the students (in their design-challenge handout) were prompts designed to reveal student thinking:

• Q1 What is your understanding of the challenge described above? Describe using your own words, in a few sentences.

• Q2 Based on what you wrote above, draw a sketch to describe the scenario, and label the sketch to show the information provided above (e. g. the depth of the well is 10 m). Use variables for what you do not know.

• Q3 How could you determine the power of the water pump? State any laws and equations you would use and explain your strategy in a few sentences.

• Q4 Based on your response to question 2 go through the process you have outlined and show your calculations to determine the power of the pump (in horsepower).

• Q5 Based on your work in question 3, what is the power of the motor you need? Using the motor pricing information provided, which motor would you pick and how much will it cost?

For the last question, a price sheet was provided to the students, which included four options of submersible pumps with various horsepower options and their related prices.

Research into the nature and characterization of problem-solving over several decades has identified a set of student abilities requisite of success for solving authentic problems outside the confines of a typical classroom (Simon and Newell, 1972; Polya, 1980; Perkins & Salomon, 1989; Martinez, 1998; Jonassen et al., 2006). Specifically, these student abilities are (1) useful description, both symbolic and descriptive, (2) recognition and selection of relevant content applicable to the problem, (3) use of the principles and practices of specific content identified to solve the problem, and (4) adherence to a devised logical strategy for solving the problem. This research study used parts of the previously discussed studies to develop, validate, and utilize an assessment scoring rubric (Fig. 10.2) to score student responses.

Fig. 10.2

Final modified scoring rubric

3 Findings

Data collected was from scoring the students’ responses to the questions on the design challenge using the rubric. These responses provide insights into students’ utilization of the design-based approach and acquired science and mathematics concepts used to solve an authentic problem (DNMC) situated outside the confines of the classroom where those disciplines are taught. The metacognitive questions (listed before) in the DNMC were specifically developed to uncover students’ ability to restate a problem in words showing their understanding of the critical aspects of the provided information and to create a sketch, not unlike the idea of free-body-diagrams (FBD) taught in Physics and Mechanics. The third and fourth questions were used to direct students to identify and recall specific content in physics and mathematics that would help them solve the problem at hand. The last question was aimed at getting students to make a choice that would meet the criteria for a successful solution, specifically—suitably sized and priced, from the available choices. This also reveals if students had a logical progression of connected concepts and calculations that were used to select the final product.

Scorers would need to score the responses using the rubric to convert the qualitative responses to quantitative measures for each question that would result in a cumulative score. For this, two scorers trained on using the scoring rubric previously developed in a pilot study scored the students’ written responses to the DNMC from the main study. The scores obtained were then analyzed to answer the research questions.

The primary finding was that students immersed in an integrative STEM education program where the pedagogical approach is design-based learning (the Academy), performed significantly better (as assessed using their overall success score) in designing a solution to the design-no-make-challenge (DNMC) when compared with the performance of students in a traditional classroom. A secondary conclusion of this study was that four specific student skills (out of five identified in this study) that are collectively known as problem-solving skills, were strongly related to students’ performance in authentic problem-solving. The following sections describe the findings in detail.

Research Question 1: Overall Performance of Students in the program

Research Question 1 (RQ1) was associated with measuring the extent to which students were successful in solving an authentic design-based problem. The overall performance of students was assessed by the overall success score (OSS) achieved on the written responses to the DNMC. The sum of the individual scores for the five components representing five key student abilities (SAs) identified as the essential aspects of problem-solving and critical thinking resulted in the OSS. The five SAs are described in the following paragraph.

Useful Description reflected a solver’s skill in identifying the relevant information in the problem statement or design challenge that would be useful for consideration in developing the solution. Sketch reflects a solver’s ability to represent the information in the problem symbolically and graphically stating qualitative expectations and quantitative known values described in the problem. Student abilities associated with Specific Application of Physics and Application of Mathematics, reflect a solver’s ability to select relevant physics and mathematical content or principles and applying them to the specific context of the problem. Logical Progression reflects a solver’s ability to communicate reasoning and laying out a clear and focused strategy in achieving the goal.

From the data presented in Table 10.1, students achieved an average score of 16.45 points (out of 25 possible points) which represents a 65.8% score. The t-test results showed statistical significance to the higher mean overall performance score of students in the Academy (higher mean by 4.455; 95% CI, 1.78 to 7.13) when compared with a hypothesized mean which represented a 48% score). It can be inferred that the students in the Academy had a higher mean performance score than the hypothesized mean used as a benchmark. The hypothesized mean was obtained from the pilot study conducted in a traditional classroom (not within the Academy), where the students were completing the same physics course (using the same curriculum) as students in the Academy. The calculated effect size (Cohen’s d) of 0.8 indicated a large effect which implies that the strength of significance of the t-test is large enough to be significant.

Table 10.1 T-test for student scores

Research Question 2: Correlations between Overall Performance and Student Abilities

Research Question 2 (RQ2) was aimed at investigating the strength of the relationships between students’ overall performance (OSS) and each of the five key student abilities (SAs) in designing a written solution to an authentic problem as posed in the DNMC. What follows is a discussion of the conclusions drawn from the data analysis.

For a small sample size, as in this study, it is recommended that the adjusted correlation be calculated and used for interpretations of the strength of correlation between the two variables. A correlational statistic value greater than 0.5 indicates a strong correlation (Cohen, 1988). Table 10.2 summarizes the (Pearson’s) correlational strengths between the overall performance (OSS) and the five student abilities (SAs).

Table 10.2 Pearson’s (PPM) correlations between OSS and the five SAs

To reiterate, Sketch reflects a solver’s ability to represent the information in the problem symbolically and graphically stating qualitative expectations and quantitative known values described in the problem. Student abilities associated with Specific Application of Physics and Application of Mathematics, reflect a solver’s ability to select relevant physics and mathematical content or principles and applying them to the specific context of the problem. Logical Progression reflects a solver’s ability to communicate reasoning and laying out a clear and focused strategy in achieving the goal. Results of the correlational analysis showed that these four SAs were strongly correlated (significance at p < 0.05) to their overall performance (OSS) in designing a solution to the DNMC (presented in the adjusted correlation statistic column in Table 10.2). The resulting conclusion drawn from this analysis is that these student abilities or skills are critical to students’ successful problem-solving in situations outside the context where the specific content was learned.

Contributions of Specific SA’s toward the Variability in Students’ Overall Performance

The coefficient of determination is calculated as the square of the correlation coefficient. This statistic represents the percent of the data points that are closest to the line of best fit in the model and is a measure of how well the regression line represents the data. A higher coefficient is an indicator of better goodness of fit and can provide a good indication of prediction of the variations of one variable with respect to the other in the regression model (Howell, 2010). By no means is this an indication of causality, but it best represents a measure of variability in OSS that can be predicted by the variability of those SA’s. The calculated coefficient of determination for each of the five correlational analyses is summarized in Table 10.3.

Table 10.3 Pearson’s correlations and calculated coefficient of determination for the SAs

The most significant contributions of students’ abilities attributable to their overall success in designing a solution to the DNMC (from the regression model) come from their ability to select and utilize relevant content and practices in science (84%) and mathematics (91%), and from their ability to logically progress through the process (84%) to design a solution to an authentic T/E design problem. These SA’s were found to be strongly represented by the correlational (linear) model in this dataset (Table 10.2).

4 Contributions Toward Teaching and Learning

The primary conclusion of this study is that four specific student abilities (out of five identified and used in this research study) are strongly related to students’ performance in authentic problem-solving. The four specific abilities are—Sketch, Specific Application of Physics, Application of Mathematics, and Logical Progression. In other words, these skills are critical to students’ successful problem-solving in situations outside the context where the specific content has been learned. A secondary conclusion is that students immersed in an integrative STEM education program performed significantly better (as assessed using their overall success score) in designing a solution to the design-no-make-challenge (DNMC) when compared with a hypothesized mean for students in a traditional classroom.

These conclusions have direct implications for instruction in K-12 T/E design education, student learning and assessment, and engineering program design in secondary schools. One of the primary motivations for this research was the need for STEM literate graduates prepared for the problem-solving and critical thinking skills needed to tackle the challenges in the twenty-first century. US students lag in science and mathematics literacy and many studies have linked the lack of preparation of students to use high school science and mathematics knowledge to high rates of attrition in STEM programs at the undergraduate level (NCES, 2012; Pope et al., 2015; Budney et al., 1998; Steif & Dantzler, 2005).

The mean overall performance of students in the academy was shown to be statistically significant. The practical interpretation of this statistically significant difference does not necessarily mean that students’ overall performance in the DNMC or the situational problem-solving performance was good. The percentage score of mean OSS (65.8%) represents a C grade performance. The lowest OSS score achieved was 10 points (40% or F grade) and the highest was 23 points (92% or A grade). Nine of the eleven students in the sample achieved exactly or above passing grades (greater than or equal to 15 points or 60% and above). Therefore, the average student success score does not represent good performance in demonstrating problem-solving skills on the DNMC. Of course, this study was not designed to identify the strength of successful problem-solving skills, but rather, to identify the skills that strongly correlate too successful problem-solving. Yet, this lack of strong performance in problem-solving success raises curiosity. We speculate that many factors could have contributed toward the low-performance scores associated with this study. One possible factor could lead us to surmise that while the students in the academy were high achievers and generally maintain high subject grades, the overall problem-solving performance still lacking could imply a systemic instructional deficit of focusing instruction on the individual subject content knowledge proficiency embedded in the standardized testing culture.

A shift in the orientation of teaching toward the application of the content through authentic mini problem-solving activities where student’s gain more experience in utilizing the various subject-specific content in designing solutions to the problems presented. Additionally, it is rare that teachers focus project-based learning (PBL) on the integration of science, mathematics, and communication in authentic (real-world) design challenges in the US. This may be the result of a lack of time to plan or implement such instructional approaches. Teachers have back-to-back classes that they teach through the day and if they have time, it is usually to contribute their time to assisting with supervising lunchrooms or hallway discipline. Usually, the curriculum is also standardized and scripted by the school district and the State, which again, is focused on achieving standards that can be easily measured.

This study generated preliminary and limited data on the benefits of the technological/engineering design-based learning (T/E DBL) pedagogical approach within an integrative STEM education program as implemented in the Academy where engineering, mathematics, and science courses are integrated and progressively sequenced within the four-year curriculum. Further research on student learning, specifically on how students select and utilize science principles previously learned in solving T/E design-based problems, using a qualitative approach would provide additional insights into student learning and transfer of their learning. Such a study would potentially involve developing design-no-make challenges that are aligned with the various grade levels. These challenges would have to be evaluated for grade-level alignment, validity, and reliability by instructional experts in science, technology, engineering, and mathematics.

The correlational analysis between students’ abilities and overall performance revealed that specific skills involving selecting and utilizing science and mathematics content and practices were statistically significantly related to the overall performance in designing a solution to the Design-No-Make Challenge (DNMC) provided. The implications from these results are that when designing a solution to the DNMC, students’ abilities to recognize, recall, select and utilize science and mathematics content and practices are significant to successful T/E design-based problem-solving (outside the confines of the classroom where the science or mathematics was learned). This finding may have broader implications for classroom assessment and student learning. However, further research will be needed to explore those avenues for improving student outcomes.

From a practical perspective, the lower average (percentage) scores in the Specific Application of Physics (61.8%) and Application of Mathematics (52.8%) reveal that students need to improve their ability to recognize, select and utilize relevant science and mathematics content and practices in designing a solution to an authentic T/E design-based problem (such as the DNMC). This could imply that instructional strategies need to be further strengthened to help students learn to select and utilize science and mathematics in problem-solving in diverse contexts. There may be reason to also investigate the same skills in students in the lower grades to focus on helping develop these skills at an earlier grade level for all students. The statistically significant coefficient of determination (r²) associated with the student abilities of Specific Application of Science, Application of Mathematics and Logical Progression in contributing to the variation of their Overall Success Score corroborates the importance of these student abilities in engineering design-based problem-solving.

The rubric (refer to Fig. 10.2) developed in this study has the potential to be used as an assessment tool in the technology education classroom. The rubric has five scoring categories that relate to the five skills deemed to be critical for successful problem-solving in an authentic context. Teachers in Virginia (and probably everywhere in the US) are required to demonstrate student growth as a means of setting a performance goal for self-evaluation. Specific student abilities could be targeted, or the overall success score can be a benchmark for demonstration of student growth using pre-and post-assessments. While teachers in core disciplines use statewide testing for setting their performance goals, some technology education (Tech-ed) teachers use industry credentialing for specific technology for their performance goals, teachers in those disciplines or subjects that do not have credentialing (such as engineering in high school) can use the modified rubric developed in this study to set up performance goals and indicators.

5 Suggestions for Technology and Engineering Educators

To develop students’ ability to problem-solve in authentic (i.e., real-world) contexts, it is essential to utilize a technological and engineering design-based approach with an emphasis on two key aspects:

• (1) Select a design challenge that is based on real-world problems, and

• (2) Create an intentional focus on developing the five skills identified in this rubric.

First, instructors need to find authentic, relatable problems that are community-based so that it becomes relevant to the students. Examples might be based on water conservation, waterway clean-up efforts, and rainwater harvesting, storage, and distribution. There could be any other community-based problem that students may also help identify as a preliminary exercise in problem identification, which is also an integral part of engineering design. Taking time to do this will not only make the topic relevant to the community within which students are situated, but also inherently motivate students to become engaged as they relate to the context. This will require that the instructors provide some guided discussions that lead students to topics or areas relevant to the context of the lesson or unit.

Instructors could then utilize the five student abilities (Useful Description, Sketch, Specific Application of Science, Application of Mathematics, and Logical Progression) identified in this study as a method to develop their questions and questioning strategies. By utilizing a backwards-design approach, instructional goals could be aligned with the development of the specific student abilities which can then provide a focus on those aspects during classroom instruction.

In traditional tech-ed classrooms in the USA, assessments are focused on the skills outlined as competencies in the Career and Technical Education (CTE) course framework (CTE, 2015). This framework provides guidance and curricula set forth by the Department of Education, for vocational and technical education in the United States. Specific competencies in this framework are measurable skills to be attained by students who take the career and technology education coursework. As previously noted, a lack of focus on solving authentic problems is reflected in the competencies and therefore the shortcomings are reflected in students’ poor performance in any testing modalities that test problem-solving that are not directly related to a discipline (such as physics or mathematics) and/or in a particular discipline or classroom. Examples would be in competitions that throw out design challenges that test students’ problem-solving skills situated in a complex real-world simulation.

A follow-up study to this study could focus on creating templates for DNMC development and rubrics to add to the richness and usefulness of resources available for Technology-education (Tech-ed) courses and focus on solving authentic problems not currently addressed by the curricula. The rubric categories used to assess the problem-solving skills of students in this study could be further expanded for use by Tech-ed educators to prepare instructional goals for their teaching and to assess their students’ performance. Such a study could be designed to use a Delphi approach with disciplinary experts to develop content areas suitable for design-no-make challenges in the secondary school curriculum, and the related question prompts needed to effectively focus student thinking on the significant student abilities identified in this study.

Furthermore, the modified rubric could also be aligned for use with the design challenges developed. Such resources could help introduce the twenty-first century skill of “Critical thinking and problem solving” (P21, 2015a) more effectively within traditional Tech-ed courses and in non-Tech-ed science classrooms where engineering design is introduced. Classroom teachers who are accustomed to using project-based learning would have a ready-to-use rubric without the time commitment involved in creating a method of assessing their assignments given to their students. Additional refinement of the modified rubric used in this study would be needed to ensure its usefulness in the sciences and technology education, along with a study to establish the reliability of the rubric.