Introduction

As the knowledge and capabilities required for solving technological problems have become increasingly integrated and complex, the capability to apply interdisciplinary knowledge to solve complex problems is highly needed (Bybee 2013; Havice 2009). However, traditional school curricula have long been organized into separate subject areas. Many reports have warned that school education often failed to prepare our students to solve real-world problems because of the disconnected knowledge acquired from individual school subjects (Bybee 2013; National Governors Association [NGA] 2007; National Academy of Engineering [NAE] and National Research Council [NRC] 2014). For example, Kelley et al. (2010) found that even among students who had completed advanced courses in mathematics, very little mathematics was employed to solve problems during the design process. In addition, Taraban et al. (2007) noted that many engineering students lacked the flexibility to apply higher-order thinking skills and continued to use low-level conceptual knowledge to solve problems. In order to address this concern, Science, Technology, Engineering, and Mathematics (STEM) education has gained significant attention as a plausible solution for developing a better instructional approach to aid students in developing capabilities for solving complex real-world problems (NAE and NRC 2014), and the use of engineering design in instructional strategies has become a mainstream process for implementing integrated STEM curricula. Many studies in science education have also confirmed that using the integrative STEM approach through engineering design activities can improve student learning pertaining to the application of scientific and mathematics knowledge (Cantrell et al. 2006; Mehalik et al. 2008; Schnittka and Bell 2011; Wendell and Rogers 2013).

During the last two decades, engineering design has been an important integral element of the study of technology education (Kelley and Kellam 2009). Engineering design is a complex decision-making and problem-solving process. It requires the application of scientific, mathematical, engineering, and technological knowledge to use resources optimally for solving ill-structured problems. Additionally, during the engineering design process, higher-order thinking abilities are indispensable for analyzing problem factors, predicting the feasibility of different solutions, evaluating results, and optimizing the solution. In brief, the competency that technology education seeks through the teaching of engineering design is to help students gain flexible problem-solving capabilities and STEM literacy (International Technology Education Association [ITEA] 2000).

A long-standing problem is that many technology teachers do not emphasize the application of science and mathematics knowledge during the teaching and learning processes of engineering design activities (Kelley et al. 2010). Trial-and-error remains a common learning experience in technology education classrooms, thus many students do not acquire a real understanding of how science and mathematics knowledge should be used to support their design process and consequent solutions (Lewis 1999; Mativo and Wicklein 2011). In the last decade, many researchers have reflected on the problems occurring in the instructional practices and learning experiences based on engineering design curricula (Herschbach 2011; Kelley 2010; Ritz 2009; Sanders 2009), and they have stressed that the technology education community should pay more attention to enhancing the application of STEM knowledge during the engineering design process, especially the application of scientific and mathematics principles to engineering problem-solving (International Technology and Engineering Educators Association [ITEEA] 2009). In brief, the integrative STEM approach in technology education is aimed towards helping students acquire additional knowledge and capabilities in the application of science and mathematics when solving problems. Hence, this approach can help students understand the connections between the four STEM subjects, know how to bring these content areas together to generate more effective solutions, and consequently become better problem-solvers (Bybee 2013; Sanders 2009).

Recently, it has become widely recognized by technology education professionals in Taiwan that implementing STEM-oriented engineering design in high school, which follows a design and technology curriculum at the junior-high level, would be an appropriate curriculum progression. The engineering design approach at the high-school level would enhance the integration of knowledge pertaining to the areas of science, technology, engineering, and mathematics. In this study, we intended to initiate and provide a STEM engineering module that demonstrates how an integrative STEM curriculum could be designed and taught at the high-school level in Taiwan. This study is important for facilitating the efforts of technology instructors to acquire a better understanding of how the integrative STEM approaches can be more closely aligned with the relevant and respective content of science and mathematics.

Research goal and objectives

The following goal was established to guide this study: to examine the effectiveness of the application of the integrative STEM approach within engineering design practices in high school technology education in Taiwan. Firstly, we developed a STEM engineering module that emphasized the application of integrative STEM understandings and higher order thinking skills and, secondly, we conducted a quasi-experimental study designed to examine the following three objectives in order to address the research goal:

RO1:

Determine whether the STEM engineering module can improve students’ understanding of relevant conceptual knowledge

RO2:

Determine whether the STEM engineering module can improve students’ understanding and application of higher-order thinking skills

RO3:

Identify the key factors that influence student’s performance during the engineering design process and progress

Literature review

Key processes and core elements of engineering design

Engineering design is a useful instructional strategy for implementing integrative STEM education in K-12 education (Crismond and Adams 2012; Lantz Jr 2009). An appropriate engineering design activity should consist of an open-ended, highly iterative process that can provide a meaningful context for learning scientific, mathematical, and technological concepts and thus increase students’ systems thinking, modeling, testing, evaluating, modifying, and other higher-order thinking abilities (NGSS Lead States 2013; NRC 2009; Wendell and Rogers 2013). Engineering design involves several essential core elements and many studies have pointed out that the core elements within the curricula of secondary-level engineering design are systems thinking, recognition of constraints, predictive analysis, and optimization (Lewis 2005; Merrill et al. 2008; NRC 2009; NGSS Lead States 2013). A focus on these core elements can guide students to undertake more effective engineering design procedures (Merrill et al. 2008; NRC 2009). Within engineering design procedures, the identification of constraints is a key factor for defining engineering problems and developing solutions. Further, predictive analysis plays an important role during the processes of selecting the best solution, modeling the prototype, evaluating the results of testing, and that efforts in optimization have been made to modify, or redesign the results to find the most appropriate solutions (Asunda and Hill 2007).

Therefore, the conceptual framework for this study was based on the models of design that were aligned with engineering design processes (Householder and Hailey 2012; NGSS Lead States 2013; NRC 2009). The steps of the engineering design process used in this study were the following: (1) identify the problems, constraints, and limitations; (2) develop possible solutions; (3) perform a predictive analysis and model the prototypes; (4) test and modify the best prototype; (5) evaluate the final design; and (6) redesign and optimize.

Many studies have encouraged teachers to use engineering design as an instructional approach to help students apply effectively science and mathematics knowledge (Cantrell et al. 2006; Everett et al. 2000; Mehalik et al. 2008; Schnittka and Bell 2011). McCormick (2004) noted that the teaching and learning of engineering design might fail if teachers simply go through the design process without the use good explanations and emphasis on the various connections between science and mathematics knowledge, and the problem-solving processes. Numerous studies have also noted that the key differences between an engineering expert and a novice are the respective abilities to define problems and analyze and predict the feasibility of ideas (Atman et al. 2007; Crismond and Adams 2012; Perez et al. 1995). For example, Crismond and Adams (2012) generated a matrix that highlights the different design habits possessed by beginner and informed designers in order to help teachers identify, diagnose, and explain ineffective design processes used by students. Meanwhile, many ineffective design habits are strongly related to the predictive analysis abilities, and most of these habits occur due to a lack of appropriate STEM knowledge and higher-order thinking skills (Crismond and Adams 2012; Merrill et al. 2008). Therefore, there is an urgent need to assist high school students to develop their STEM knowledge and higher-order thinking skills pertaining to the predictive analysis abilities for engineering design problems.

STEM knowledge and higher-order thinking skills

STEM knowledge relevant to the project design and higher-order skills are important integral elements for solving engineering problems (NRC 2009; NGSS Lead States 2013). STEM knowledge consists of both conceptual and procedural knowledge. The definition of conceptual knowledge includes knowledge about particular concepts associated with particular technologies as well as knowledge about general principles (McCormick 2004). To be more specific, conceptual knowledge includes an understanding of broad concepts and recognition of their various applications (i.e. scientific principles, mathematical formulas, and mechanisms). Procedural knowledge is the necessary knowledge and thinking skills relevant to the design process, such as problem-solving, modeling, predictive analysis, and optimization. It is focused on crucial aspects of practice and implementation (Leppävirta et al. 2011; McCormick 2004; Rittle-Johnson and Alibali 1999). The higher order thinking skills of engineering design can be defined as a combination of the problem solving and critical-thinking abilities of procedural knowledge. These higher-order thinking abilities are all relevant to basic logical thinking skills which play an important role in integrating STEM knowledge.

Conceptual and procedural knowledge and higher-order thinking skills are complementary during the design and problem-solving processes (Schneider et al. 2011). In addition, Baroody et al. (2007) emphasized that conceptual knowledge should be linked to an extensive knowledge framework and that procedural knowledge should be highly flexible, thus fostering the flexible application of conceptual knowledge to adjust the processes of problem solving where necessary, analysis of different possibilities, and finding the best solution. Simply stated, students need to know not only, “what” and “how,” but also “why” they require STEM knowledge, along with their higher-order thinking skills to put STEM into practice. To solve engineering problems, students need to be able to identify constraints, collect and analyze data and information, identify useful applicable information and generate possible solutions, and then develop carefully thought out and reasoned actions to implement the best solution. Therefore, higher-order thinking skills such as assumption, induction, deduction, interpretation, and argument evaluation are required (Yeh 2003).

Methods

Research design

In order to gather information relevant for the various inquiry aspects of the research goal and objectives, a quasi-experimental study (Campbell and Stanley 1963) was adopted in order to examine and compare the learning effects evident between the STEM engineering module and the technology education module. As random assignment of individual students to new classes is not feasible in the education system of Taiwan, an intact class was the unit of the quasi-experimental design. Ten intact classes, which came from two high schools, were chosen. Five intact classes at one school were assigned to the technology education module as the control group, and five intact classes at another school were assigned to the STEM engineering module as the experimental group.

Care was taken to ensure that an appropriate comparison was attained between the two groups. Two college professors and five high school teachers with extensive teaching experience in the field of engineering and technology education comprised the research team. The team worked together cooperatively to design and develop both learning modules (STEM engineering module and technology education module) and the instruments (e.g. the Mechanical Conceptual Knowledge Test, the design project rubric, interview to participating teachers and students, and the High-order Thinking Skills Test) used in this study. Meanwhile, two technology teachers, who were also members of the research team, assisted with the conduct of the teaching experiment in their schools. Thus, both of them clearly understood the content of two instructional modules. They also had a full, continuing discussion about their respective instructional approaches to limit other differences that may affect the experiment.

Participants

The research study was conducted during the 2012–2013 school year. Two public high schools in Taipei, Taiwan, with a total sample of 332 students aged between 16 and 17 years old, participated in the study. The experimental group (STEM engineering module) included 171 students in five classes, and the control group (technology education module) included 161 students, also in five classes. These two schools were chosen mainly because they provided their teachers with greater flexibility in curriculum construction, compared with most public schools in Taipei. The male to female ratio in both schools was about 50 %. Both of the schools recruit students who attain the top 10–15 % average scores at Taiwan’s national exam for junior high school graduates, i.e. The Basic Competence Test for Junior High School Students. Therefore, the participants in both schools were deemed to be quite similar in academic performance. The two teachers who helped conduct the teaching experiment were certified to teach technology education at the secondary level. All teachers had a master’s degree in technology education and each one had more than 10 years experience in teaching high school technology education.

STEM engineering module and technology education module

To ensure that both modules were aligned to the same content, and to eliminate factors other than the instruction module that might interfere with the results, the content knowledge of the mechanism and each sub-concept component item was pre-defined, as shown in Table 1. Each module consisted of several instruction units (e.g. the lever scales unit, the gear-set model unit, etc.) and a design project (the mechanism toy). Both of the modules provided teacher lesson plans, assessment, instruction materials, and the necessary equipment. The total instruction periods were 10 weeks in duration (2 h of class each week). In both groups, the instruction units were conducted during the first 4 weeks. Students then had the remaining 6 weeks to design and construct a toy with various mechanical structure types (e.g. linkage, crank shaft and connecting rod, gears, cam mechanism, etc.), using LEGO parts and materials from the instructors (as shown in Fig. 1). All students were provided the same design brief which introduced the problem, specified the design constraints and limitations of the problem solution, and explained how the students’ solutions would be evaluated.

Table 1 Mechanism content knowledge taught in both modules
Full size table
Fig. 1
figure 1

Samples of the mechanism toy design project

Full size image

The STEM engineering module

The basis of this STEM module was derived from integrative STEM education which is a technological/engineering design-based learning approach that intentionally integrates content and the processes of science and mathematics with the content and processes of technology and engineering (Sanders 2009). The STEM knowledge which was taught in the module was pre-defined, as shown as Fig. 2.

Fig. 2
figure 2

STEM knowledge in STEM engineering module

Full size image

In order to help the high school students acquire a better understanding of how to use STEM knowledge during the engineering process, the research team developed four instruction units to help students learn to use their knowledge for design, predictive analysis, construction, evaluation, and redesign of mechanism models. Each instruction unit integrated one or two mechanism concepts (Table 1) as well as related science and mathematics knowledge. These instruction units consisted of lever scales, a cam toy, a gear-wheeled range finder, and a gear set, as shown in Table 2. When teaching the four units, the teacher would use hands-on LEGO models as well as virtual computer simulation models to explain the application of the respective mechanism and STEM knowledge consisting of scientific principles, mathematical applications, in order to frame structure design, and assembly methods. The example of the lever scales unit is depicted shown in Fig. 3.

Table 2 The brief of four LEGO instruction units
Full size table
Fig. 3
figure 3

Example of the lever scales unit in STEM engineering module

Full size image

The implementation steps of these four instruction units (Table 2) were designed as follows. First, the teacher explained the mechanism functions and STEM knowledge via virtual simulation models, and then students worked to assemble the physical, hands-on, LEGO model. The virtual simulation and physical hands-on experiences aided students in gaining knowledge of the mechanism and learning how to apply STEM knowledge in the design, predictive analysis, and improvement of the mechanism structure. Secondly, students were asked to make a predictive analysis of the dynamics of the mechanism under different conditions and answered the assigned questions. For example, in the lever scales unit (Table 2), the teacher would ask questions like “what movement will be made through parallel linkage? What is its function?” or “If you change weight or the position of weights, what is the impact on the scale platform?” Finally, students had to re-design and optimize the original physical LEGO model to complete an advanced challenge. For example, in the gear-set instruction unit, students were asked to optimize or re-design the model to produce three different speed ratios and change the motion direction, as shown in Fig. 4. In order to ensure the quality of instruction of STEM engineering module, we provided the participating teachers with 2 weeks of training to explain clearly the content and process of the module.

Fig. 4
figure 4

Example of the advanced challenge in gear set unit

Full size image

Technology education module

The technology education module focused on a design process that was aligned with Taiwan’s technology curriculum content standards (Ministry of Education of Taiwan 2010). In Taiwan, technology education is a required course for senior high school students aged from 16 to 18. The content of technology education in senior high school includes: (1) Technology and Society, (2) Technological World, and (3) Innovation and Design (Ministry of Education of Taiwan 2010). In brief, the Taiwan’s technology curriculum content standards are similar to the Standards for Technological Literacy of USA (ITEA 2000). In this study, the technology education module was modified and re-organized from an innovation and design activity which has been taught by the participating teacher for many years.

The design process of the technology education module emphasized selecting a design idea, testing the idea through project building, and making final design decisions based on a general design process. In the technology education module, the instruction that the teacher typically employed was to illustrate the design process, mechanism, and manufacturing skills in the technology class. During the instruction, the teacher explained the different types of mechanisms via three instruction units, which were also aligned with the content of Table 1. Each instruction unit was designed with a physical LEGO model in mind that could demonstrate clearly the functions of different mechanisms and was similar to what students would develop in their design project. For example, the work in Fig. 5 consists of a crank and connecting-rod mechanism, screw rotation mechanism, gear transmission mechanism, and cam mechanism. It can be used to display several different movements, such as straight up and down, rotary motion, and horizontal reciprocating. The teacher would require students to assemble the same hands-on LEGO models after explaining each instruction unit. Through imitation and hands-on practices, students were able to learn the functions and operation of the relevant mechanism.

Fig. 5
figure 5

Example of a LEGO work in technology education module

Full size image

Instruments

Mechanical conceptual knowledge test

The Mechanical conceptual knowledge test (MCKT) was developed by the research team to assess students’ understandings of the conceptual knowledge associated with a relevant mechanism. The test consisted of 14 multiple-choice items and two open-ended items. It was designed as three subtests: basic mechanical knowledge, science and mathematics principles applications, and problem prediction and analysis. The basic mechanical knowledge and science and mathematics principles application subtests were entirely multiple-choice items.

The basic mechanical knowledge subtest was designed to assess students’ knowledge of mechanisms. The science and mathematics principles application subtest was designed to assess students’ knowledge of scientific principles and application of mathematics to mechanism concepts. Consisting of open-ended questions, the problem prediction and analysis subtest was designed to assess students’ knowledge and skills for analyzing and predicting different engineering design problems. Table 3 shows sample questions for each subtest. A panel of specialists, including two university professors and five high school teachers, established the content validity of the MCKT. We also conducted a pilot test, and the results revealed that the average discrimination index of the MCKT was 0.40 and that the average difficulty index was 0.47. The Cronbach α coefficient of the total test was 0.77. The subtest scores and total score were significantly correlated, and the correlation coefficient was 0.64–0.76, p < 0.001.

Table 3 Example questions on the MCKT
Full size table

The design project

This project enabled students to design a mechanism toy as shown in Fig. 1. The construction of the toy produced a concrete result that could help the researchers understand how the students applied and integrated their STEM knowledge and thinking skills. The design project was assessed and scored by three technology teachers using a five-point rubric with three items generated by the research team. The rubric items were mechanical design, working function of the toy, and materials/tools used, as shown in Table 4. The maximum score for the design project was 15. To establish the rater reliability of the assessment, the research team discussed and reached a consensus view related to assessment standards. In the next step, the teachers assessed the students’ design projects individually. The result of the Kendall coefficient of concordance was 0.84 (0.000), indicating a significant correlation among the teachers’ assessments.

Table 4 Assessment rubric of the design project
Full size table

Higher-order thinking skills test

The higher-order thinking skills test (HTT) was developed by the authors and research team to measure students’ thinking skills for problem prediction and analysis in the engineering design process. The test consisted of 25 multiple-choice items that aligned to the following thinking skills: assumption, induction, deduction, interpretation, and argument evaluation. Each item consisted of one statement and three multiple-choice answers, as shown in the Exemplar below:

  • John designed a cam mechanism toy that could move up and down. During testing, he found that the follower often deviated from both sides of the cam; therefore, the mechanism toy did not work smoothly. After he increased the contact area between the cam and the follower, John solved this problem successfully.

    1. A.

      In this problem, increasing the contact area between the cam and the follower is the only solution.

    2. B.

      In this problem, increasing the contact area between the cam and the follower prevents the follower from slipping down from both sides of the cam.

    3. C.

      Using the cam mechanism to provide an up-and-down movement for the mechanism toy is a poor choice.

These questions were intended to test the students’ thinking skills used to explain the reason for an engineering design result, or action. The possible total score of the HTT is 25 points. The content validity of the HTT was also examined by two college professors and five high school teachers. We also conducted a pilot test to establish the internal reliability. The results of the pilot test revealed that the average difficulty index was 0.66, and the average discrimination index was 0.40. The Cronbach α coefficient of the total test was 0.72. The subtest scores and total score were correlated significantly, and the correlation coefficient was 0.61–0.74 with p < 0.001.

Semi-structured interview and informal observations

To support the analysis and discussion of quantitative data, this study also employed a semi-structured interview to obtain the reflections and opinions from the teacher who taught the STEM engineering module as well as the opinions of five students from the experimental group. The interview was directed by several questions which were in accord with the research objectives. The teachers’ interview included questions that probed: (1) the primary problems in teaching the STEM engineering module, (2) the key factors that affected students’ learning performance, and (3) the feasible instructional strategies that would improve the teaching. The students’ interview included questions that were designed to assess: (1) the key knowledge that was learned from the STEM engineering module, and (2) the primary problems encountered when designing and constructing the artefact during the design project. The researchers were also present in the classes in order to observe how the teacher taught the instructional module, and how the students responded to the teaching and the content of the module.

Procedure

Before the formal study commenced, the two participating teachers provided students with a semester of practical training in material fabrication processes (such as woodworking, welding, machining, use of mechanical fasteners, cutting, and bending metals). In the second semester, the MCKT pretest was conducted before the teaching experiment. The pretest was followed by 10 weeks of the experiment intervention, which was, in turn, followed by the MCKT posttest and the HTT posttest. Within the 10-week period, the participating teachers implemented the instruction units during the first 4 weeks. Students in both groups were then asked to complete a mechanism toy design project during the remaining 6 weeks. The students were provided with LEGO parts and materials such as wood, metal, plastic, and were able to access other resources that were available in the technology classroom.

Data analysis

For quantitative data analysis, the MCKT and HTT were used as dependent variables. Dependent t tests and an analysis of covariance (ANCOVA) were used to assess check whether progress was statistically significant between the pretest and posttest and between the two groups for the MCKT and HTT. Consequently, student performances in their design project were used as the dependent variables, hence independent t tests were used to compare differences between the two groups. A correlation analysis was also conducted to examine the relationships between students’ conceptual knowledge, higher-order thinking skills, and design-project performance. Additionally, to determine if there were any major advancements among students working in the STEM engineering module, students in the experimental group were divided into three sub-groups according to their prior STEM knowledge, and an analysis of variance (ANOVA) was performed to examine if there were differences among the three sub-groups. Meanwhile, all qualitative data were analyzed. The analysis was designed to categorize, analyze, and summarize the teachers’ and students’ reflections on their teaching and learning. This would assist the researcher to identify the problems that were not found from the sole use of quantitative data.

Findings

Performance on mechanical conceptual knowledge

Before the results of differential conceptual performance between the groups were gathered, an independent t test performed for the pre-MCKT scores showed no statistical significant difference between the two groups (t = 0.15, p > 0.05). Hence, the two groups commenced with similar incoming mechanical concept understandings and STEM knowledge. The means and standard deviations for the pretests and posttests of the three subtests for the experimental and control groups are shown in Table 5. A further dependent t test analysis revealed that the experimental group achieved significant progress (p < 0.01) on all subtests of the MCKT, including basic mechanical knowledge, science and mathematics principles application, and problem prediction and analysis. In the control group, student posttests for basic mechanical knowledge, and science and mathematics principles application subtests were significantly higher than the pretests. However, there was no significant difference in the pre–post analysis results for the problem prediction and analysis subtest.

Table 5 Dependent sample t test of mechanical concepts knowledge test
Full size table

To determine the effect of the two modules, ANCOVA was used to examine whether any significant differences occurred in the MCKT between the experimental and control groups. The posttest for the MCKT was used as the dependent variable, and the pretest was used as the covariate. From the sources-of-variance result, it was found that there were significant differences between the two groups on the three subtests of the MCKT. As presented in Table 6, the result of the ANCOVA for basic mechanical knowledge was F(1, 329) = 29.36, p < 0.01, η 2 = 0.08; the result of the ANCOVA for science and mathematics principle application was F(1, 329) = 16.37, p < 0.01, η 2 = 0.05; and the result of the ANCOVA for problem prediction and analysis was F(1, 329) = 34.46, p < 0.01, η 2 = 0.10. Furthermore, the result of the ANCOVA for the total score was F(1, 329) = 66.14, p < 0.01, η 2 = 0.16. Based on Cohen’s work (1988), the effect size of the MCKT total scores was large, which indicates that the explanatory power of the independent variables on the dependent variable was high. Based on the analysis results of t test and ANCOVA, these findings revealed that the STEM engineering module promoted significantly better learning of mechanical concepts and STEM knowledge than did the technology education module. In addition, the largest advancement of the experimental group was in their problem prediction and analysis performance. The control group did not show significant improvement in problem prediction or analysis performance. Thus, this result emerged as the key difference between the two modules.

Table 6 Analysis of co-variance for MCKT by group
Full size table

Performance on higher-order thinking skills

To examine the students’ performance in higher-order thinking skills of the engineering design process, an ANCOVA was conducted to examine whether any significant differences occurred between the experimental and control groups. The HTT was used as the dependent variable, and the Critical Thinking Test-Level I (CTT-I) was used as the covariate to control the effect of the students’ prior thinking skills. The CTT-I, developed by Yeh (2003), consists of 25 multiple-choice items designed to examine general critical thinking skills (e.g., assumption identification, induction, deduction, interpretation, and argument evaluation). The CTT-I has high reliability and validity, which makes it appropriate for use as a covariate to control the effect of prior thinking ability. The average HTT score of the experimental group was 19.09, and that of the control group was 15.33. The result of the ANCOVA showed that there was a significant difference between the two groups: F(1, 329) = 70.58, p < 0.01, η 2 = 0.18 (see Table 7). Based on the work of Cohen (1988), the effect size of the HTT was large. The result revealed that the experimental group students acquired more complex higher-order thinking skills compared to the control group students. That is, the STEM engineering module can improve effectively students’ higher-order thinking skills in the engineering design process.

Table 7 Analysis of co-variance for HTT by group
Full size table

Performance on the design project

For the design project, the results in Table 8 reveal that the experimental group students outperformed the control group students in all three categories of the design project (i.e., materials and tools used, mechanical design, and working function). According to the assessment rubric, most of the experimental group students could accomplish an understanding of the conditions, and complete their design project.

Table 8 t test results of the students’ design project
Full size table

Relationships between conceptual knowledge, higher-order thinking skills, and design project performances

Based on the analysis of results above, we found that the performance of the experimental group of students was significantly better than that of the control group in every assessment. In order to understand the relationships between conceptual knowledge, thinking skills, and design project performances, a correlation analysis was conducted to examine the MCKT, HTT, and design project scores of the experimental group. The results are shown in Table 9.

Table 9 Correlations among the MCKT, HTST, and design project (experimental group)
Full size table

As presented in Table 9, the correlation between the MCKT pre- and posttest scores (r = 0.22, p < 0.01) and the correlation between the MCKT pretest and the HTT (r = 0.19, p < 0.05) were significant. The correlation between the MCKT posttest and the HTT (r = 0.22, p < 0.01) was also significant. However, both the correlations between the MCKT (pre- and posttest) and the design project total score were not significant. A further examination of the sub-items shows that the correlation between students’ problem analysis performance on the MCKT posttest and the HTT (total) was significant (r = 0.19, p < 0.05). Additionally, the correlation between working function in the design project and their problem analysis performance on the MCKT posttest was also significant (r = 0.16, p < 0.05). Overall, the correlation analysis indicated that the students’ prior knowledge may influence their learning performance. However, by combining the findings above, we found that problem analysis ability through integrated STEM knowledge might be the factor influencing the students’ design project performance.

The major benefit of the STEM engineering module

To ascertain the influences of prior knowledge and the STEM engineering module, students in the experimental group were divided into three sub-groups according to their respective MCKT pretest scores, which were deemed to be representative of their prior STEM knowledge. These groups were identified as follows: (1) the low prior STEM knowledge group, consisting of the bottom 27 % of students; (2) the high prior STEM knowledge group, consisting of the top 27 % of students; and (3) the middle prior STEM knowledge group, comprising the remaining 46 % of students. Subsequently, a single-factor analysis of variance (ANOVA) was performed to determine differences between the student sub-groups in terms of conceptual knowledge, higher-order thinking skills, and design project performance. The results of the ANOVA (see Table 10) indicated that students’ prior knowledge differed significantly among the three groups. However, the performances on the MCKT posttest, HTT, and design project were not significantly different among the three groups. All groups showed significant improvement in their MCKT scores. Nevertheless, we found that the low and middle prior STEM knowledge groups made large advancements in conceptual knowledge. On further review of the correlation analyses described above, it can be tentatively claimed that students’ prior knowledge might influence their conceptual knowledge, higher-order thinking ability, and design project performance. Therefore, the results presented in Table 10 show that the STEM engineering module can help promote in students with low prior knowledge an enhancement of their understandings and applications of STEM knowledge and higher-order thinking skills, especially in their problem prediction and analysis abilities.

Table 10 ANOVA analysis among the three different prior knowledge groups (experimental group)
Full size table

Discussion

The research goal of this study was to examine the effectiveness of an integrative STEM approach within engineering design practices. The purpose of the integrative STEM engineering design module was to improve high school students’ STEM knowledge and higher-order thinking skills during the engineering processes. In brief, the main questions addressed in this study were the following: “Can integrative STEM knowledge and higher order thinking skills pertaining to engineering design be fostered through the use of engineering design modules?” and “What were the major benefits?” Two teaching modules were compared in the research design. In the technology education teaching module, instruction was focused on the use of a traditional technology education approach, which included lectures, demonstrations, and hands-on activities. The STEM engineering design teaching module was aligned with engineering design processes and supplemented with integrative STEM knowledge via the use of virtual computer simulations and physical models. The STEM engineering design teaching module not only emphasized increasing students’ understanding of mechanism concepts but also focused on promoting students’ abilities to use scientific and mathematics knowledge to predict, analyze, and solve engineering problems.

The t test and ANCOVA results showed that students in the STEM engineering teaching module outperformed their counterparts in conceptual knowledge and understandings. In addition, the subtest analysis of the MCKT showed that the largest difference between the experimental group and the control group mean scores was in problem prediction and analytical skills. The experimental group students demonstrated a significant advancement on their problem prediction and analysis subtest mean scores (Table 5), whereas the control group students showed little or no improvement in the subtest scores for problem prediction and analysis. This may be the factor causing the significant difference in the ANCOVA result.

Numerous studies have noted that problem prediction and analysis are core abilities of engineering design (Atman et al. 2007; Crismond and Adams 2012; Merrill et al. 2008; Perez et al. 1995). In addition to the quantitative analysis results, responses received during the interviews with students and teachers indicated that the four LEGO instruction units of the STEM engineering module played important roles in improving students’ problem prediction and analysis abilities. Clark and Ernst (2009) highlighted the advantages of virtual and physical modeling in their research. In this study, the instruction units enabled students to grapple with (1) abstract concepts of scientific principles and mathematics computation using a computer simulation and (2) mechanism design and working function with physical models. The computer simulation was an aid in helping to explain the movement of the mechanism using mathematical and physical principles (e.g., movement distance, power transfer, and speed change). By assembling the LEGO models, students became familiar with the functions of different LEGO parts and enabled them to acquire procedural skills and hands-on-activity experiences (e.g., how to select different gear parts to change speed, how to enlarge moving distance through the use of levers and linkage, and how to transmit power between motors and mechanisms). Therefore, the module can help students to improve their problem prediction and analysis performance, increase the feasibility of their design ideas, and optimize their project outcomes.

Higher-order thinking skills require a high level of flexibility in understanding how to integrate and apply conceptual knowledge and procedural skills within a complex problem context. The analysis of the results reported above indicated that higher-order thinking skills were related significantly to students’ application of conceptual knowledge, especially for the ability of problem prediction and analysis. According to the correlation analysis (Table 9), performance on problem prediction and analysis affected the working function of the design project. As Hayes (1989) pointed out, the process of “thinking before acting” is critical if designing is to be a well-planned and predictive process, rather than a trial-and-error process. In design activities typical of existing high school technology education, teachers allocate a lot of time and effort explaining basic knowledge and skills and providing numerous demonstrations for students. However, without a specific connection presented between the problem context and STEM knowledge, students often lose interest in learning basic conceptual knowledge. As a result, they rarely develop a feasible solution during their design project process due to the lack of appropriate conceptual knowledge and skills. Additionally, when their solution fails, students may have difficulties determining the problem with their design, because they do not know how to develop a scientific and mathematical analysis. Simply stated, they do not know “why”.

Based on the results of the design project, students in the experimental group also outperformed their counterparts in design project performance. Based on informal classroom observations made by the research team and an analysis of the results of the teacher’s interview, we found that the STEM engineering module could more effectively help most students understand the design problem of their project. In other words, the experimental group students showed better performance in predicting and analyzing problems that might occur in the design process before action was taken. For example, the students could understand the importance of frame stability during mechanism design, or they could analyze ways to avoid interference during motion. When a problem occurred, the students were more clearly aware of the source of the problem because of their observation and analysis abilities, thereby reducing the time and frequency of aimless trial and error, and they were able to demonstrate superior performances in their design project. In contrast, the control group students lacked these advanced capabilities and ways of working.

Many educators argue over the purpose, instructional approaches, and effectiveness of the integrative STEM curriculum in high school technology/engineering education (Bybee 2013). In many studies, a STEM curriculum at the high school level has been designed as a pre-engineering curriculum to help students learn science and mathematics concepts and increase students’ interest in engineering careers (Bayer Corporation 2010; Project Lead the Way 2014). In technology education, the integrative STEM approach is used to help students learn how to solve real-world problems via integrative knowledge of STEM subjects and to improve their STEM literacy (Havice 2009; Salinger and Zuga 2009; Sanders 2009). Without appropriate instruction to integrate STEM knowledge, most Taiwanese students who lack relevant prior conceptual knowledge and hands-on experiences would face many difficulties in engineering design courses and may fail to complete a design project during their design processes activities. We found that the integrative STEM engineering module did assist students to integrate their conceptual knowledge, higher-order thinking, and engineering design skills, but importantly, the module more effectively helped students who lacked prior STEM knowledge. As indicated by the interview with the participating teacher in the experimental group, the STEM engineering module can help students to reduce conceptualization time and, consequently, minimize the timeline from trial and error, commencing with a hazy idea, to a complete physical product of a mechanism such as a toy. From first ideas to design sketches, to the mechanism design process, conceptual knowledge of the mechanism and science and mathematics principles play important roles in foreseeing the overall design. The results also indicated that higher-order thinking skills play important roles in problem solving and trouble shooting. Overall, the key factor that underpins the engineering problem solving of students is their ability to define, predict, and critically analyse problems.

Conclusions and recommendations

Developing ways to help students acquire STEM knowledge and higher-order thinking abilities for solving complex problems is the core objective of engineering and technology education. However, the integrative STEM curriculum consists of vast and complex knowledge structures, which cause many difficulties in instructional design and implementation programs. In this study, we successfully improved many weaknesses that we found in our prior research and, again, supported the positive effects of the integrative STEM instructional approach in high school engineering/technology education. More importantly, this study demonstrated an effective approach for helping students, especially those who lack prior knowledge, to achieve success in increasing their STEM knowledge and understandings, and problem prediction and analysis capabilities. In summary, a successful engineering design project depends on the student’s STEM knowledge and how efficiently the student can effectively combine the relevant knowledge with higher-order thinking skills thus enabling them to predict and analyze problems during the engineering design process. We found that the virtual and physical modeling within the engineering design instruction module were also effective practice for teaching STEM knowledge and for enhancing students’ problem prediction and analysis capabilities.

The following two recommendations should help facilitate improved high school engineering instruction in the future. First, modeling is an effective approach for implementing an integrative STEM curriculum. A modeling-based integrative STEM curriculum can be presented simultaneously through the use of both virtual 2D/3D models, and physical hands-on models. For students who lack sufficient prior knowledge in engineering design, specific and intuitive learning experiences should be provided to establish essential conceptual and procedural knowledge before the students engage in an engineering design project. Teachers could use modeling to explain abstract scientific principles and mathematics predictive analysis, thus enabling students to transform their ideas into reality through physical modeling. Therefore, the development and research into modeling-based integrative STEM curricula is a focus worthy of in-depth investigation in future research studies.

Secondly, problem definition, prediction, and analysis are core components of engineering design in high school. These abilities require effective integration of STEM knowledge and higher-order thinking skills. Technology and engineering educators and teachers should place emphasis on helping students to become aware of these abilities during their design activities. Therefore, more specific instructional strategies and practices which focus on teaching problem definition, prediction, and analysis should emphasized. Another issue may be involved: students’ metacognition during their integrative STEM engineering design process experiences. Therefore, this issue is also an important research area worthy of in-depth investigation in future studies.