Introduction

Educators and policymakers agree that in order to meet the needs of the twenty-first century, and to prepare our students for the challenges ahead of them, engineering content must be part of K-12 classrooms. This agreement is evident in the growing availability of K-12 engineering curricula, increased research in this area (e.g., a new journal dedicated to Pre-Collegiate Engineering Education Research), and national policy documents arguing for the inclusion of engineering into K-12 science classrooms (National Academy of Engineers & National Research Council 2009; National Research Council 2012).

As summarized by the National Academies of Engineering and National Research Council (2009) joint report, engineering in K-12 classrooms is expected to both: (1) provide an opportunity to engage with science content in an applied context (which is expected to have positive motivational effect); and (2) introduce students to the field of engineering (previously underrepresented in K-12 education; pp. 49–50). Similarly, when articulating the new Framework for K-12 Science Education (National Research Council 2012), the National Research Council argued that engineering is an important aspect of science education both because it can support student learning and application of traditional science content and because engineering is an important domain in and of itself. As this is a relatively new field, there is limited research examining the efficacy of these claims (National Academy of Engineers & National Research Council 2009). Much of the research that exists underscores that, while engineering can be a powerful context in which to achieve these goals, leveraging that potential is challenging for students, teachers, and teacher educators (Barnett 2005; Fortus et al. 2004; Kanter 2010; Prevost et al. 2011; Stohlmann et al. 2012; Tran and Nathan 2010).

Design is increasingly viewed as a possible approach for addressing the challenges associated with meeting these two goals. As stated in the Framework for K-12 Science Education (National Research Council 2012), “…from a teaching and learning point of view, it is the iterative cycle of design that offers the greatest potential for applying science knowledge in the classroom and engaging in engineering practices” (pp. 201–202). This emphasis on engineering design is seen throughout policy documents (National Academy of Engineers & National Research Council 2009; National Research Council 2012) and research (e.g., Daly et al. 2012; Dym 1999; Sheppard et al. 1997). As Dym (1999) argued: “beyond being the ‘capstone’ of engineering education, design should be the very cornerstone of engineering learning — or, in an anthropomorphic metaphor, design activities and courses should be the backbone of the engineering curriculum” (p. 2). In addition, engineering design is one of a few core ideas found in the new framework guiding the development of K-12 science standards (National Research Council 2012), demonstrating the emphasis that the National Research Council believes it deserves in educational settings. Indeed, design is the central process of professional engineering, and providing design based learning opportunities therefore aligns classroom activities to the experience of career engineers.

In the current study, we explore the ways in which participating in design supports or inhibits the two goals associated with the integration of math, science, and engineering: applying math and science knowledge, and engaging in engineering practices. We examine this by first defining the engineering design process (EDP) and discussing the opportunities for integrating math and science content into that process.

What Is the Engineering Design Process?

There is no single EDP. The process differs by discipline and project in the work of professional engineers, and this variation is also seen in existing curricular approaches to supporting engineering design. Even with this variation, the educational community has identified a number of core characteristics of the EDP, including (1) the design process begins with problem definition; (2) design problems have many possible solutions and engineers must find systematic approaches to choosing between these; (3) design requires modeling and analysis; and (4) the design process is iterative.

Existing K-12 engineering curricula generally concretize these characteristics by articulating steps of an EDP. A close analysis of the various EDP models found in existing curricula reveals that the characteristics identified above are embedded throughout these models [as reviewed in Guerra et al. (2012)]. That is, the vast majority of EDPs used in curricular contexts include steps such as identifying the problem and generating solutions, as well as opportunities to iterate on design ideas. The inclusion of these characteristics across the EDP models found in the reviewed curricula suggests that these characteristics are consistent with the EDPs currently being used in K-12 classrooms. In the following paragraphs, we more carefully define each of these characteristics of engineering design and discuss the opportunities they create for integrating math and science content into design work.

Defining the problem requires moving from a broad statement of need (i.e., providing access to food throughout the world) to a specific problem that an engineer might solve (i.e., developing roads in rural areas, or designing food crops that can grow in arid climates). As stated by the Framework for K-12 science education (National Research Council 2012), “Engineers ask questions to define the engineering problem, determine criteria for a successful solution, and identify constraints” (p. 50). Doing this can involve interviewing users and stake holders; defining success criteria; researching existing solutions; identifying sub-problems and goals; and learning the underlying math and science needed to analyze data and work toward a solution. While the time spent on this task varies according to complexity of the problem and engineer expertise with the domain (Pahl et al. 1999; Atman et al. 2005), there is general agreement that successful completion of engineering design work requires careful attention to gathering the necessary information to understand the problem. For example, Yang (2005) found that time spent on design broadly was not a key factor in the individual’s success but that time spent on problem definition—or scoping—seemed to support more successful designs. However, students often minimize their work on researching and understanding the problem context [see review in Crismond and Adams (2012)]. In addition to being vital for engineering, the work of identifying the success criteria which often requires quantification of the qualitative needs and goals, and understanding the math and science concepts that are central to the challenge are both clear opportunities for students to engage with math and science content while tackling their engineering work.

The second characteristic of engineering design identified by the national syntheses of the literature documents (National Academy of Engineers & National Research Council 2009; National Research Council 2012) is that problems will have multiple possible solutions between which the designers must select. This characteristic embeds two key features of design: first, engineers must generate multiple solutions; and second, they must develop systems for choosing between those solutions. It is generally considered good design practice to avoid being overly focused on a single idea too early in the design process for fear that a better solution will be overlooked. Early focus on one idea is referred to by various names such as “design shutdown” (Newstetter and McCracken 2001) and “design fixation” (Purcell and Gero 1996). Although they did not use an engineering context, Dow et al. (2010) found that designers who presented multiple solutions to be critiqued in a single session (parallel prototyping) created a more diverse set of ideas than designers who had multiple critique sessions but only presented one design at each (serial prototyping): While participants engaging in serial prototyping reported that they tended to refine the same idea between critiques, participants engaging in parallel prototyping tried to create ideas that were independent of each other. In addition, the parallel designers scored higher in the quality of their designs.

Indeed, the emphasis placed on generating multiple solutions in the policy documents (National Academy of Engineers & National Research Council 2009; National Research Council 2012) reflects the practice of experienced designers, who typically identify multiple ways to address the engineering problems (Ball et al. 1997). However, given a list of design activities, engineering students place “generating alternatives” among the least relevant activities (Newstetter and McCracken 2001). Moreover, novices tend to generate one design and focus on it until that design is proven to fulfill the project expectations or proves too difficult to continue (Anderson et al. 1984). Thus, while the literature and expert practice typically emphasizes the identification of multiple solutions, novices struggle with finding this aspect of the EDP relevant.

Once multiple possible solutions have been identified, engineers must work to select one. Because projects and problems often have multiple sub-goals, which may conflict, this requires assessing priorities and making trade-offs (Jonassen et al. 2006). Thus, assessing a proposed solution may not be a binary measurement, but rather a complex judgment of the quality that accounts for the number of goals fulfilled and the degree to which each is met. The selection of which possible solution to embody requires balancing these different goals—or optimization. This is typically done using systematic strategies that enable engineers to choose between the competing project goals and user needs.

This characteristic of generating and selecting from multiple possible solutions creates some opportunities for students to integrate their math and science content in two ways. First, knowledge of the underlying science can help individuals determine feasibility and efficacy of design ideas. Second, the systematic strategies for balancing potentially competing design goals are an additional opportunity for students to mathematize the qualitative measures of success.

The third characteristic of engineering design is that it is achieved through modeling and analysis. This characteristic is best understood in the context of the other characteristics—modeling and analysis are strategies for choosing between possible solutions and can be key tools for quantifying the qualitative needs during problem definition. For example, key opportunities to model occur early when defining the problem—in this case, engineering students might model parts of the existing system into which the solution must fit. Additional opportunities for modeling occur later in the process as students collect data regarding efficacy of various solutions or aspects of the system and develop mathematical models to analyze that data. Along with analytical thinking and conceptual problem solving, surveyed engineers identified mathematical modeling as a critical part of engineering work (Perlow and Bailyn 1997). However, research has shown that students struggle with engaging in model-based reasoning around their design work [see review in Crismond and Adams (2012)]. In addition to being a fundamental aspect of engineering design, modeling and analysis are key opportunities to integrate math and science content—these are key practices in both domains (Governors Association 2010; National Research Council 2012). As argued by Julie (2002), “It [modeling] also entails the scrutiny, dissection, critique, extension, and adaptation of existing models with the view to come to grips with the underlying mechanism of mathematical model construction.” Thus, we see modeling as both a “vehicle” for learning and applying math and science content and as “content” in its own right (Galbraith 2012).

The final characteristic of engineering design is that this is an iterative process. Iteration is a critical component of design; “Design work cannot be completed as a once-through procedure, because of the complex information dependencies that typically exist” (Smith and Tjandra 1998, p. 108). Iteration refers to the process of revisiting previously completed steps in order to improve design. It can occur on at least two scales: physical repetition of steps or processes or mental thought experiments used to generate and evaluate possible solutions (Jin and Chusilp 2006). For the classic model of an engineer designing for an external client, iteration may be forced on the engineer by changes in client requirements or supply issues. But even in a stable situation, several studies have investigated the necessity of iteration. Dorst and Cross (2001) and Maher and Tang (2003) found evidence that creative solutions can be created as the germ of an optimization or idea leads to a loop where the designer reframes or redefines the problem, which leads to further refinement of the idea. They refer to the condition as a co-evolution between the problem-space and solution-space where the idea forms a bridge between the two. After reviewing a number of models of iteration, Smith and Tjandra (1998) found that the quality of the designs produced by their subjects was less dependent on the iteration process that they used than if the groups revisited their concept generation stage in their iteration.

Despite the view that iteration is a key component of design, not all K-12 engineering programs include explicit points of iteration (or loops) within the EDP model. Of the 11 K-12 EDPs reviewed by Guerra et al. (2012), only eight of them include some form of loop or other structure that enables iteration while the remaining three EDPs describe the process as a list of steps to be followed linearly. Moreover, we note that while iteration is central to successful engineering design, it does not inherently create opportunities for exploration or application of math and science content. That said, as students iterate through steps of the EDP that do integrate math and science they are creating additional opportunities for that work.

Table 1 summarizes these four characteristics of engineering design, articulating the key aspects of each characteristic and highlighting the quantitative aspects—or those aspects that create the most opportunity for math and science integration. As highlighted in Table 1, there is much to learn about engineering design in and of itself. In addition, we see that engagement in this process creates multiple opportunities for students to explore and apply more traditional math and science content. This paper explores these relationships. In particular, K-12 educational institutions are shifting to incorporate engineering into science education, increasingly through an emphasis on engineering design. This shift is a strategy to enhance instruction of traditional math and science content as well as to teach engineering practices. Thus, this paper examines the ways in which engaging in engineering design can support students in participating in engineering practices and applying math and science knowledge. To accomplish this, this paper will examine student understanding of the four characteristics of engineering design and will analyze whether and how students perceive those characteristics as creating opportunities to learn and apply math and science content.

Table 1 Summary of the four characteristics of engineering design
Full size table

Methods

We explore the student perceptions of engineering design and the opportunities to learn and apply math and science content therein within the context of a high school engineering course: UTeachEngineering’s Engineer Your World. For this study, we examined 179 student questionnaires about the EDP to generate hypotheses and additional questions about their understandings of the EDP. We then used interviews conducted with 16 focus students to further explore these issues. We describe the study context, data collection instruments, and analytical methods in more depth in the following sections.

Study Context

Engineer Your World is designed to follow learning sciences and engineering education best practices for supporting student learning through disciplinary practices [guiding design principles described in Berland (2013)]. It is a year-long course that involves six separate units. Following trends in engineering education, each unit is organized around a key design challenge (Klein and Harris 2007; National Academy of Engineers & National Research Council 2009; Sadler et al. 2000; Tate et al. 2010). These design challenges were selected to introduce students to a range of engineering disciplines and include the following:

  • Unit 1: Reverse engineering a hand-cranked flashlight or a hair dryer.

  • Unit 2: Designing and building a pinhole camera.

  • Unit 3: Designing and building a system to take aerial images.

  • Unit 4: Analyzing data and redesigning a prototype wind turbine.

  • Unit 5: Designing and programming robotic vehicles.

  • Unit 6: Analyzing data and redesigning construction helmets (only completed in one participating class).

Throughout the course, students utilize a common EDP, represented in Fig. 1. This EDP was developed in consultation with engineering faculty (Lattuca et al. 2006) and the National Research Council and National Academy of Engineer’s research joint report (2009), after benchmarking eleven of the most popular K-12 engineering curricula [full description of this process and the resulting EDP can be found (Guerra et al. 2012)]. As such, it reflects the four key characteristics of engineering design including an emphasis on defining the problem (seen in the combination of the identify and describe steps), generating and selecting from multiple possible concepts (seen in the generate super-step), using mathematical modeling (through the characterize and analyze the system, and test and refine sub-steps), and enabling iteration (represented in the multiple loops in the EDP).

Fig. 1
figure 1

EDP model used in Engineer Your World

Full size image

Study Data Sources

Data Sources for this Study Include

  1. 1.

    Questionnaires All students participating in the course were given a questionnaire asking them to: describe how they would apply the EDP to a specific design problem; describe the EDP in general; and identify the most and least important steps of the EDP. Questionnaires from the 179 students that were consented and present at the time of the administration were collected.

  2. 2.

    Interviews Sixteen students from two classes were interviewed about their questionnaire responses. Interviewed students were selected based on their willingness and availability to participate in interviews.

We provide detailed descriptions of each instrument, including the respective participant pools and analytical methods, in the sections that follow.

Questionnaire

Questionnaire Participants

The questionnaire was distributed to students enrolled in Engineer Your World at seven high schools in central Texas, during the 2011–2012 school year. These schools included both urban and suburban communities. Over 250 students were consented to participate in research activities associated with the design and study of the project curriculum. 179 of these students participated in this study (the reduction in participants is due to availability when the questionnaire was administered). While some students were high school sophomores and juniors, the majority were seniors who had completed, or were concurrently enrolled in, Physics.

Questionnaire Design

During the final week of class, students completed a questionnaire (see “Appendix 1”) designed to assess their understanding of the EDP. This questionnaire asked students to draw a model of the EDP from memory, apply the EDP to a situated design problem, and identify which steps they found most and least important.

Questionnaire Analysis

In order to analyze data from the questionnaire, each question was considered separately. Blank and illegible responses were removed from consideration. For the question that required students to list steps in the EDP, attrition was particularly high (n = 117 usable responses out of 179 total participants).

Preliminary analyses of the question in which students described how they would apply the EDP to a specific design problem revealed significant problems with the question. In particular, we designed the question to reference a prior project the students had completed in the course. We did this to ensure that students would have the content knowledge necessary to respond. However, this connection resulted in student responses that were either too specific (i.e., describing specific tasks they performed, such as cutting the cardboard) or that described the redesign process rather than the complete EDP, in which we were most interested. For these reasons, this question is excluded from the analysis.

Two researchers analyzed student responses to the three remaining questions. The analysis included a theory-driven approach in which we worked to match the students’ representations of the EDP to the EDP in the curriculum. Through this matching, we noted which of the nine EDP steps found in the curricular materials appeared in the student representations of the EDP. In addition, students’ representation was coded for whether they depicted iteration. This approach reflects our need to standardize the students’ representations in order to compare across them. We used the curriculum’s EDP as a model because it provides insight into the four characteristics of engineering design on which this study focuses. In addition, we recorded the steps that students identified as being most and least important, respectively.

To establish analysis procedures, two researchers worked closely, iterating on the analysis of 40 % of the questionnaires. At the conclusion of this process, an analytical scheme had been solidified. The two researchers each analyzed two-thirds of the remaining questionnaires independently using these agreed upon coding guidelines. The overlapping third (approximately 20 % of the total questionnaires) was used to establish an inter-rater reliability score. For each questionnaire in this group, the two researchers compared the EDP steps that they observed in the student’s drawing to the nine steps in the EDP used in the course curriculum. Each of the nine steps provided an opportunity for agreement between the raters. An inter-rater agreement was based on whether both raters noted the same step in his or her drawing of the EDP. A final reliability number was calculated from the percentage of times that the raters agreed compared to the total opportunities for agreement (i.e., nine possible steps on 117 analyzed worksheets). The final calculation revealed an inter-rater reliability of 91 %. This exceeds the conventionally accepted minimum level of 85 %.

Semi-Structured Interviews

Interview Participants

In addition to the written questionnaires, eight students were interviewed from each of two participating classes to obtain a fuller description of the students’ views about the steps of the EDP. The 16 interviewed students were chosen based on availability and willingness. The teacher deemed students available depending on their progress on class work such that the time dedicated to the interview would not be critical to their success in class.

As our purpose was to use the interview data to better understand the questionnaire responses, we compared the interview participants to the questionnaire respondents. Table 2 compares the demographics of the interviewees to the demographics of all student participants. The demographic survey and questionnaire were administered on different days so the sample sizes are different. Nonetheless, the demographic data depict the general make up the classes with which we were working and can thereby be compared to those of the interviewed students to ensure that the interviewees were reflected the overall study population.

Table 2 Comparing demographic characteristics of interviewed students to full sample
Full size table

In addition, we performed a chi-squared test to determine whether the EDPs depicted by the interviewed participants on their questionnaires were significantly different from the EDPs depicted by the non-interviewed participants. Details of this analysis can be found in “Appendix 2.” In short, this analysis did not show significant differences in EDPs between the samples [χ2(6, n = 117) = 11.75, and p = 0.16]. The worksheets of the interviewed students appear to be broadly similar to the worksheets non-interviewed students.

Interview Design

The interviews were semi-structured, such that interviewers engaged students in a conversation focused on students’ explanations of their questionnaire responses. We consider the interviews a “negotiated text” (Fontana and Frey 2000, p. 663) that was co-constructed through the conversation between the interviewer and interviewee. The conversation was designed to elicit students’ thoughts on the purpose of each step of the EDP, the importance of each step, and whether a professional engineer would use each step. During the conversation, students had access to the questionnaires that they completed. The interview focused on the five super-steps found in the EDP for the curriculum used in this study, including

  1. 1.

    identifying need,

  2. 2.

    describing the need (includes sub-steps: describe and characterize/analyze the need),

  3. 3.

    generating concepts (includes sub-steps: generating and selecting concepts),

  4. 4.

    embodying the design (includes sub-steps: embody, testing and evaluate; refine), and

  5. 5.

    finalizing the design (i.e., preparing to share the design with outsiders).

Graduate student researchers conducted and video recorded the interviews. Depending on student responses, each interview lasted between 10 and 30 min.

Interview Analysis

We used a three-step process to analyze the interviews. First, we assigned “descriptive codes” (Miles and Huberman 1994, p. 57) or engaged in “categorization” (Kvale and Brinkmann 2008, p. 203). In this process, we segmented the interview according to the EDP super-steps that were addressed. In doing this, an interview segment was defined as the entire interviewer–interviewee exchange in which the topic was discussed, in addition to any contextualizing discourse that supported the interpretation of that exchange (a segment typically ranged from 2 to 10 turns-of-talk). This process was fluid—a single utterance could speak to multiple EDP steps and each step could be addressed multiple times throughout the interview.

Second, we did an additional categorization step. In this, we looked at the kind of information being communicated about each step. In particular, we differentiated between instances in which students

  1. 1.

    defined the step,

  2. 2.

    discussed the value of the step for themselves as students in an engineering class, and

  3. 3.

    discussed the value of the step for professional engineers.

We used this categorization to “quantify how often specific themes [in this case, the definition and value of each of the EDP steps] are addressed in a text [interview]” (Kvale and Brinkmann 2008, p. 203). Through this process, we were able to make claims about how many students discussed various aspects of the EDP. Given that the interviews were a “negotiated text,” not all interviewees were explicitly asked each of these three questions about each of the five steps. Instead, researchers were interpreting student responses to a variety of related questions in order to address the three above-mentioned central questions.

Third, we engaged in “meaning interpretation” (Kvale and Brinkmann 2008, p. 207) in which we synthesized across all instances of a single category to develop a description of that student’s understanding of that aspect of the EDP. For example, we described each student’s definition of the “identify a need” step of the EDP as well as the student’s perception of the value of that step for themselves and for engineers. As a result of this meaning interpretation, we were able to make claims about the types of things students said about each of the five steps of the EDP. For example, we examined the clarity of their definitions and the valence of their value statements. This process of interpretation is, by its nature, subjective. Two researchers engaged in this process independently and interpretations were refined through iterative discussion between these researchers, curriculum designers, and the rest of the team.

Triangulation Between Questionnaires and Interviews

Since both the written questionnaires and the interviews examined students’ recollection and interpretation of the EDP, both were considered when forming and evaluating hypotheses regarding the students’ understandings. As such, we used each data source to check our interpretations of the other. Hypotheses about observed patterns were typically generated starting with patterns in the questionnaire data. Then, the interview analysis was consulted for corroborating or contraindicating evidence predicted by a given hypothesis. When supporting evidence was found, we were more confident in our interpretations. If contraindicating information was found in the interviews, the hypothesis was revised to account for all of the data, or discarded. The results presented in this paper are based on patterns that are consistent with both questionnaire and interview data.

Findings

In this study, we explore the ways in which the students perceive the EDP and how their understandings create or eliminate opportunities to apply math and science knowledge and engineering practices. We organize our findings around the four key characteristics of engineering design, as specified by the national syntheses of the research (National Academy of Engineers & National Research Council 2009; National Research Council 2012): (1) the design process begins with problem definition; (2) design problems have many possible solutions and designers must find systematic approaches to choosing between these; (3) design requires modeling and analysis; and (4) the design process is iterative.

Engineers Define the Problem

The National Academy of Engineers and National Research Council have identified problem definition as a key characteristic of engineering design (2009, 2012). Problem definition consists of identifying qualitative and quantitative needs associated with the design problem, articulating success criteria for the design, and learning the math and science that will serve as the foundation of any design solutions (as necessary). All of these aspects of defining the problem require that students use and explore underlying math and science principles. This characteristic of engineering design most closely aligns to the curriculum EDP steps called “identify the need” and “describe the need.” “Describe the need” includes the sub-steps of “describe the need” and “characterize and analyze the system.” Our exploration of whether and how students interpret this characteristic of engineering design is therefore focused on students’ discussion, both explicit and implicit, of these particular steps throughout their questionnaires and interviews. Our analysis of these data suggests that students are more familiar and comfortable with the qualitative aspects of this work than the quantitative ones.

Define the Problem: Questionnaire Analysis

Looking across the student questionnaires suggests that students are generally familiar with the “identify” and “describe” steps of the EDP. 51 and 40 % of the students included these respective steps in their depictions of the EDP. In addition, only 5 % of the students named “identify” as one of the least important steps and 11 % named it as the most important. 12 % mentioned “describe” as least important and 11 % named it as most important. These findings suggest that students understand the importance of defining the problem—they frequently included at least one EDP step in which the problem definition would occur and clearly valued the identification of the problem.

Given that the majority of the quantitative aspects of defining the problem occur in the “characterize and analyze” sub-step, we also looked specifically for inclusion of it in the students’ questionnaires. We found that, of 117 questionnaires analyzed, only 16 students (14 %) explicitly included the “characterize and analyze” sub-step in their questionnaire. It is difficult to make strong claims from this finding given that the super-step and sub-step are both called “describe the need”—that is, it might be that students did not include “characterize and analyze” explicitly because it was implied by the “describe” super-step. Alternatively, the lack of attention to the “characterize and analyze” sub-step might indicate that students did not remember or value this aspect of the EDP. The richness of the interview data enabled us to explore these different interpretations of the questionnaire data.

Define the Problem: Interview Analysis

All sixteen of the interviewed students discussed the importance of defining the problem. For example, Tobias states: “You have to explain the need, which, I mean, it’s important. You have to know why you’re doing it.”

Ten of the 16 students who discuss the problem definition do so in terms of identifying and understanding customer needs. For example, Martin says that “you analyze what they [the customer or client] said to apply to whatever design you’re making.” Mitchell similarly states that the “customer needs to be happy.” Adam’s discussion of defining the problem step stood out from the others. Adam was the only student to discuss the importance of understanding and building off of existing solutions, as he said: “You should research about it because, uh, it’s really hard to reinvent the wheel.” The remaining interviewees did not move beyond the initial vague discussion of “explaining” or “describing” the need. Thus, looking across the students’ discussions of defining the problem, we see that attention to user needs was the most emphasized aspect of this work. Other aspects—including researching existing solutions and identifying sub-problems and goals—were rarely discussed.

Given that analysis of questionnaire responses suggests that students pay little attention to the “characterize and analyze” sub-step, we categorized and interpreted all instances in which this sub-step was discussed in interviews. Through this analysis, we found that 13 students mention this step, providing short and vague descriptions of what happens during it and admitting their lack of certainty with qualifying their statements. For example, Zoran stated that he did not remember the step, while David stated:

…characterizing it and analyzing it, um, you know, it’s just kind of…Describing it is kind of touching the surface. And the, characterize and analyze it, I know it’s vague but you have to, you know, go beneath the surface and dive a little deeper.

Only one interviewed student, Dana, seemed to understand the essence of “characterize and analyze.” In his interview, he presented a pithy definition of this sub-step: “describing the need would be trying to figure out how to change that from like a qualitative term to a quantitative term.” Dana’s understanding aligns with the science and math tasks described in the characterize step. However, his understanding is unique among students interviewed.

Two more students, Chase and Tobias, were able to provide examples of the work done during the characterize step, such as “black box diagrams” and “activity diagrams.” For example, Tobias stated “the best way that we described the need is we used the black box diagrams.” Identifying these procedures indicated to us that students are aware of particular tasks associated with this step of the EDP. However, students’ failure to explain the questions that these strategies answered, or how they related to the overall EDP suggests a lack of understanding about why these quantitative analyses are so important to the work of engineers. For instance, a black box diagram may be applicable in the development of some engineering designs but not in others. Ideally, students would understand not only when to perform the technique but also why and how. Dana, Chase, and Tobias were the only interviewees that defined the “characterize and analyze” the system sub-step at all.

Along with these vague descriptions of the “characterize and analyze” sub-step, interviews suggest that the students perceive the step as less valuable than other steps in the EDP: of the 16 students interviewed, only five students reported that they valued the step for students in an engineering course. Furthermore, only three students suggested that the step was valuable to professional engineers. The remaining 12 or 13 interviewed students did not discuss whether this step was valuable to themselves or to professional engineers. Taken together, these data suggest that the majority of students did not understand or value the “characterize and analyze” sub-step.

Define the Problem: Integrative Analysis

Table 3 summarizes the patterns revealed through the interview and questionnaire analysis and compares it to the qualitative and quantitative aspects emphasized in the literature. Through this comparison, two main conclusions are evident. First, students did not generally think about the project need in a quantitative manner. Second, students’ qualitative analyses lacked depth.

Table 3 Student understanding of defining the problem in engineering design
Full size table

Engineers Develop and Choose Between Many Possible Solutions

One of the “distinguishing features of engineering design include…the embrace of multiple possible solutions” (National Academy of Engineers & National Research Council 2009, p. 41). Thus, the second characteristic identified as central to engineering design is that engineering problems have multiple possible solutions and engineers must identify and choose between these possibilities. These multiple ideas enable engineers—and engineering students—to search for the best solution rather than enacting the first idea that comes to mind. Engineer Your World’s EDP captures the idea of generating and selecting between multiple ideas in the “generate” step, which includes the sub-steps “generate concepts” and “select concept.”

Developing and Choosing Between Many Possible Solutions: Questionnaire Analysis

The generate and select step includes two distinct sub-steps (see Fig. 1). First, generation of multiple concepts. Most students recall the generating multiple possible solutions as a part of the EDP. Specifically, 81 % of students included an idea-generation step in their representations of the EDP on questionnaires. Furthermore, students seemed to value this step: Approximately one-third of students (58 out of the 154) reported that this step was the most important of all steps in the EDP. Some students understood that by generating concepts with others, engineers can benefit from differences in perspective, which can lead to a better concept. As Aaron wrote: “Generating Ideas [is the most important] because not one person see[s] the object the same and ideas can be enter mixed with each other.” Other students saw generation as the step in which creativity and insight can distinguish good designs from others. For example, Jesus argued that “Brainstorming is the most important part of engineering, this is where brilliance and innovation of designs come from.” While most students seem to value this step and provide a variety of reasons for its value, 15 (11 %) of the students considered “generating and/or selecting concepts” to be the least useful step for professional engineers. These students failed to recognize the value of the step for one consistent reason: They believed that one viable concept is sufficient. For example, Rory reported that “brainstorming multiple ideas [is least important]. I brainstormed multiple pinhole camera ideas and used my first one.”

The second sub-step of the “generate” step in EDP is selection of the best solution. While the need to select is implied in the act of generating multiple ideas, engineers go through specific processes to ensure that the idea selection is driven by rigorous attention to the needs of the project. Students did not mention these processes in their questionnaires. In fact, only 15 % of students included selection of possible solutions as a step in their representation of the EDP. We turn to the interview analysis to examine whether students have a more detailed understanding of the value of and processes for carefully generating and selecting multiple possible solutions to their engineering design work.

Developing and Choosing Between Many Possible Solutions: Interview Analysis

The interview data similarly reveal that students understand the importance of generating multiple solutions to a single problem. In fact, of the 16 interviewed students, 10 explicitly discuss the importance of having multiple ideas and only 2 suggest that one idea is sufficient (The remaining four students never address the importance of generating multiple solutions.). For example, in his interview, Dana said, “you have to come up with multiple um, which we did a ton of.” In his next response, Dana explains why it is important to generate “a ton of” concepts:

sometimes the first concept isn’t always the best, so you just try to come up with multiple ideas…and then when you find one that looks the best, that’s when you select it. But you have to come up with a lot of them, not just one.

Similarly, Jeff describes how his group would generate and select concepts: “each of the class members would… give a–an example of what a good solution might be. Then we would narrow it down and find out what all of us think the best solution.”

While interviewed students generally seemed to value the generation of multiple possible solutions, only three of the 16 interviewed students discussed the idea that evaluating design solutions might require a systematic approach to comparing success on competing criteria—for example, balancing potentially competing needs of weight and cost—before designs were built. Zoran stated that he selected among multiple ideas by considering the “practicality of the product.” In addition, Aiden discussed a more systematic approach to selecting between competing designs, stating:

Well, you really need to break down what you need to accomplish and then you brainstorm ideas to address those problems. What I like to do is, I have a list of pros and cons for each, uh, idea. And then I would choose the one that would be most beneficial with, uh, while being the least detrimental in its own way.

In these quotes, we see Zoran and Aiden discussing criteria and possible processes for selecting between competing design solutions before any solutions are built. The third student stated that he (David) would present all possible ideas to the customer and “we’re going to, you know, narrow the concepts down and finally come up with one.” Comparing these responses suggests that these three students were aware that concept selection might need to occur before anything could be built, but only Aiden described a process consistent with the systematic approach idealized in the literature on engineering design.

Two other students discussed the selection process as occurring after competing design ideas are built and tested. For example, Tobias stated “…after we’d come up with the multiple designs, we would pick a select few that we would test and generate and try to see which one worked out the best.” The remaining 11 students either did not mention the need to select a design or described it as a vague process. For example, Mitchell said that his groups would combine across all of their ideas or “one idea stood out as just better.”

Developing and Choosing Between Many Possible Solutions: Integrative Analysis

Overall, this analysis suggests that students value the generation of multiple possible solutions to a single problem but rarely engaged in systematic procedures for choosing between these possibilities. The lack of systematic selection limits students’ opportunities to apply math procedures to this aspect of engineering as this process typically requires mathematizing goals and constraints. Table 4 relates this finding that students valued the generation of multiple ideas but did not discuss systematic approaches to selecting to these possibilities to the opportunities for using math and science while engaged in this aspect of design.

Table 4 Student understanding of developing multiple ideas in engineering design
Full size table

Engineering Requires Modeling and Analysis

The third characteristic of engineering design is that individuals or teams must use models to analyze data and potential solutions. “By creating the representational models of potential solutions and then mathematically characterizing them, engineers can predict the behavior of technologies before they are built, and the predictions can be tested experimentally” (National Academy of Engineers & National Research Council 2009, p. 42). This evaluation process is the focus of the 4th unit in Engineer Your World. In this case, students developed a procedure to test multiple variations of wind turbine prototypes and then used a mathematical model in order to identify the most efficient wind turbine. In addition, models and prototypes were also used to other ends throughout the course. For example, when designing the pinhole cameras students used a curriculum provided mathematical model to explore the relationship between the size of their pinhole, the film and the object they were photographing, and the distance they would stand from that object when taking the picture.

Given the variation in whether, when, and how models were used in this curriculum, the questionnaire analysis could not help us explore student understanding of this characteristic of engineering design. As such, we explore student understanding of modeling by looking across the EDP steps and identifying instances in which they discussed these processes. During our categorization of the interview segments, we found only one instance of a student discussing modeling in engineering in a way that was consistent with the modeling and analysis characteristic of engineering design. This exception to the norm of not discussing modeling is a statement by Joseph, who says:

Yeah. I guess like, um, it’s kind of like in math, towards like you come up with the whole equation, try to solve the whole equation, and your finalized answer isn’t the one that you’re getting at. It’s like you go back into your first step and make sure everything is correct and trying different kinds of parts of the equation to end up with the final answer.

Given the paucity of relevant interview segments identified by the categorization process, we did an additional word search to determine whether we were missing any discussions of mathematical modeling. For this, we looked for every instance in which students used words related to modeling and math in their interviews. In generating the list of possible words, we looked for general words about math as well as the specific applications from their engineering work. The words we searched for included: calculate, equation, formula, model, predict, redesign, excel, math, proportion, scale, triangle, geometry, angle, variable, radius, graph, chart, degrees, speed, velocity, and force. In 16 student interviews, these words, taken together, were found approximately 45 times. We then examined each instance students used one of these words to determine whether the students were talking about modeling in ways that were consistent with the modeling and analysis characteristic of engineering design.

This process uncovered only one other student discussing work that is consistent with idea that engineers mathematically characterize the systems on which they are working. In this case, Tobias specifically described his group’s efforts to design and build a pinhole camera, stating:

…you have to calculate the sort of distance that they [the customer] want. And once you calculate the distance, and you calculate the size of the pinhole that you need to make to where you embody the whole picture, that sort of takes into account what the customer wants and what the customer needs.

The calculation Tobias is describing was necessary if students were going to design and build cameras that would take the target picture given specific customer needs (i.e., distance from the photographed object, size of film, and amount of light). Thus, in this quote, Tobias connects the work of analyzing the system to his ability to fulfill the project requirements.

The statements of Tobias and Joseph shown here are the only instances in which students used the aforementioned terms in a way that was consistent with the engineering design work of modeling and analyzing. Other instances of these words suggest that students were using some relevant concepts but not substantively engaged with modeling and analysis. For example, Tim said: “we created a small scale, which was the model. And then we go out and create it, and if it still didn’t work, then we have an expensive redesign to do.” In this case, Tim is using some engineering vocabulary, but his usage of that vocabulary does not reflect an understanding of the concept of mathematical modeling—instead he might be equating “model” with “prototype.” Furthermore, Art explicitly addresses the question of mathematical modeling. He states explicitly that they did not use this technique: “Oh, yeah. Oh, yeah. I think, I think we…. I think we did a lot of research on bridge designs, but I think as far as incorporating the math into it, we really didn’t.”

Thus, the preponderance of the evidence indicates that the students did not refer to mathematical modeling. As such, we conclude that students generally did not view mathematical modeling as an important aspect of their engineering work. This finding is summarized in Table 5.

Table 5 Student understanding of modeling and analysis in engineering design
Full size table

The Design Process Is Iterative

Iteration is a key aspect of engineering design and is therefore the final characteristic that we examine in this study. Iteration refers to everything from thought experiments about how ideas might play out to repetition of particular processes (Jin and Chusilp 2006). Regardless of the scale of the iteration, the goal is to reconsider previously made decisions in light of new information or data discovered through the design process. We examined students’ understandings of iteration in engineering design by examining both their questionnaires and interviews.

Iterative Nature of Design: Questionnaire Analysis

Of the 117 questionnaire representations of an EDP, 68 % of them included a representation of iteration in the model. Most commonly, iteration was represented through inclusion of loops leading back to a previous step (see images A and B in Fig. 2). In other cases, iteration was represented by the articulation of steps that explicitly mentioned revision of product, such as “test again” (see image C in Fig. 2).

Fig. 2
figure 2

Exemplar student depictions of the EDP highlighting an iterative refinement process

Full size image

The iteration students depicted typically occurred during the “embody” step, which students also referred to as the “build” step, without returning to the generate concept step (this can be seen in the EDPs represented in parts A and C of Fig. 2). In these situations, it appears that students are focused on iterating to refine or fix a design idea—to make it work—without changing the original design significantly.

Iterative Nature of Design: Interview Analysis

The interview results similarly suggested that students generally understood the importance of iteration used to refine project designs: Of the 16 students interviewed, 14 discussed iteration as refinement process. For example, Tobias stated “If that design didn’t work, then we would make sort of refinements.” Another student, Art, explained how he used refinement in a project: “we had to — we had to totally rethink [the cover for the pinhole camera] because just our design didn’t work physically on it.” In this case, Art and his team redesigned a small component of their camera without considering whether the overall design was appropriate.

Iterative Nature of Design: Integrative Analysis

Thus, looking across the interview and questionnaire data, we see students valuing iteration but only to the extent that it allows them to refine existing project ideas rather than revisiting the concept generation or selection. Table 6 relates the finding that students emphasized refinement of their existing designs without revisiting concept generation to the opportunities for using math and science while engaged in this aspect of design.

Table 6 Student understanding of iteration in engineering design
Full size table

While Ullman et al. (1988) report that professional engineers similarly “patch” their developing solutions rather than identifying possible alternatives, we would argue that this view of why engineers might iterate on their designs might limit students from finding stronger solutions by revisiting their overall design (particularly given their lack of systematic process for evaluating and choosing between possible solutions). Moreover, this apparent similarity across novice and expert designers might obscure underlying differences in how and why iteration occurs. In particular, expert designers engage in purposeful iteration (Ball and Ormerod 1995), whereas novice engineers, like the students in this study, are more likely to engage in a more haphazard, trial-and-error style of iteration.

Discussion

Table 7 summarizes the students’ understandings of each of the hoped for aspects of the engineering design characteristics identified in the literature. This table differentiates between those aspects that create the richest opportunities for students to engage with traditional math and science content through their engineering work—the quantitative aspects—and those that do not—the qualitative aspects.

Table 7 Results of this study
Full size table

Looking across this analysis of student understandings of engineering design shows that they have developed sophisticated understandings of the qualitative aspects of this work: With the exception of some of the aspects of defining the problem, student responses suggest that they understand and value the aspects of each characteristic of design that are not associated with math and science knowledge. For example, they seem to understand the value of identifying user needs, coming up with multiple possible solutions for addressing that problem, and iteratively testing and refining a solution. The student understanding and valuing of the qualitative aspects of engineering design reveal that students are learning many of the complexities associated with engineering design—that they are partially meeting the goal that students will gain familiarity with the engineering field.

However, the students’ understandings appear to undervalue the aspects of engineering design that are more quantitative. In particular, these students seemed not to value the work of: quantifying the need; developing systematic approaches to choose between possible solutions; or developing mathematical models. These findings are largely consistent with those challenges identified in the existing literature (c.f., Crismond and Adams 2012). We add to this literature an understanding of how these challenges fit together: This analysis reveals that the aspects of engineering with which students struggle the most are also the aspects that require students or engineers to engage in learning and applying relevant math and science content. As such, we see challenges emerging with the two goals of integrating engineering design into K-12 classrooms: (1) the ways in which students engaged in engineering design limited their opportunities for successful application and exploration of relevant math and science content and (2) the ways in which students engaged in engineering design missed key complexities associated with professional practice.

In our opinion, this finding does not contradict previous research that has found the positive impact of design activities in learning environments. Previous research has demonstrated that engaging in design can support students in representing (and hence learning) disciplinary content (Harel and Papert 1992) as well as improving student motivation, ownership, and engagement (Barab et al. 2000; Barnett 2005; Hmelo et al. 2000). Our finding adds to this literature, suggesting that educators must work hard to develop contexts and problems that challenge students to engage with the quantitative aspects of engineering. By ensuring that students fully engage with the quantitative aspects, engineering educators can promote the most learning in their students both in terms of math and science and in terms of engineering.

Related research offers a possible explanation and subsequent strategies for supporting students in engaging in these challenging aspects of systematic engineering design. Edelson (2001) argues that students will develop richer conceptual understandings that are accessible in novel environments when the scientific ideas being studied are useful. Building on this work, Kanter (2010) developed and implemented design strategies for supporting students in viewing science content as relevant to their design work and demonstrated student learning in this supportive environment. The first author and colleagues (Kuhn et al. 2006) extend this theory from looking at supporting content learning to supporting student engagement in practices. In this case, we found that students would engage in the disciplinary practice of scientific argumentation if it fulfilled a need—if it was useful. Across these studies, we see an emphasis on developing contexts that help students see the target content and practices as fulfilling a need—as helping them complete their design work.

We therefore posit that the challenges with engaging in, understanding, and valuing the systematic and quantitative aspects of engineering design emerge from educational contexts that do not create a need for students to explore and apply the underlying math and science concepts. This interpretation is consistent with work such as Barnett’s (2005) and Hmelo et al. (2000) studies demonstrating that students emphasized the aesthetics and surface features of their designs over the functionality. These authors argued that the qualitative aspects of the design were tractable, familiar, and provided a sense of forward progress. In other words, students were able to feel successful without focusing on functionality—which would create an opportunity for exploration of the underlying science. As such, we argue that functionality (and the underlying science) did not feel necessary to these students. Extending this work to the current study suggests that the qualitative components of engineering design (i.e., identifying a problem, coming up with multiple solutions, and iteratively refining solutions) provided enough of a sense of forward motion to the students that they had little motivation to move to the more systematic aspects of engineering design (i.e., quantifying the need, developing and using mathematical models, and systematically balancing project goals).

Based on the present findings and building on these studies suggests that the field must work to ensure that the practices and content that we hope students will learn are necessary for the students—that they have a reason to go beyond design to engage in the hard work of integrating math and science into their approaches to design. Not only does this motivate the exploration and application of disciplinary content, but it might also reduce the sense that engaging solely in the qualitative aspects of engineering design is sufficient. To that end, we as educators must develop engineering design challenges in which students are unable to succeed without engaging with the quantitative—with the math and science—aspects of the work [see Kanter (2010) for an example of such curricula].