Introduction

Due to the very nature of design as an interdisciplinary problem solving process, a number of educational approaches have used design as a vehicle for learning math and science concepts (Kangas et al. 2011). The current trend in K-12 science education in the United States is to infuse engineering design as an effective approach for enhancing science education (Fortus et al. 2004; Kolodner et al. 2003). More recently, the engineering design process has become an integral part of the newly proposed Conceptual Frameworks for New Science Education Standards (NRC 2012). A national response to this endorsement of engineering design is evident by newly adopted state-level standards that require students to engage in the engineering design process and/or explore the nature of technology and engineering practices that lead to design. What were once considered “scientific process skills” and conceptual knowledge necessary for young learners to engage successfully in science have now evolved into a focus on major “practices” (NRC 2012: 30). These practices emphasize that conducting scientific investigations and engineering design-based tasks requires not only skill but also knowledge that is specific to each practice. In short, skill and knowledge are positioned as a collective enterprise and no longer as characterized separate entities. According to the NRC (2012), engaging in the practices of science and engineering helps students understand how scientific knowledge develops and the work of engineers as well as links between engineering and science. Although design is a relatively new school topic in the United States, many countries have been teaching Design and Technology (D&T) beginning in the elementary level for several decades and often design is used as an inquiry-oriented pedagogical approach (Department For Education and Skills 1999; Kangas et al. 2013; Ministry of Education 2007). In the U.S., the current science education reform effort is the call for students to engage in design by asking questions, defining problems, developing and using models, planning and carrying out design tasks, and using evidence-based and newly acquired scientific knowledge to reason and argue for the best solutions to a problem (NRC 2012). Assessment of these practices within the K-12 area is both inevitable and currently undefined in the United States.

In this study we explore the use of think-aloud protocol analysis as a method to examine how elementary students (grades 5 and 6) approach engineering design. Think-aloud protocol analysis has been used in many contexts (Atman and Bursic 1998; Adams et al. 2003); however, little research has been conducted using this method as an assessment strategy for examining how elementary students engage in the engineering design process and how enduring these practices may be from one task to another.

Related literature: concurrent think-aloud protocols

The following review of literature on concurrent think-aloud protocols provided a rational for using CTAs a method to assess students design capabilities within the context of learning science. The review of literature helped the researchers identify examples of CTAs as a research method for understanding cognition within problem solving as well as a method to analyze design thinking in expert and novice designers. Although CTAs have been used to analyze adult expert and novice designers’ abilities (Ball et al. 2004; Donovan et al. 1999; Perez et al. 1995), prior to the review of literature it was unclear to the researchers if CTAs could be effective research methods for elementary student designers. Currently, there is a void in the literature regarding the use CTAs among elementary school students’ design thinking. The following discoveries within CTA literature provide guidelines for proper procedures for conducting CTAs to capture design thinking.

CTAs capture problem solving cognition

Concurrent think-aloud protocol (CTA) is a research methodology that requires one or more participants to speak aloud his/her thoughts as he/she performs a task or activity (Ericsson and Simon 1993; van Someren et al. 1994). CTAs began as a popular methodology in cognitive psychology (Cooke 2010). Early examples of CTAs include the work of Otto Selz in the 1930s who investigated creative thought processes of individuals, and the work of de Groot in the 1940s used the think-aloud method to study cognitive strategies employed by expert chess players (van Someren et al. 1994). Ericsson and Simon (1993) assert that concurrent think-aloud and talk-aloud verbal reports are the closest reflection of cognitive processes of individuals not modified from their natural state, making it an excellent research methodology to understand human cognition. Ross (1984) employed verbal protocols to discover that participants are capable of retrieving previously encountered problems and solutions to those problems when presented with similar but different new problems. van Someren et al. (1994) theorized that think-aloud methodologies are easier approaches for a participant to engage in naturally when compared to more structured elicitation techniques because the participant is able to use their own language within their own natural dialogue.

Allowing students to communicate their design thoughts in a genuine way is critical when working with children; it removes the risk of the researcher imposing upon the participant natural cognitive strategies while designing (Ericsson and Simon 1993). Allowing students to express their design thinking intuitively will be critical when collecting CTA data with elementary students.

CTAs capture design thinking

van Someren et al. (1994) provide discussion on the difficulty of understanding the thought processes of designers. van Someren et al. cite examples of research on professional designers such as architects requiring assessment methods beyond interviewing an architect on how he or she goes about designing. Designers, even expert designers with multiple years of experience, struggle to self-reflect on the thought process employed while designing. Furthermore, van Someren et al. also indicate that the study of drawings of architects provides limited understanding of the process taken in designing because the researcher is looking at a design ‘product’ instead of the process itself. The order of approach to design is also important in understanding how the designers will lead to a design solution. Atman and Bursic (1998) endorse the power of using think-aloud protocols:

By measuring both the ‘product’ and the ‘process’, we can then explore whether a relationship exists between the type of process a student uses and the quality of the final design. Knowing this relationship, we can then distinguish between good and poor processes and indicate specific problems that must be addressed as we teach design. (p. 130)

Dorst and Cross (2001) identified that designers navigate back and forth from problem space to solution space as the designer works through the design process. The problem space is the phase of the design process where the designer focuses on understanding the problem itself. It is the “What am I asked to do?” question that students often ponder when presented with a task or design challenge? In the think-aloud protocol session, participants work within the problem space often reading and re-reading the transfer problem, seeking to identify constraints and criteria, asking questions about the problem, any dialogue where the participants seek to clarify the assigned task (Cross and Dorst 1999). Understanding the thought process of a designer requires a concurrent study of the designer while the individual or teams of designers are engaged in the design process.

Donovan et al. (1999) state that research on expert designers has revealed that when given the opportunity to speak aloud their thoughts, designers were able to express their conceptual understandings and reflect on prior experiences as they developed deeper understanding. Atman and Bursic (1998) make the case to use verbal protocol analysis as an assessment method on design-based and problem-based pedagogical approaches teaching engineering design in undergraduate engineering programs. Atman and Bursic proposed that researchers should employ think-aloud verbal protocol analysis to assess students’ design processes as they solve open-ended engineering design problems.

In this study, the review of literature provides rationale for the application of CTAs and key procedures for developing and conducting think-aloud protocol sessions with elementary school students. For example, van Someren et al. (1994) indicated that think-aloud protocols could be easier methods for capturing individuals’ thoughts in a natural way instead of using direct elicitation techniques with these young designers. The researchers for this study gleaned from the literature review the following:

  1. a.

    CTAs can be used to understand the cognitive process taken by designers. CTAs were selected as a research methodology to better understand how children learn design as an approach to teaching science.

  2. b.

    CTAs provide opportunities for designers to express design thinking by naturally engaging in design. The CTA procedures for this study were created to allow students to express their design thinking naturally by structuring the protocol sessions similarly to how the students experienced design in the classroom. This was achieved by placing students in triads with their classmates and providing students with transfer design problems similar to the in class design task.

  3. c.

    Analysis of the patterns in dialogue as well as sequence of cognitive strategies within CTA sessions provide insight into how designers navigate through the design process. The pattern of cognitive capabilities was carefully studies to better understand how students have learned design.

Theoretical framework

This study draws upon two theoretical constructs: (1) transfer of learning, and (2) community of practice. The research frame for employing a think-aloud protocol in conjunction with a transfer problem is based on the transfer of learning theory (Bransford et al. 1999). Transfer of learning is a theory that provides students the opportunity to transfer existing and newly acquired knowledge to new situations. The transfer of learning theory suggests that when transfer is identified, it is an indicator of understanding. Royer (1986) expresses the theory this way: “Used as an index of understanding is equivalent to the idea that the ability to transfer learned information is evidence that understanding is present” (p. 95). Royer also indicates that near transfer occurs when problems are solved that are similar to those problems encountered previously. The concept of transferring new and existing knowledge to new situations is a target outcome for both problem and project based approaches to K-12 engineering design (Dixon and Brown 2012; Prevost et al. 2009). A near transfer approach was used in this research study by creating transfer problems that were similar to the design task assigned to students in class, but contained different context within the transfer problem scenario. More details regarding the creation of transfer problems will be discussed later. The effectiveness of transfer of learning can be increased through metacognitive approaches to learning that allow students to reflect upon their learning in order to improve strategies for problem solving (Bransford and Schwartz 1999; Brown 1978; Flavell 1976) and engineering design allows for this type of metacognitive development.

The transfer of learning also entails an examination of the community in which the learners engage. Participation among newcomers as well as experienced members is defined within a community of practice (Lave and Wenger 1991). Lave and Wenger describe this as legitimate peripheral participation when a community of practitioners support the learner as he or she moves from the apprentice level in understanding of knowledge, skills, and practices toward mastery as they participate in “in a social practice of a community,” (1991, p. 29). Researchers studying approaches to the Design and Technology National Curriculum programme in England also identify situated learning and communities of practice learning as critical to teaching design and providing students with authentic learning experiences (Hennessy and Murphy 1999; Nicholl et al. 2013). Students participating in this study are engaged in authentic engineering design activities. Students are grouped into design teams by the teacher and openly share individual design ideas with their design team and later report out to the entire class (community of learners). As a whole class, students are encouraged to ask challenging questions about their design ideas and those of other design teams, thus, allowing each individual to build their own understanding of science and engineering design.

Unique to this study is the applications of the think-aloud approach whereby students are exposed to a new task (transfer problem) in a smaller group setting (a sub-set of the community of learners) away from the entire class. Researchers carefully examine if and how students use, apply, and transfer what they learned in their entire class setting in this smaller context. This learning may include key engineering and science practices as well as addressing specific scientific concepts.

The primary purpose of this study was to examine the implementation of the think-aloud protocol as a methodology for characterizing elementary school students’ abilities to engage in the engineering design process. Researchers employed this method in an effort to identify and classify key reasoning skills grades 5 and 6 students used while engaging in an authentic engineering design-based task.

Research questions

The questions guiding this study include the following:

  1. 1.

    How do grades 5–6 school students conceptualize and learn design?

  2. 2.

    Which aspects of the engineering design process do students tend to emphasize?

  3. 3.

    Which aspects of the engineering design process do students tend to overlook?

Context of the study

The context of this study is a National Science Foundation Math Science Targeted Partnership entitled Science Learning through Engineering Design (SLED) (see https://stemedhub.org/groups/sled). The partnership entails a collaborative effort across four colleges within a large, research-intensive university and four school corporations located in the central Midwest region of United States. The primary goal of the partnership is improve grades 3–6 student achievement in science learning through the engineering design process. Over the course of 5 years, approximately 100 pre-service and 200 in-service teachers and over 5,000 students in grades 3–6 will participate in the partnership. For the purpose of this study, the researchers analyzed data collected during the first year of implementation of the initiative that focused on grades 5–6 students.

This research study was drawn from a large urban school district located in the north central Midwest of United States. In the 2010–2011 school year, there were 7,075 students enrolled in the school district, with 59.5 % of the student population identified as White/Caucasian Indiana DOE 2012. The school district had 20.5 % Hispanic, 12.3 % Black/Non-Hispanic, 0.7 % Asian, 6.6 % multiracial, and 0.4 % American Indian, with just over 60 % on free- and/or reduced-lunch. During the 2010–2011 school year, there were 1,008 students enrolled in the grade 5–6 intermediate school (school site #1), with 59.9 % of the student population identified as White/Caucasian (Indiana Department of Education 2012). The school had 18.1 % Hispanic, 13.4 % Black/Non-Hispanic, and 7.7 % multiracial, 0.4 % American Indian, just over 60 % on free- and/or reduced-lunch.

Participants

This study used criterion sampling whereby researchers selected cases to satisfy a specific criterion (Gall et al. 2007). Participants for the think-aloud protocols were purposefully selected by their classroom teachers. Teacher recommendations were based upon: (a) students’ ability to express themselves verbally, (b) student record for successfully functioning as a contributing member of a design team, (c) the student’s willingness to volunteer for the study, (d) returned required university internal review board (IRB) parent consent and student assent forms. Triads of student design teams were created for each classroom participating in the research. Welch (1999) suggests that pairing or grouping student participants allows for the design process to emerge naturally as most design efforts occur in groups of two or more people working together. All groups were heterogeneous, similarly to how most groups were formed in their classrooms. Table 1 outlines the distribution of students by classroom and within each triad selected from its respective class at school site #1. For example, Classroom #1 has a total of 53 students and one male student and two female students were selected as a triad.

Table 1 Classroom demographics: class size and gender information for triads
Full size table

Methods for data collection and analysis

Concurrent think-aloud procedures

There are a variety of approaches to providing instructions for the protocol session ranging from simply requesting participants to speak aloud their thoughts to requiring participants to explain the approach they are using (Dunker 1926; Ericsson and Simon 1993; Patrick 1935; Smith 1971). Researchers with the help of school personnel located a quiet location within the school to conduct the protocol session. A triad of students was selected from the classroom and taken to the protocol session location. Students were positioned at a work table opposite a digital video camera and the researcher. The camera was positioned to capture students’ faces as well as the table surface. This allowed researchers to analyze facial nonverbal cues such as eye movement as well as capturing sketching and handwriting on provided paper. The session began with a review of IRB assent forms, and students were informed that the protocol session required students to discuss a design problem that they would be given and to speak aloud their design thoughts to the group. Proper assents were granted with student signatures only after parent consents were collected with their signature of approval. Next, the researcher started the camera and handed students a paper containing the transfer design problem. One student was assigned to read the transfer problem out loud. Once the problem was read, the participants could begin their design thinking dialogue and were encouraged by the researcher to continue to speak aloud their thinking throughout the protocol session. Additionally, participants were given sketch paper, allowing each participant to graphically communicate their design ideas. Design sketches were digitally scanned, labeled, and included in CTA data files to provide additional evidence for analysis of protocol sessions.

Transfer problems

Robertson (1990) defines transfer problems as “problems that are structurally, but not conceptually unfamiliar, to the solver—have long been used as a measure of understanding” (p. 253). When creating the transfer problem, careful consideration in crafting a problem statement that provides an opportunity for participants to transfer their science knowledge obtained from the activity is critical. Furthermore, the transfer problem must be authentic to design. Cross (1994) argues that all design problems have (a) a goal; (b) constraints to work within; and (c) criteria to determine an achieved successful solution. The transfer problems were created to share characteristics of an ill-defined, complex, situated, and dynamic design problem as defined by Jonassen (2000). Researchers in this study also developed authentic design transfer problems using these features described by Cross (1994). Careful consideration was also taken in selecting a context to which students could relate as well as excite and inspire students. Transfer problems created for the study included scenarios describing situations that students might experience in their daily lives as the context for the transfer problem. For example, one transfer problem described two siblings who get a Frisbee (flying disk) stuck in a tree and want to develop an effective way get it down. The transfer problem encouraged the students to use their knowledge from the classroom activity to devise a solution to help the children retrieve the stuck Frisbee. The scenario was created as an authentic life experience where students could self-identify and in a context to which they could relate.

School site #1

Think-aloud protocol sessions were conducted in each classroom for each implemented design activity. A total of 33 CTAs were collected during the 2011–2012 school year. In this study, we present the CTA data collected at school site #1 from seven classrooms observing seven different triads of student design teams. The design activity, Prosthetic Limb, was implemented across all 5th grade classrooms at school site #1, see Appendix 1. This mass implementation provides an excellent opportunity to compare CTA data across multiple classrooms at the same school, during the same time of the school year.

Design activity: prosthetic limb activity

The prosthetic limb design activity required students to work in teams to build a prototype for a prosthetic leg to function like a human leg joint and strike the ball (see Appendix 1). Lessons delivered by the classroom teacher for the prosthetic limb included information about the human musculoskeletal system as well as recent developments in prosthetic technology. The students were given some basic model building materials and common fasteners to create a leg model to kick a plastic golf ball. Similar model building tasks have been developed to teach about human musculoskeletal systems and the functions of a human elbow (Penner et al. 1997). Table 2 profiles an example of the sequence of instructional activities implemented by a classroom teacher within the context of the prosthetic limb task.

Table 2 Overview of instructional activities implemented in classroom #7 in prosthetic limb unit
Full size table

Transfer problem: paper football kicker

The transfer problem for the prosthetic leg activity was created to allow students to apply their knowledge about prosthetics as well as their science knowledge about the human musculoskeletal system. Think-aloud protocol participants were required to design a device that would mimic a human finger flicking motion to “kick” a paper triangle football to propel it through a model football goal post at various distances (see Appendix 2). Student participants were also required to consider how the device would be tested to determine the effectiveness of the design. Again, the transfer problem was created in a context that students could relate and identify while also providing opportunity to apply science knowledge related to prosthetics. A level three approach was employed to identify transfer of knowledge as described by Ericsson and Simon (1993) that required students to verbalize their design thinking and link the current design problem to their previous thoughts and design experiences.

Data analysis

Halfin (1973) used a Delphi method to identify the cognitive strategies universal in the works of ten highly successful professional engineers and inventors. Halfin studied the writings of such well-known designers as Frank Lloyd Wright, Orville and Wilber Wright, Buckminster Fuller, and Thomas Edison, to name a few. Halfin’s research findings generated a list of seventeen cognitive strategies commonly used by designers. Halfin’s cognitive strategies list created the defined coding categories for data analysis of this study as suggested by Merriam (1998). Using this methodology allowed the researchers to investigate students’ ability to transfer their design and problem solving capabilities to a transfer problem (an open-ended, ill-defined problem) (see Appendix 3).

Results

In this section, we present verbal protocol data captured during the CTA session. The protocol analysis process involves the researcher applying Halfin codes to the dialogue of the student triads. The protocol analysis allows the researcher to then organize these codes by time (minutes) and percent time on code (%), (see Tables 1, 3). This protocol analysis method allows the researchers to then organize the data by total time on code and create charts illustrating how each student triads used their design session time and employed design thinking as identified by the Halfin code (see Fig. 1). Table 4 provides results from inter-rater reliability assessment on the coding scheme of two raters computing Pearson r for measure of consistency. The two researchers coded all seven protocol sessions independently using OPTEMP software (Hill 1997). To indicate consistency in coding, the researchers attempted to achieve a correlation close to 1 indicating a consistency and reliability in coding. Pearson correlations computed for this data set ranged from 0.927 to 0.994 indicating a high level of consistency. Ericsson and Simon (1993) indicate that reliability of coding should increase when the entire protocol is segmented and coded independently. The researchers segmented and coded the entire seven protocol sessions independently as oppose to one researcher segmenting protocols and then two researchers coding those segments. The results of high Pearson correlations and the procedure of two independent researchers segmenting and coding the entire protocol session are evidence of consistency in rater reliability.

Table 3 Cognitive strategies (Halfin code) by total time (minutes) on code
Full size table
Fig. 1
figure 1

Coded results of cognitive strategies for % time on task for the prosthetic leg activity

Full size image
Table 4 Inter-rated reliability SLED School #1 Prosthetic Limb results R1 = rater 1, R2 = Rater
Full size table

Figure 1 shows a general pattern of cognitive strategies for the seven triads of grade 5 student design teams across seven classrooms, one triad per classroom. (Appendix 4 contains black and white versions of the pie charts). The Halfin code key at the bottom right of the figure provides identification of 11 of the Halfin codes used by participants. Reviewing the pie charts, readers will note important cognitive strategies that indicate problem spaces and solution spaces of the design process (Dorst and Cross 2001; Kruger and Cross 2001). Color code red indicates Defining the Problem (DF) and Analysis (AN) is coded blue; these two Halfin codes represent participants working within the problem space of the transfer problem. During the problem space of the protocol, students are working to understand the problem and identify constraints and criteria. Color code green indicates Designing (DE) and indicates the solution space where students are brain storming and refining design solutions. Classrooms 1–6 yielded ranges from 43 to 67 % for DE (designing) and 10–18 % for DF (defining the problem) and 4–23 % AN (analysis). The exception to this was the design team for the classroom #7. Classroom #7 participants spent more time in the problem space than any other triad, resulting in 36 % of their time defining the problem, 27 % of their time analyzing the problem. However, classroom #7 only spent 22 % of their time designing.

Classroom #7 spent over 23.2 % more time in the problem space compared to the other six participating classroom’s group mean scores and less than-half the time or 27 % less time in the solution space when compared with the group mean score percentages (see Table 5). One might conclude that participants in classroom #7 were trapped in the problem space, a typical occurrence for inexperienced designers and a phenomenon that restrictions designer’s capacity to efficiently create design solutions (Cross and Dorst 1999; Dorst and Cross 2001; Kruger and Cross 2006). However, additional data analysis using NVivo software revealed that classroom #7 were not stuck in the problem space; rather, the design team spent the first several minutes carefully framing the problem with multiple iterations back to the problem definition throughout the design process (see Figs. 2, 3). Jain and Sobek (2006) have discovered that student designers who give special attention to defining the problem are generally more efficient designers. Classroom #7 triad’s careful attention to the problem space was appropriate and may have led to their overall design efficiency. Furthermore, Cross and Cross (1998) observed that effective designers locate a way to view or frame the problem in a creative and inspiring way to challenging themselves to innovate a solution. The technique of framing the problem and returning to it throughout the process is an example of the design team functioning at an expertise designer level (Atman and Bursic 1998; Dorst and Cross 2001; Dym et al. 2005; SchÖn 1983).

Table 5 Halfin code percentage comparisons of classrooms 1–6 compared to classroom 7
Full size table
Fig. 2
figure 2

Classroom #7 design activity timeline

Full size image
Fig. 3
figure 3

Classroom #2 design activity timeline

Full size image

The data revealed that classroom #7 triad diverted from the average results of the sample drawn from school #1 regarding the time spent on defining the problem. One possible explanation for the results may be due to the fact that the classroom #7 teacher had extensive experience (over 5 years) teaching science through a design based approach and additional professional development on design-based teaching. The other six classroom teachers did not have extensive experience teaching design based activities beyond a 2-week summer professional development program created by SLED. Classroom #7 triad began the design session by having each design team member creating a one sentence statement that defined the task within the transfer problem. This was not observed in other transfer problem sessions; hence, it was determined that this was an approach unique to the classroom #7 triad. Also classroom #7 triad worked cooperatively to refine this statement during the CTA session until all members of the team were satisfied with the problem statement. Classroom #7 triad next steps were to create a list of materials that they would need to address the problem. The design team also sought to identify the client and user within the transfer problem. Throughout the think-aloud session, classroom #7 triad were revisiting these statements as well as ensuring that they were meeting the identified constraints and criteria. Although classroom #1–6 design teams also addressed these phases of the problem space at some point within their dialogue, classroom #7 design team approached these phases of the problem space in a systematic manner to ensure that all design ideas would adequately address the problem defined. Furthermore the design team frequently returned to problem definition throughout the protocol session to remain focused on solving the correct problem as they worked though the design task.

Comparing timelines

The researchers determined that careful study of the sequence of the coded cognitive strategies would help better understand how triads were approach design and managing their time during the protocol session. Classroom #2 was selected as a typical approach for classrooms 1–6 to be further analyzed using the NVivo software. The selection criteria for classroom #2 was based upon comparing mean scores of coded cognitive strategies from classroom #2 with group mean scores for all 33 protocols collected during the 2011–2012 school year. Classroom #2 cognitive strategies percentages were the most representative to the group mean results. Figure 2 illustrates the typical timeline of iterations of cognitive strategies throughout the design dialogue. Figure 2 reveals that classroom #2 defined the problem at the onset of the protocol session and rarely (twice) and briefly (seconds) returned to the problem statement. This was a typical pattern seen in classroom 1–6 protocol sessions.

Classroom #7 also used the least amount of time of all the sessions for a total of 15:51 min, working efficiently to complete the design task (see Fig. 3). Working as efficient design team is important to the success of this type of design task because teachers have limited class time to dedicate to designing and brainstorming session. Classroom #7 triad effectively engaged problem scoping and was a prime example of a triad of students moving into expertise level of designing. Expert designers use a systematic approach to defining the problem as soon as they start designing and carefully and accurately frame the problem (Cross and Cross 1996; Dorst and Cross 2001; Kilgore et al. 2007).

Conclusions and implications

Vital to the success of design-based education is the careful assessment of the use and application of the design process as an approach to transfer learning to various life applications and problem encountered. Using a concurrent think-aloud protocol as a method to assess student’s cognitive capabilities provides insights into how students manage their time during brainstorming sessions.

Our first research question was: How do grades 5–6 school students conceptualize and learn design? Findings of the CTAs protocols indicate that teaching grades 5 and 6 students’ systematic approaches to solving problems can be transferred to new contexts. Students across all triads in this study were able to transfer their learning of essential features of the engineering design process including but not limited to practices such as problem identification, analysis, and refinement of design solutions. Additional data analysis illustrated that classroom #2 did not iterate back and forth from problem space to solution space compared to classroom #7. This suggests that these students may not understand that design is iterative and therefore requires designers return to, review, and in some cases refine problem definition as they design solutions to ensure that they are solving design problems that will meet clients’ needs.

Our second research question was: Which aspects of the engineering design process do students tend to emphasize? Results from CTAs collected for this study indicate that students across all triads, with the exception of classroom #7, emphasize brainstorming design solutions by spending on average half of the protocol session (49 % of time) generating solutions to the transfer problem. Results from CTA protocol sessions also reveal that student participants spent over 20 % of their time in the analysis phase of the design process trying to identify and address constraints and criteria the transfer problem. Considering that many students struggle to properly identify constraints and criteria, this is a promising finding (Hill 2006).

Our last research question was: Which aspects of the engineering design process do students tend to overlook? The findings from this study indicate that students often overlook prototype testing. Students in less than half of the protocol sessions addressed testing of the prototype in their design discussions and of the three triads that did address testing, 5 % or less time was dedicated to this cognitive strategy coded TE.

There are several limitations to this study. First, the protocol approach was limited to a finite number of student teams. Additional data collection is clearly warranted in order to make generalizations about the use of the protocol in the elementary school setting. Second, the study is limited to data from student participants. Data from classroom teachers including observations of their enacted practices, interviews regarding their pedagogical decisions, and lesson plans depicting their proposed plans from implementation, would provide a more comprehensive perspective on how students learn and engage in engineering practices.

Implications for this study suggest that more research using CTAs should be conducted to further investigate how elementary students learn and teachers use design experiences to enhance science learning. Instructing students how to frame design problems and manage brainstorming sessions are critical to students’ success as designers. Teachers incorporating the engineering design process into their lessons should provide multiple opportunities for students to explore science content, skills, and practices related to the design problem as well as lessons designed to explicitly teach required science content in order for learning transfer to occur (Rennie et al. 2012). Some possible research questions for future studies focused on teaching science through engineering design include the following:

  1. 1.

    Under what learning conditions do students transfer science conceptual understandings into their design dialogue?

  2. 2.

    How do students translate and apply their existing science conceptual understandings into their design dialogues?

  3. 3.

    In what ways can students construct new science conceptual knowledge when engaged in design dialogues?