Introduction

Despite around 1,000,000 incoming high school freshmen expressing interest in science, technology, engineering, and mathematics (STEM) in the United States, by some estimates, only 42.6% will actually maintain STEM career interest through high school (Munce and Fraser 2013). However, this loss of interest is not the full picture; accounting for students who develop an interest in STEM careers during high school, the proportion of young men interested in STEM remains unchanged whereas the proportion of young women decreases (Sadler et al. 2012). There is also evidence that while girls often achieve at an equal or higher level than boys, girls’ STEM interests tend to be lower and decline more rapidly than boys’ (Brotman and Moore 2008). These motivational differences impact college major and career trajectory. The percentage of female students is lower than male students for STEM majors (Yoder 2016; National Center for Educational Statistics 2016), and later in the workforce (US Bureau of Labor Statistics 2017; Ashcraft et al. 2016). Collectively, these findings speak to the gender gap in STEM fields and demonstrate that as the gap widens during secondary education, there are lasting impacts on students’ career trajectories. Adolescents’ interests and experiences grow to shape their career choices (Betz 2006) and therefore this is a critical time at which we might intervene.

What factors are affecting this decline in interest? Expectancy-value theory (Eccles et al. 1983) suggests two main factors, expectancy for success and value of the experience engaged in, are needed to maintain interest. Toward a solution, how could these factors be leveraged to foster students’ STEM interest? Using expectancy-value theory, we parse some evidence of gender disparities in STEM fields and describe a 9th grade, soft robot design curriculum which might positively impact students’ interest and choices for STEM. The soft robot design curriculum is theoretically aligned with growing students’—especially young women’s—expectancies for success and subjective value in STEM through an accessible approach that is culturally relevant and socially important. Using data collected from the first year of high-school implementation, the present study uses multilevel models, also called mixed effects models, to answer the following research question: Do soft robot design experiences improve engineering expectancy for success, subjective value, and interest as compared to traditional robotics experiences? Next, we follow-up to investigate the changes in individual student perceptions following the curriculum.

Robots in education

A variety of engineering and technology activities, both formal and informal, are used to capture students’ attention and interest as they approach career decisions; among these, robotics is a growing part of formal education. Robotics has been demonstrated as an effective vehicle for hands-on learning of technical and tinkering skills (Barker et al. 2012; Hamner et al. 2008; Stubbs and Yanco 2009). There are a number of robotics kits and tutorials to facilitate classroom implementation (e.g., Fischertecknic or LEGO robotics). A review of robotics curricula showed that a number of learning outcomes can be targeted, from specific STEM concepts—physics, force, or mathematics subjects—to broader competencies—communications, problem-solving, or creativity (Benitti 2012; McGrath et al. 2008). In addition to in-class robotics, informal or extracurricular experiences with robotics have bolstered the popularity of educational robotics. Competitions such as FIRST or LEGO robotics competitions are used to leverage the hands-on and interesting aspects of robotics to grow students’ interest. However, there often remains disproportional participation of boys and girls, and, when girls do participate, they are often in non-technical roles (Center for Youth and Communities 2011). Subsequently, one study noted that girls are “more likely to report impacts on attitudes related to teamwork or communication skills, and boys [more likely to report] gains in technical skills” (Center for Youth and Communities 2011). While there is some mixed or anecdotal evidence of the success of robotics programs to influence girls’ engineering perceptions (e.g., Hendricks et al. 2012), the widespread use of robotics and the persisting challenges of gender disparity frame robotics as a rich context for intervention. These findings are consistent with other observations regarding robotics and gender, all of which note that educational robotics programs do little to decrease gender gaps or increase the technical role of female participants (Ball 2005; Hartmann et al. 2007; Milto et al. 2002; Rusk et al. 2008; Skorinko et al. 2010; Stein et al. 2004; Terry et al. 2011; Diekman et al. 2011).

In contrast to existing educational robotics experiences, we have made a paradigm shift to design materially soft robotics, transforming the design and fabrication process, while targeting similar learning outcomes. The novel soft robotic design experience uses a silicone rubber, one of many material approaches for soft robotic fabrication. While silicone may have been familiar to students, the silicone fabrication process was unfamiliar. Moreover, the design of elastically deformable soft robotic actuators with silicone was a new experience. Several advantages of soft robots and aspects of the materials used for fabrication position soft robotics to be favorable for perceptions of success and value among students. Soft robotics is an emerging type of robotics (Bao et al. 2018) which use compliant, flexible components for construction (e.g., rubber and fabric) rather than hard, rigid materials (e.g., wood and metal). A great number of new robotics applications may be available with this change, as different material properties are incorporated into robotics systems (Trimmer 2013). Generally, soft robot systems are more versatile (Trimmer 2013), comfortable, and safer for human interaction (Trimmer et al. 2013; Roche et al. 2017) than rigid robots. They also offer new types of motion that are difficult to achieve in traditional robotics design (Ilievski et al. 2011; Lipson 2014). Because the surfaces deform to interact with objects they are able to handle delicate and irregularly shaped objects well—these robots also embed safety at the material level (Ilievski et al. 2011; Majidi 2013). Another affordance of soft robotic systems is their inspiration from biological systems—these robots represent an opportunity to learn from and mimic natural systems (Lipson 2014).

By changing the nature of robotics design experiences, soft robotics may dispel misconceptions or stereotypes about abilities to succeed and find value in engineering practice. Furthermore, this new robotics design experience is situated in an emerging engineering context, introducing students to future career possibilities that they may find interest in. One study by Hartmann, et al. (2007) suggested that “the materiality of robotics itself plays an important role here i.e., the fact that it [robotics] already comes along as gendered material,” implying that a change in material itself may allow females to see robotics differently. Furthermore, another study by Terry et al. (2011) found that young girls displayed greatly increased interest in robotics when they were able to identify a direct connection between robotics and human benefit, such as with biomedical and surgical robots. This argument aligns with results from Diekman et al. (2011) who suggested that content with societal implications is more likely to interest female learners and that STEM careers are not typically associated with societal goals. The value of aligning learning experiences with socially relevant and culturally situated contexts was found to be critical for engaging underrepresented populations in engineering through messaging campaigns by the National Academy of Engineering in their 2008 report, Changing the Conversation: Messages for Improving Pubic Understanding of Engineering. Next, we describe expectancy-value theory (Eccles et al. 1983), which we leveraged as a framework for our research.

Expectancy-value theory

Achievement motivation theories suggest that specific beliefs towards the outcome and value of an activity are critical for choices to explore, give effort, and persist in those activities (Wigfield and Eccles 1992). Expectancy-value theory (Eccles et al. 1983) is one motivational theory which builds on previous theories by suggesting that these beliefs are contextualized among other social, historical, and cultural factors for each student, and that both expectancy of success and value for the task are required for students’ choices, as shown in Fig. 1 (Nagengast et al. 2011; Trautwein et al. 2012; Wigfield and Eccles 2000; Wigfield et al. 2004). The differentiation of expectancies and values is a similarly important contribution of the theory, and both have been related to achievement and choices (Schunk et al. 2014).

Fig. 1
figure 1

Expectancy-value theory model from Eccles and Wigfield (2002). Shaded elements highlighted represent the expectancy-, value-, and choice-related outcomes targeted in this research

Full size image

According to the theory, students’ judgments on likelihood of success and task value are especially instrumental and immediately antecedent to engagement choices. Expectancy for success relates to the question, “Am I able to do this task?” and is complimented by perceived value, “Why should I do this task?” (Schunk et al. 2014).

One factor described in influencing a student’s achievement and choices is expectation for success. Expectations for success may be associated with either short- or long-term goals (Eccles and Wigfield 2002). This is a “belief about how well they will do on an upcoming task” (Wigfield et al. 2004) and is distinct from broad beliefs about ability level. Bandura (1977) similarly separated beliefs about ability and outcomes, noting that an ability to perform an action is separate from the consequences of the action. However, in practice it is difficult to distinguish between these levels of belief—they tend to be measured in a similar way (Eccles and Wigfield 2002). Expectancies are closely related to other conceptions of self-belief such as self-concept or self-efficacy (Trautwein et al. 2012).

The other factor is a value of the task itself. Eccles et al. (1983) distinguished different types of value that might be placed on a task: intrinsic value, attainment value, utility value, and cost. Intrinsic value, also called interest value, is enjoyment from doing the task (Hulleman et al. 2008). This leads to engagement and persistence at the task. Intrinsic value in expectancy-value theory is conceptually similar to intrinsic motivation, though they come from different frameworks (Wigfield et al. 2004). Attainment value stems from the importance of doing well on the task. For example, a task may be important for confirming an aspect of one’s identity. Utility value is based on the usefulness of the task for current and future goals, especially career goals. “It is related more to the ends of a task than to the means” (Schunk et al. 2014); a task may be seen as instrumental for achieving a goal, and completed regardless of interest or enjoyment. Finally, costs are the negative aspects of engaging in a task. These might include necessary effort, the tradeoffs of participating in the activity (e.g., missing out on something else), or risk of negative outcomes (e.g., embarrassment).

Expectations for success and the value placed on an activity are intuitively related. “Even during the very early elementary grades children appear to have distinct beliefs about what they are good at and what they value” (Wigfield and Cambria 2010). Research has consistently demonstrated that expectancies and values develop over time (Denissen et al. 2007; Eccles et al. 1983). These beliefs are reciprocally related in that successful experiences may enhance intrinsic value and the value of a task may lead to students to seek further experience and proficiency (Schunk et al. 2014; Trautwein et al. 2012; Wigfield and Cambria 2010). Both expectancy beliefs and subjective values are related to achievement and choices, as outlined in the expectancy-value model, though expectancy beliefs better predict achievement while subjective values better predict choice (Meece et al. 1990; Nagengast et al. 2011). Still, evidence attests that both constructs interact to lead to student choices (Nagengast et al. 2011; Trautwein et al. 2012).

In the theory, each student’s expectancies for success and values for the task are also fostered by their own self-schemas and goals, affective memories, and interpretations of past experiences. For example, the student’s goals and aspirations—the beliefs about the kind of person they are and want to be—will shape effort on tasks they deem instrumental and doable. Past memories of an experience will come to shape the values attributed to different tasks: “Achievement outcomes give rise to general positive or negative emotional reactions to the outcome. Thus children should come to value those activities they have succeeded on more than the activities on which they have failed” (Wigfield and Eccles 1992). It is also important to note that the social context of tasks is interpreted by the student. Interpretations of gender or activity stereotypes or relative definitions of success, failure, and ability are filtered through the student’s perceptions to subsequently affect their expectancies and values for the task (Eccles et al. 1983). Collectively, these factors represent the left part of Fig. 1. As suggested by the specificity of goals and past experiences, the social contexts, expectations for success, task values, and, ultimately, choices and achievement are domain specific (Bong 2001; Denissen et al. 2007).

Use of expectancy-value theory

Expectancy-value theory has been used in similar contexts previously to explain student performance. Berland and Steingut (2016) used expectancy-value theory to explain variation in student efforts towards applying math and science knowledge in engineering contexts. Their work demonstrated attainment value was a significant predictor in student effort and expectancy-value theory was appropriate for modeling high school student STEM integration effort. Jones et al. (2010) used expectancy-value theory to model college engineering freshman beliefs as predictors for achievement and career plans. Their research suggested that expectancy more strongly predicted achievement and value more strongly predicted career plans. Both of these studies used expectancy-value theory in contexts similar to our research indicating it as an appropriate and viable theory to drive our investigation. Using the lens of expectancy-value theory, the scarcity of women in STEM fields mentioned in the Introduction comes, in part, from a lack of interest (value) or lack of confidence (expectancy) regarding pursuit of those activities, or both. The theoretical implications include interventions that address stereotypes, expectations for success, and task values of STEM activities for students (van Aalderen-Smeets and Walma van der Molen 2016).

The present research

The present research considers the efficacy of our soft robot design experience compared to a traditional robot design experience following the first year of implementation in high-school classes. Prior to lesson delivery, participating teachers received standardized materials related to both lesson types, through the STEM Center for Teaching and Learning (STEM-CTL), the professional development arm of the International Technology and Engineering Educators Association (ITEEA), and our research team. Teachers received a detailed lesson plan which used the 6E model (Burke 2014) and included engagement, exploratory, explanatory, engineering, enrichment, and evaluation components, as well as video and troubleshooting materials for teacher preparation in soft robotics fabrication. Teachers also participated in content-focused professional development which engaged them in the curriculum and offered an opportunity for reflection on the lesson materials. During the next school year, each teacher received the necessary materials and implemented the soft robot curriculum in about one-half of their classes and the traditional rigid robotics design curriculum in the other half. All participating students received surveys administered to evaluate perceptions. Reflections by teachers and researcher observations demonstrated fidelity of the implementation in both lessons—minimal changes were made on the basis of teacher’s professional judgment and classroom needs (e.g., adapting to school interruptions).

Informed by expectancy-value theory, the underlying hypothesis of this work was that a paradigm shift to soft robotics curricula would impact engineering expectancy for success, subjective value, and interest of participating students compared to traditional robot design experiences. Sub-components of this hypothesis were three concepts of expectancy-value theory under investigation—expectancy for success, subjective value, and interest—and the particular effect of this experience for young women. We hoped that participation in the soft robot lesson might grow students’ perceptions at least as well as the traditional robotics lesson and that the gender gap might be lessened through the experience. The next section describes our research and assessment approach to evaluate the soft robot design experience in contrast to a traditional robot design experience.

Methods

Participants and design

The pilot implementation of the soft robotics curriculum involved seven high-school technology and engineering teachers from four schools in a Mid-Atlantic district. Each teacher had at least two sections of a required 9th grade course, Foundations of Technology (sometimes called Foundations of Engineering), in the Engineering byDesign curriculum provided by the STEM-CTL. The 9th grade curriculum includes a traditional robot gripper unit which afforded the opportunity to have a quasi-experimental design where both conditions were taught by each teacher: the control condition was the traditional robotics unit and the treatment was the soft robotic design unit. Participating teachers reported between 4 and 25 years of teaching experience.

Students were recruited in each participating teachers’ 9th grade sections, a total of 25 sections throughout the 2016–2017 school year (13 were taught in the Fall, 4 were taught in a yearlong format, 8 were taught in the Spring). We estimate there were about 790 total students enrolled in the foundations courses; class sizes ranged from 19 to 35 students. We received research responses from 431 students overall (54.56%). Because the course is a state graduation requirement we expected the gender ratio to be roughly equal. Among students reporting gender, this was the case (53.05% female, 46.95% male; see Table 1).

Table 1 Demographics for research participating students
Full size table

Procedure

At the beginning of the course teachers informed students and parents of the study to improve robotics educational experiences and collected parent consent forms and student assent forms. As the introductory course proceeded, teachers delivered the standardized lessons—the traditional robotics lesson (control) to approximately half of their classes (n = 10) and the soft robotics lesson (treatment) to their remaining classes (n = 15). For each teacher, control and treatment lessons began at the same time and lasted about 10 class sessions (depending on external schedule conflicts that may have postponed any lessons). The soft robotics lesson was also designed to meet the same learning objectives as the traditional lesson including Next Generation Science Standards (NGSS) Engineering Design standards (NGSS Lead States 2013) and Standards for Technological Literacy #11 and #12 related to design (International Technology Education Association 2007). Both lesson types began with participating students taking a pre-lesson survey, aligned with Expectancy-Value outcomes and described in the next section. At the conclusion of the lesson students repeated the online survey.

Traditional robotics lesson The existing unit engaged students in the design of a robot made from traditional rigid materials including popsicle sticks, string, syringes, tubing, lumber, and dowels. Their objective was to create a crane-like device that can pick up blocks and stack them in a simple pick-and-place maneuver remotely controlled by a bank of four hydraulic syringes used to actuate the arm (see Fig. 2). The introduction to the lesson focuses on the importance of design documentation to communicate solutions and has supplemental content on hydraulics to support student design work. Then, students work in pairs to engage in the engineering design process while individually documenting their work in an engineering design journal. The design journals include typical elements such as Define the Problem, Brainstorm Possible Solutions, Research Ideas and Explore Possibilities (with citations), Specify Constraints and Identify Criteria, Consider Alternative Solutions, Select an Approach, Develop a Written Design Proposal, Sketch the Final Design, Make a Model/Prototype, Test and Evaluate, Refine/Improve, Create/Make Product, and Communicate Results. After designing their robot, the teams present their design journals as well as their solutions to the design challenge. Assessment is based on the engineering design notebook, performance of the robot, and final presentation.

Fig. 2
figure 2

Examples of traditional robotics gripper design (left) and soft robotics gripper design (right) based on the lesson materials

Full size image

Soft robotics lesson Engineering byDesign uses standards-based lessons which support the replacement of design content and activities while maintaining learning outcomes. The soft robotics procedures are congruent with the same standards, although situated in a different context aligned with the expectancy-value outcomes described previously. For the intervention lesson, the design objective was to use a two-part silicone rubber to fabricate a pneumatically actuated gripper which could pick up and sort a golf ball, which was symbolic of farm produce harvesting or sorting, without damage. The introduction of the lesson was based on the same design documentation instruction and had supplementary content on pneumatics principles (instead of hydraulics). The activity was completed in pairs while students individually documented their design work and students presented their final design process and solution (just like the existing lesson).

The lesson was also refined through pilot testing with middle- and high-school students. Since the design and fabrication of silicone actuators was unfamiliar to students, each teacher provided basic instructions on how to mix and cure the silicone using the provided materials (e.g., mixing cups, 3D-printed molds, and toaster oven incubators; Jackson et al. 2017a). Students began by assembling the reconfigurable, 3D-printed mold based on their design configuration—which might vary in configuration from other designs. After mixing and pouring the liquid silicone into the mold, it cures in about 4 h at room temperature or 15 min at 150°F. Students used more silicone to attach an inextensible layer made from cotton fabric, and then allowed the addition to cure. Finally, they tested the gripper through inflation with a squeeze bulb pump (see Fig. 2).

The fabrication approach, using a two-part silicone elastomer, was chosen because it was one of the most developed, having previously been used in a successful outreach effort with children (Finio et al. 2013). The soft robotics lesson materials were adapted in order to support redesign opportunities for students; we have reported on this iterative process in other work (Jackson et al. 2017b; Zhang et al. 2017). Using a developed 3D-printed mold, students were encouraged to explore design variation in their fabrication of the soft robot gripper. Design opportunities include variation in the number, spacing, and clustering of air chambers in the actuator, as well as the size of the actuator—each variable and their interaction has potential impacts on final performance of the gripper.

The initial attempts at making grippers showed that it is more complicated than it seems to be: the variety of design variables affect the functionality of the gripper and variation in the fabrication process can impact its actuation characteristics. In fact, inspection of a subset of student-designed grippers showed that 81% of failures were related to the fabrication process (Zhang et al. 2017). Consequently, the unit was created with three distinct design cycles. In the first two design experiences, students design, fabricate, test, and reflect on the performance of just one finger (rather than the entire gripper). Students are guided by the teacher in hypothesizing and testing which design and fabrication variables impact performance. Informed by a web search and their empirical results, student teams proceed to design, prototype, fabricate, and test a gripper, documenting their work in their design journals.

Measures and outcomes

The online survey, taken at the beginning and end of the lesson, used existing measures of student perceptions, as well as asked about demographic information. We reviewed the language of each scale to consider its use with high-school students and found each scale to be appropriate. We describe each scale and how it aligns with the Expectancy-Value framework undergirding this analysis.

Self-efficacy—expectancies for success We used engineering self-efficacy as a domain-specific measure for expectancies for success. “In empirical studies, the two components have shown very high intercorrelations, and competence and expectancy beliefs have typically been collapsed into a single construct or used interchangeably” (Trautwein et al. 2012). Engineering self-efficacy was measured by the Engineering Skills Self-Efficacy Scale (Mamaril et al. 2016). This contains 12 questions measured on a six-point scale of confidence for the activity from 1 (Completely uncertain) to 6 (Completely certain). Although the scale was developed for undergraduate engineering students, the questions theoretically relate to self-efficacy for engineering (1) experimentation, (2) tinkering, and (3) design, which are relevant outcomes for our intervention at the high school level. The scale structure and alignment with other motivational constructs was verified for use with high school students (Jackson 2018). In both undergraduate and high-school levels, all three self-efficacy constructs significantly correlated with intention to persist in engineering (each r > .11, p < .01) and were reliable (.79 < ωh < .93). The three skills are highly correlated and we used an average score on the pre- and post-survey to calculate students’ gain in engineering skills self-efficacy.

Situational motivation—subjective task value We used the Situational Intrinsic Motivation Scale (SIMS; Guay et al. 2000) to represent subjective task value for the material. The SIMS has 16 questions on four subscales aligned with forms of intrinsic and extrinsic motivation: intrinsic motivation, identified regulation, external regulation, and amotivation. Each question is measured on a seven-point scale according to whether the statement corresponded with the student’s reasons for participating in the activity, from 1 (Corresponds not at all) to 7 (Corresponds exactly). The SIMS has been used in prior design activity research (Stolk 2013). In five development and validation studies, the SIMS showed satisfactory reliability, all subscales had Chronbach’s α greater than .62, and prediction of behavior and task interest (Guay et al. 2000).

Even though the SIMS is based on a different theoretical framework than expectancy-value theory, the four types of subjective task value described in expectancy-value theory are conceptually represented by the SIMS subscales (see Fig. 3). First, intrinsic value is based on interest in the task; the Intrinsic Motivation subscale asks about similar motives for participating in the activity (e.g., “…I think this course is pleasant” or “I feel good when taking this course”). Attainment value is derived from alignment between the activity and personal identities; the Identified Regulation subscale uses similar reasons such as “doing it for my own good” and being “good for me” to justify engagement. Next, utility value stems from usefulness for future goals as do the External Regulation questions of the SIMS. Finally, relative cost in expectancy-value theory is represented by Amotivation on the SIMS. Both cost and amotivation are given a negative weight which corresponds to their negative impact on choice and behavior. Expectancy-Value research and Intrinsic Motivation instruments have also been paired in previous research (e.g., Berland and Steingut 2016).

Fig. 3
figure 3

Alignment between subjective value and SIMS instrument constructs with fictitious student motives

Full size image

Student gains in subjective value were based on a composite value, the self-determination index (SDI). This index incorporates all four SIMS constructs and weighs each scale according to its relative effect on choice (Vallerand 2001). Intrinsic Motivation and Identified Regulation are positive, self-imposed sources of motivation and weighted + 2 and + 1, respectively. External Regulation and Amotivation are less self-determined, due to external pressures, and weighted − 1 and − 2, respectively. “The total score reflects the person’s relative level of self-determined motivation” (Vallerand 2001), in other words, the relative value and importance of the activity. The total score ranges from − 18 to + 18 with a higher score representing increasingly perceived value from the task; value gain was the difference between the pre- and post-lesson SDI scores.

Career interest—intrinsic value Finally, we used the STEM Career Interest Survey (STEM-CIS; Kier et al. 2013) to represent the Intrinsic Value component of expectancy-value theory. The original development of the STEM-CIS sampled middle-school student perceptions of each STEM domain. For this research we used only the Engineering Scale of the instrument, which has 11 questions. Responses range from 1 (strongly disagree) to 5 (strongly agree). Calculating individual scores for each content area is an appropriate use of the instrument based on the factor structure (Kier et al. 2013). The scale developers found good internal consistency (α = .86), good model fit (χ2 = 50.27, df = 38, p = .91, NFI = .95, CFI = .99, RMSEA = .017), and factor loadings ranging from .33 to .78 with these questions. We calculated each student’s gain in interest value as the difference between their average pre- and post-lesson responses.

Analysis

To compare the efficacy of the hard and soft robotics lessons, we used multilevel modeling to predict gains in student outcome expectancies, task value, and interest value. Multilevel modeling takes into account the “nested” structure of students in classes and allows hypothesis testing of the varying impacts that classroom characteristics might have on individual outcomes (Raudenbush and Bryk 2002). These models are also called hierarchical models and have been used in evaluation studies similar to ours, where the new curriculum experience is given at the classroom level (Lachapelle et al. 2015). Model assumptions for multilevel modeling include normally distributed error at all model levels and constant variance of the residuals and across groups. Assumptions were found to be met through visual inspection of residual plots.

We considered the amount of variation in student outcomes that existed across classes, using the intra-class coefficient (ICC), before proceeding with the multilevel analysis. In multilevel models, the significance of predictors was determined using the Wald t test statistic and Satterthwaite approximation to take into account different class sizes (Satterthwaite 1946). In cases where between-class variation was negligible (a low ICC), we reverted to ANOVA models using the F-test statistic. Statistical analysis was conducted in R software (R Core Team 2017) using the lme4 package (Bates et al. 2015).

Following statistical analysis on the change scores for student perceptions, we used descriptive statistics to understand the changes in student perceptions after the lessons. These procedures complement the statistical analysis in several ways. First, in contrast to the statistical procedures which aggregate data to look for group differences and reduces findings to coefficients and probabilities, we could uncover insight by looking at the trajectory for each student. Next, it allowed us to adopt an exploratory approach to explain the previous statistical findings. Finally, it was a preliminary opportunity to conjoin the self-efficacy, motivation, and interest outcomes in a single analysis. Based on the descriptive, exploratory approach we did not consider statistical likelihood in follow-up analyses.

Results

Seven hundred fifty-two survey responses were collected from participating students, including pre- and post-lesson surveys from both types of robotics lessons. Responses which were missing two or more questions on any of the measurement instruments were removed from analysis (2.93%). Responses where we noted failure to legitimately engage with the survey, such as improbable completion times or no variation on reverse-scored items, were also removed (2.19%). Once pre- and post-lesson surveys were matched per student, gain scores were calculated from the composite scores for each of the three outcomes (described previously in the Measures and Outcomes section). Of the 424 students with completed surveys, 290 students (68.40%) had completed both attempts and were retained. Eleven students (3.79%) were identified as outliers because of the extremity of their standardized score (p < .001; Tabachnick and Fidell 2007) on either the self-efficacy gains, motivational gains, or interest gains and were removed from the data leaving a final 279 student scores, from 22 classes, to be used in statistical modeling. The average number of responses per class was 12.7 (SD = 6.03). The responses retained for multilevel modeling were similar in proportions of lesson type (control or treatment) and gender to the sample of research participants (in Table 1).

Means, standard deviation, and reliability indicators for each of the measurement instruments are reported in Table 2. Reliability was calculated using hierarchical omega (ωh; McDonald 1999) which is more appropriate than alpha reliability for multidimensional measures (Zinbarg et al. 2005). The high reliability of each set of questions supported our use of the scale composite scores for the remainder of the analysis (Raykov and Zinbarg 2011). Student gains in all three outcomes were not highly correlated (Table 3). The lack of correlation among outcomes may demonstrate that each measure represented a distinct perception of engineering expectancy or value. Alternatively, features of the robotics lessons may be differentially impactful for students’ perceptions. The proportion of variation between classes differed across the three outcomes we investigated (Table 3). Based on the low ICC for self-efficacy gains and interest gains, these two dependent variables were fit with two-way ANOVA models; motivation was fit with a multilevel model. Models for all three outcomes were fit with gender and lesson type (treatment vs. comparison group) and the gender-lesson type interaction as independent variables to address the research question of whether the new lessons are helpful for student perceptions, especially young women’s perceptions. These self-efficacy, motivation, and interest model results are presented sequentially in the next sections.

Table 2 Psychometric properties of pre- and post-lesson outcome measures
Full size table
Table 3 Descriptive statistics, intra-class correlation, and intercorrelations for outcome gains. (n = 279)
Full size table

Self-efficacy gains

We used a two-way ANOVA model to determine whether or not students’ self-efficacy increased differently as a result of the lesson type or student gender. Initial analysis demonstrated a non-significant interaction effect, F(1, 275) = 0.41, p = .52, which was removed and the model rerun. The final model included student gender and lesson type as predictors of self-efficacy gains following the lessons. None of the results were significant; in other words, self-efficacy gains are not different between young men and young women, F(1, 276) = 0.093, p = .76, or between lesson types, F(1, 276) = 2.376, p = .12.

Motivation gains

Motivation gains did show variability by class according to the ICC. Therefore, we used a multilevel model which specified the class for each student, as well as the predictor variables of interest to the research. There was not an interaction effect for gender and lesson-type, so the term was removed and the model rerun. None of the predictors in the final model were statistically significant in predicting motivation gains following the robotics lessons. The overall model estimate for motivation gains was roughly even, − 0.44, and the effects of gender and lesson-type were similarly small, − 0.85 and 0.69, respectively. Each of these coefficients is interpreted as a change in motivation difference, which had a wide initial range. Differences in motivation gain following the lessons are not related to student gender or the robotics lesson type, as indicated by the lack of significant predictors (Table 4).

Table 4 Multilevel model parameter estimates for effects of gender and lesson type on motivation gains
Full size table

Interest gains

As with self-efficacy gains, the interest gain scores did not exhibit variability across classes so the model was fit using two-way ANOVA. The interest gain results imitated the results from self-efficacy and motivation student perceptions. The potential interaction between student gender and lesson type was not significant, F(1, 275) = .13, p = .71, and was removed. In the final model main effects for student gender and lesson type were also not significant, F(1, 276) = 0.23, p = .63, and F(1, 276) = 2.42, p = .12, respectively. The interest gains following participation in the robotics lessons were the same for both genders and lesson types.

In summary, group differences for each outcome, by lesson type and gender, are shown in Fig. 4. The figure illustrates the similarity of outcomes of different groups. Given the overlap of the groups, the statistical procedures are unable to associate changes in outcomes to either the lesson type a student participated in or the students’ gender.

Fig. 4
figure 4

Group differences in self-efficacy, motivation, and interest by lesson type and gender. Due to differences in scale of the variables, each gain score was standardized; the y-axis is in standard deviation units and the points represent the average standardized score by group

Full size image

Exploratory analysis

With the statistical procedures yielding similar results by lesson type and gender, follow-up analysis was conducted to explore whether, and how, student scores were changing following the lessons. Descriptive statistics reported in Table 3 indicate that individual gain scores are variable even though the average change over time is negligible. Therefore, we used the standard deviation of each pre-test to code how student perceptions changed over time: each student was coded for whether they increased by at least a standard deviation, decreased by at least a standard deviation, or held the same perception (stayed within a standard deviation) after the lessons, on each outcome (receiving three codes in total). The proportion of students increasing or decreasing on each measure was then considered by lesson type and gender (Table 5).

Table 5 Count and proportion of students changing on self-efficacy, motivation, and interest outcomes
Full size table

Most students did not change perceptions (65.95%) following the robotics lessons. There were also no students whose self-efficacy, motivation, and interest increased or decreased congruently, again indicating that these are distinct perceptions. In fact, only seven students demonstrated changes on more than one outcome (4.67%). In most cases, students increasing in their perception of engineering following the lessons were met by a similar number of students decreasing their perceptions. Student self-efficacy perceptions were the most volatile, with 45 students (16.14%) reporting changes, and interest was the least changed (21 students, 7.53%). This suggests that a focus on self-efficacy supports in the curriculum might be necessary.

Discussion

Our interest in student perceptions following the robotics lessons was to identify potential supports for students’ engineering career choices, especially for young women. The soft robotics intervention is theoretically positioned to influence student perceptions. We hypothesized an improvement in students’ engineering perceptions, and a decrease in the expected gender gap. However, based on changes in three measures—self-efficacy, motivation, and interest—we did not observe significant differences between the soft robotics and traditional robotics experiences, or between young men and young women. Also, based on the low correlation scores among outcome gains and few students changing in more than one, we reason that students’ trajectory on each outcome is distinct. Potential changes in self-efficacy, motivation, and interest do not seem reciprocal.

Yet, a closer look at the distribution of change scores on these outcomes showed that student attitudes were malleable. This evidence is consistent with findings from Hernandez et al. (2014) which demonstrated changes in students’ STEM perceptions throughout high-school. In their research, overall changes in student perceptions were negligible, however a further analysis showed a nearly even number of students increasing and decreasing in STEM perception. Many students in our robotics lessons had changing perceptions, it may be a matter of identifying the underlying student or environmental characteristics that affect change. It is possible that student or environmental characteristics (for example, early elements included in Fig. 1) may explain changes in student scores, especially the direction of change. The need to identify underlying factors is more salient given the contradicting changes in students’ perceptions that we observed—for as many students as were growing in positive perceptions of engineering, a similar size group was becoming disenchanted. Of the three outcomes we investigated, more students’ self-efficacy perceptions (aligned with expectancy for success) changed following the robotics design experiences, than did students’ perceptions of motivation or interest. The hands-on nature of the robotics design may facilitate this effect, since it generates immediate feedback as students build and test their designs.

Given the newness of the soft robotics lesson materials to the instructors, the similar levels of performance on self-efficacy, motivation, and interest outcomes between the two lesson types suggests that improvements to this design context may uncover benefits in the future. Put in context of the timeline for of our soft robotics curriculum development, in the first year of implementation the new soft robotics lessons yielded results similar to traditional (widespread) robotics design lessons on the measures used. The multilevel modeling and ANOVA tests showed no negative changes in self-efficacy, motivation, or interest as a result of the experimental nature of the soft robotics lessons. Despite a lack of observable differences by lesson type, the curriculum, as designed, is aligned with expectancy-value theory and expected to support student perceptions; it is possible these differences were not detected, whether by the instruments used or in the limited timeframe of the robotics design experience.

Implications

These reported findings have impacted our research program moving forward. The incongruity of failing to observe changes using multilevel and ANOVA procedures, yet seeing a large proportion of individual students with shifting perceptions, has caused us to reflect on the assumptions and limitations of our analytical methods. While commonly used, ANOVA methods group participants and can mask individual differences, as evidenced here. Further research could seek to identify student characteristics which predict individual trajectories. Furthermore, the intention of our research—to change student perceptions of engineering—leads to potential risk of response-shift bias (Howard & Dailey 1979). Response-shift bias is caused when self-evaluation before and after an intervention are conducted from different frames of reference; in our case, by attempting to broaden students’ perceptions of engineering, initial reports may not take into account the understanding and experiences realized in the design experience. This risk is especially problematic for the soft robotic design experience since it exposes students to novel uses of silicone and applications within engineering, compared to the familiar, traditional robotics design experience. In addition to considering retrospective reporting to alleviate this bias, our future work will more heavily consider qualitative methods from student participants. Qualitative strategies afford a focus on the experiences and meaning-making of individuals, as well as a complex understanding of issues (Creswell and Poth 2017).

Beyond the difficulties of instrumentation just described, we observed several challenges which emerged as the planned curriculum was presented in the classroom. Implementation challenges are common for new instructional programs—the process of revising instructional materials based on data is built into “almost any instructional design model” (Dick et al. 2015)—and what is planned, even best practice, is often different than what occurs in the classroom (Perrow 2013). However, it is likely these challenges have impacted students’ and teachers’ perceptions of success in robotics design. Specifically related to the expectancy-value outcomes, several major revisions have been made for teaching and data collection in the second year of implementation. These changes may address the differential trajectories of student perceptions by improving the design experience. First, robot fabrication materials have been changed after discovering nitrile (used in gloves and rubber bands) prevented the silicone robots from curing. Changing these materials should support student success in the design experience by reducing the number of gripper failures caused by incompletely cured silicone. Second, emphasis on iteration through the design process has been reframed, which should also promote expectancy for success. Students’ beliefs in their ability to complete a task are most strongly influenced by their perspectives on prior experience (Britner and Pajares 2006; Usher and Pajares 2009). We have reframed the finger design and testing, previously considered prototyping, as a research process; in the research process failure may be more accepted, and less impactful for students’ expectancy of success (Lottero-Perdue and Parry 2015). And third, the design context has been adapted to emphasize societal benefits of the design solution, and subsequently subjective value. Previous evidence has suggested that the among students selecting engineering careers, the choice is at least partially based on perceived benefit to society (Benson et al. 2013). The human-oriented nature of careers seems especially for women, who tend to demonstrate greater interest in “people” activities (Su et al. 2009).

With these refinements to the soft robot design lessons, our research is ongoing. The teacher impacts of this research have included more than 8 h of face-to-face professional development and teaching observations annually, classroom resources for lesson implementation, and community building with researchers and teachers for curriculum support. Following the first year implementation we have continued to work with the participating teachers, as well as provided training for teachers in a second school district who will deliver the material to their classes in the 2017–2018 school year. For all teachers involved, the soft robotics lessons have provided exposure to a new context of engineering practice. The project will analyze new data from teachers in both districts in the academic year, which may produce similar conclusions or evidence of improvement in the engineering perceptions of participating students.

Conclusion

This research has led to the development of new robotics design teaching material—for soft robotics—and demonstrated its feasibility for classroom use. The lessons offer a new perspective on engineering and have performed as well as traditional robotics design experiences, based on student engineering perceptions. The curriculum materials are available for use by practitioners and researchers and are included in the Supplemental Materials of Zhang et al. (2017) or a Google Drive folder at http://www.tiny.cc/SRMolds. Based on teachers’ enthusiasm for this new design context, the soft robotics design learning experiences are being integrated into an upper level course called Advanced Technological Applications offered by the STEM-CTL, released in April of 2018. The identification of changing student perceptions of self-efficacy, motivation, and interest also contributes to the aim of supporting student career trajectories for engineering. While we do not yet have evidence of positively influencing high-school engineering perceptions, we reiterate, with evidence, that student attitudes are changing, and that this is an important time for practitioners and researchers to intervene to sustain student interest.