Introduction

As the U.S.A continues to compete in a global economy that demands innovation, US educational institutions give even more emphasis on STEM learning, which requires helping students and teachers build skills needed to succeed in a science and technology–driven world. In this undertaking, there is a major challenge: how to design and implement projects that engage youth, educators, and other community members in STEM-rich learning experiences? This is the question the authors of this study investigated in their recent IT/STEM project, which focused on using information technology (IT) within the context of STEM through technology-enhanced, inquiry- and design-based collaborative learning experiences.

The Industrial Midwest in general and Southeastern Michigan in particular are facing new and difficult challenges in recent years due to the advance of competitive global IT developments. This global competition has decimated the region’s traditional manufacturing–based workforce. Friedman (2006) argues that the current situation has led to a steep decline in local, state, and to some degree national economies. The National Governors Association and Council for Competitiveness (2007) highlights that one of the necessary steps to face this new challenge is to competitively transform the region from a “brute-force” to a “brain-force” through K-12 student-centered research projects, which focus on inquiry-based, real-world project–based IT/STEM (Information Technology in Science, Technology, Engineering, and Mathematics) initiatives with a strong emphasis on innovative 21st-century career and educational pathways. The National Science Board (NSB 2010) echoes the earlier highlights and further recommends that in STEM areas, students should have the opportunity to experience peer collaboration and interactions with practicing scientists, engineers, and other experts. Other reports point out that the global competitive demands facing the region and the nation as a whole can only be met by diversifying the current/future IT workforce while also encouraging underrepresented and underserved populations to pursue careers in IT- and STEM-intensive fields (Mendoza and Johnson 2000; National Research Council 2011; The American Competitiveness Initiative 2007).

Despite significant funding of IT/STEM youth programs over the last two decades, African American, Latino, and female students remain underrepresented in the related workforce, particularly in the field of computer sciences. In 2004, less than 9 % of computer science majors were African American, while only 4 % were Hispanic. For women, the lack of representation is even greater, with a 70 % decline of incoming female undergraduates pursuing computer science degrees in the US between 2000 and 2005 (National Center for Women and Information Technology, NCWIT 2007). The NCWIT (2009) also reports that women make up only 25 % of the IT workforce. Women’s representation also varies by race/ethnicity; 18 % white, 4 % Asian, 2 % African American, and 1.5 % Hispanic. The National Research Council (NRC 2011) highlights the importance of providing opportunities for students from underrepresented groups because of “changing immigration patterns, the rapid improvement of education and economies in developing countries, and a heavy focus on talent development” (p. 4). Efforts in K-12 to serve these groups will play a major role in addressing these crucial issues.

Programs nation-wide offer afterschool, Saturday, and summer programs to boost interest, engagement, and achievement in IT/STEM learning. In these programs, role models demonstrate skills and mentor young students. Students construct robots, rockets, video games, or molecular models, all to learn the science and mathematics of what interests them. Evaluation findings of these programs include important information about student interest, engagement, and motivation (Kane 2004). However, few of these evaluation data are effective in providing the education community with the generalizable knowledge needed to build better interventions (NSB 2010). Not many researchers study students over multiple years to determine their choice of IT/STEM majors in college and their career path. Too little is known about what students might gain from programs that help them, over a period of years, to pass through obstacles to successfully work toward an IT/STEM-based career. Much work remains to understand what students do to sustain interest and initiate their further learning over time, what their parents do to support their interests and learning, and what roles IT/STEM programs, mentors, family members, peers, access to technology, schools, and other resources play in their learning path.

Recommendations call for more research and a new paradigm. Long-term studies that follow students through high school and college and into their careers are needed (Subotnik et al. 2006). Tracking the experiences of students in IT/STEM programs during high school years and into careers is needed to substantiate claims that such programs increase the number of underrepresented youth in IT/STEM fields. In accordance with the existing literature, this present work is aimed at studying urban high school students’ IT/STEM learning in the Fostering Interest in Information Technology (FI3T) program. The following specific research questions were examined in the study:

  1. 1.

    What was the impact of the FI3T program on student participants’ IT/STEM technology skills?

  2. 2.

    What was the impact of the FI3T program on student participants’ frequency of IT/STEM technology use?

  3. 3.

    What was the impact of the FI3T program on student participants’ understanding of IT usage in STEM-oriented fields?

  4. 4.

    What was the impact of the FI3T program on student participants’ attitude changes toward IT/STEM?

  5. 5.

    What was the impact of the FI3T program on student participants’ desire for a career in IT/STEM-oriented fields?

Conceptual Framework and the FI3T Project

The FI3T project was designed to increase the opportunities for underrepresented and underserved high school students, particularly those from disadvantaged urban communities in Southeastern Michigan to learn, experience, and more importantly use IT within the context of STEM. The FI3T project called for the investment and robust participation of postsecondary colleges and schools, area school districts, and the business, industry, and government sectors. The “Community of Designers” approach introduced by Mishra et al. (2006) provided the framework for a collaborative partnership among a range of participants involved in the program.

Community of Designers

As Mishra et al. (2006) describe, the Community of Designers is an environment in which groups of individuals work collaboratively to design and develop solutions to authentic problems. The authors highlight the essence of this approach with four key words: community, design, products/solution, and authentic problems.

Mishra et al. (2006) describe “community” as the social arrangement of the approach. The authors explain that a purposefully constructed community should include individuals with a variety of expertise and expectations, allowing members to contribute to and benefit from community engagement. Referencing earlier studies (Cole 1996; Harel 1991; Harel and Papert 1991; Vygotsky 1978), Mishra et al. highlight that within the context of social constructivism, design projects provide an environment for sustained inquiry and collective creativity.

Mishra et al. (2006) explain that “design” specifies the activity dimension of the approach. The authors argue that building upon ideas grounded in situated cognition theory (Brown et al. 1989), learning is contextualized in the process of doing-solving an authentic problem of practice. Design-based activities provide the rich context for learning, sustained inquiry, and revision and are well suited to develop the deep understanding needed to apply knowledge in the complex real-world domains. The authors point out that emphasis on design is informed by research on the use of design for learning complex and interrelated ideas with many theoretical and pragmatic connections to project-based learning.

Mishra et al. (2006) argue that while “products/solution” stresses the goal-oriented psychological dimension, “authentic problems” addresses the motivational challenge, which becomes the driving force behind the work of the community. Authentic problems that project participants face and need to work on provide the connection between what they learn and what they actually do. Citing other research in this area (e.g., Barab and Duffy 2000), the authors point out that design team participants deal with authentic and engaging ill-structured problems that reflect the complexity of the real world. The authors highlight that learners have to actively engage in practices of design, inquiry, and research in collaborative groups to design tangible, meaningful artifacts as end products of the learning process. The authors further describe that the actual process-by-design is the anchor around which learning happens. This evolving artifact is also the test of the viability of individual and collective understandings as participants test their and others’ conceptions and ideas of the project. Mishra et al. (2006) point out that implementing a community of designers breaks down into four stages that each design team experiences over its lifecycle: identifying participants and problems, forming communities, providing leadership and support, and working on authentic problems. The authors argue that identifying potential participants is the key to the success of the community of designers approach. Normally, an open call for participation with description of the program and inviting interested members is critical in identifying the design community members. Mishra et al. point out that forming communities with experts and interested individuals constitutes another important component of implementing the community of designers approach. Depending on the situation, the potential audience of the design community should include varying stakeholders. One member of the design community is often needed to provide overall leadership and serve as a resource to all the design community members at any given time. Other general support should also be available often to serve as consultants to design communities. Once the design community is formed, members of the community begin to work on identifying authentic problems and exploring solutions to the problems over a period of time, during which they may encounter the boundaries and intersections of their expertise and interest. One would argue that in many ways implementing a community of designers framework parallels with the principals of the “community of practice” idea where Lavy and Wegner (1991) describe learning through social engagement in which members share understandings regarding what they are conducting and what that means in their daily life and for their community (cited in Parker et al. 2010). Further referencing Lavy and Wegner, the authors highlight that “these communities foster mutual engagement among the members, while they work on a joint enterprise using shared repertories of terminology and skills” (p. 190).

The FI3T Project

Consistent with the aforementioned discussions, the FI3T project called for the collaborative engagement of high school students, high school STEM teachers, undergraduate/graduate student assistants (U/GSAs), and STEM content experts from university, business, industry, and government sectors to create high-quality learning projects, strategies, and curriculum models for use in afterschool, weekend, and summer settings through hands-on, inquiry-based activities with a strong emphasis on non-traditional approaches to learning and understanding. The project also distributed online learning activities using the project’s Web site and social media sites, and thus aimed at establishing a culture of collaboration and discourse to extend participation outside the confines of the formal scheduled events.

IT/STEM Concentration

The literature mostly references the National Science Foundation’s definition for what constitutes a STEM field, using a broader category to define STEM subjects which includes subjects in the fields of Chemistry, Computer and Information Technology Science, Engineering, Geosciences, Life Sciences, Mathematical Sciences, Physics and Astronomy, Psychology, Social Sciences, and STEM Education and Learning Research (STEM fields n.d.). However, the literature provides little definition of what constitutes IT/STEM. In the context of the present study, IT/STEM is defined as the use of a particular technology tools set in the areas of science, engineering, technology, or mathematics. The FI3T project concentrated on all four areas of STEM, creating four project-based design teams to address IT use in science, engineering, technology, and mathematics. The following section describes the particular focus of each IT/STEM design team.

The science team concentrated on three different but related applications of IT in the sciences: measurement, modeling, and mapping. Participating students’ learning experiences for IT/Science included making location measurements using GPS and integrating the measurements in a GIS system, using temperature and light sensors in the sciences, and creating mathematically based models using the isee Systems’ computer application STELLA that incorporates measured quantities and makes predictions.

The technology team focused on technological tools and languages for designing and developing Web applications such as Web-based games and chat-rooms. Participants gained experiences with the basics of visual programming, familiarized themselves with integrated development environments such as Visual Studio and/or Alice, and practiced designing and developing games.

The engineering team emphasized the basics of robotics and its applications as related to IT, including modeling robots, programming robots, and integrating robots into an application environment such as an industrial manufacturing system or a medical application in a surgery operating room. Learning experiences involved using the robotics simulation software package ROBCAD to learn the basics of robotics technology and how to construct robots. Participants learned how to program robots and how to design robot workcells and implement sensors to guide a robot’s motions and handling of articles within robotics workcells.

The mathematics team focused on statistical science with consideration of the two-sample comparison problem, the simple regression/correlation problem, and the simple analysis of covariance problems taking examples and assignments from public health science, environmental science, and manufacturing reliability. Participants used Minitab software to create comparative displays and regression displays and performed appropriate analyses to test for and estimate effect sizes.

Partnership

The FI3T project approached the IT/STEM-learning issue as a community-wide responsibility. Therefore, the project called for the investment and robust participation of higher education institutions, K-12 schools, and business, industry, and government sectors as well as parents and volunteers. Aligned with this notion, participating partners of the FI3T project included (a) University of Michigan-Dearborn’s College of Engineering and Computer Science, the College of Arts, Sciences, and Letters, and the School of Education; (b) Detroit Public Schools; (c) the Survivability Technology Area of the US Army’s Tank Automotive Research Development and Engineering Center (TARDEC), Dassault Systèmes’/DELMIA Corporation, Reactor Zero (a local game developer company), FANUC Robotics, Inc, the Society of Manufacturing Engineers (SME), SIEMENS, the Barbara Ann KARMANOS Cancer Institute, the Systems Analytics and Environmental Science Department at Ford Motor Company, the Advanced & Manufacturing Engineering Quality Department at Ford Motor Company, and The 21st Century Digital Learning Environments, and (d) parents and volunteers. These partnerships provided project participants with opportunities and support to work directly with IT and STEM professionals and observe examples of real-world workplace applications.

Participants

Each design team targeted to include 10 high school students, one STEM area high school teacher from the participating school district, one U/GSA and one postsecondary STEM content expert from participating higher education colleges and schools. Each design team, therefore, targeted 13 collaborating members. STEM teachers team taught and shared the planning and instruction of all project activities with postsecondary STEM content faculty. Each design team included the participation of a U/GSA as an important member of the support structure. One specialized member of the project leadership team (in STEM areas) led each design team.

Coming from four comprehensive high schools of the partnering school district, participating students were selected from underserved populations, students with special needs, and female students through a multidimensional screening process. Participating teachers were selected from STEM area–related certified high school teachers of participating high schools with a minimum of 5 years of teaching experience based on the nominations and recommendations from district curriculum supervisors. Participating STEM content faculty were selected from the schools and colleges of the participating higher education institution. The project leadership team identified faculty members whose interest, teaching, and research were in STEM areas and invited them to participate in the project. Similarly, U/GSAs were selected from the university based on their STEM discipline. Each design team leader identified a collaborating business, industry, government, or university sector in their particular area of interest. These partnerships provided project participants with opportunities and support to work directly with IT and STEM professionals and to see examples of real-world workplace applications.

The FI3T project considered parental and volunteer involvement as a critical factor for the project’s success and continued student engagement and retention. Parental involvement was fostered through scheduled seminars and outreach opportunities in order to promote their investment and robust participation in the project’s events and activities. Volunteers from the university’s students and staff, community members from participating schools, industry, and government sectors, and students’ family members were invited to strengthen the project’s support structure.

Project Events and Activities

Within two different but interrelated phases, the work of FI3T was accomplished over an eighteen-month cycle through summer and school-year activities. Phase 1, the first nine months of the program, was primarily a time for among students to increase knowledge and skills in IT/STEM-related fields. Phase 2 focused on facilitating student activities in which they engaged in designing inquiry-based authentic projects of science fair quality using what was learned in the capacity-building phase of the program. The following sections describe project events and activities in each phase.

Phase 1: Capacity Building. Phase 1 started with a kickoff meeting as the school year began for participating sophomores (10th grade), followed by a set of IT-intensive STEM area workshops for students during the school year, and seminar meetings near the end of the fall and winter semesters. Capacity-Building activities took place on the campus of the participating university. The total number of instructional hours per student during this phase was 54.

Kickoff Meeting. This whole-group orientation meeting was designed to motivate participants by informing them that their work within the project was part of a larger national initiative. Second, the meeting was utilized to explain the roles and responsibilities of all engaged members in detail. Community members from collaborating schools, institutions,and organizations, and parents and volunteers attended the meeting.

Level 1 Workshops. During the fall semester, each STEM area offered two Level 1 workshops to all participating students. These workshops consisted of brief presentations followed by hands-on activities to provide students the opportunity to learn about IT toolsets within the context of STEM. A second purpose of the Level 1 workshops was to allow participating students to identify specific areas of interest within STEM fields. During this period, STEM area teachers, faculty members, and the project leadership team observed and surveyed students for their interest in specific STEM subject areas and assisted them to narrow down their interest into two specific STEM-related fields.

Seminar Meeting 1. A whole-group seminar meeting was held near the end of the fall semester. At the meeting, students, teachers, content experts, project leaders, and parents collectively focused on finalizing decisions for two specific areas of interest in STEM fields and planned the upcoming activities during the following winter semester.

Level 2 Workshops. During the winter semester, a set of small-group in-depth Level 2 IT/STEM workshops were offered to the students in their identified two STEM areas of interest. Each STEM area offered four three-hour content-specific IT workshops, allowing each student to participate in a total of eight workshops related to their identified two STEM areas. Throughout these workshops, participating students had the opportunity to learn advanced use of IT toolsets within specific STEM areas. Level 2 workshops also allowed students to narrow down their interest into one specific STEM area and help them to join in one specific IT/STEM design team.

Seminar Meeting 2. Similar to Seminar Meeting 1, a whole-group seminar meeting was conducted in late spring of the academic year to form four design teams (one for each IT/STEM area) based on the interest of participating students. The meeting was also used to plan for externship activities involving real-world field-based experiences during the following summer.

Phase 2: Design Year. The design phase started with a summer externship followed by a series of site-based sessions for each individual design team and a whole-group seminar meeting near the end of the following school year. The design year ended with a techno/career fair.

During this phase of the project, participating STEM teachers continued to collaborate with higher-education faculty, undergraduate/graduate students, and business partners to facilitate IT-supported STEM project activities for high school students assigned to their IT/STEM design team. The overarching task of each design team in this year was to develop inquiry-based authentic projects that were of at least science fair quality using one or more content-specific IT tools explored during the previous capacity-building year and stimulating ideas/experiences gained during the summer camp. The design phase ended with a techno/career fair meeting during the following spring term. During this phase, students were expected to spend approximately four hours in each week on their projects during the anticipated 30 weeks of the school year, bringing the estimated total number of contact hours to 150.

Summer Externship. The summer externship consisted of field-based experiences and preparation for design activities. At the two-week summer program (1 week in mid-June and another one in late August), project participants met and observed the work of scientists and professionals in IT/STEM fields. Collaborating business, industry, government, and university sectors hosted these sessions. The project facilitated eight different day-long field trips (two for each STEM design team) each emphasizing IT-related career and educational pathways within the context of STEM, and including debriefing activities after each one. The summer program was also aimed at readying the students for the project development stage that occurred during the subsequent collaborating school year.

Site-Based Sessions. Aligned with the cyclic inquiry model’s 5 major steps (Bruce 2003)–Ask, Investigate, Create, Discuss, and Reflect, the design year involved five segments, each including multiple site-based sessions. As part of the summer program discussed above, the project facilitated collaborative learning experiences where students learned how to design and conduct inquiry-based authentic projects, more specifically learned how to Ask, how to Investigate, how to Create, how to Discuss, and how to Reflect. These theoretical discussions then were linked to students’ authentic projects to provide practical applications. The focuses and approaches of design steps were as follows.

Step 1: Ask. Step 1 took place during the summer program. Lead by one specialized member of the project leadership team, each STEM area held a series of meetings to discuss IT-intensive authentic project ideas aligned with appropriate federal and state standards within the focus area of each design team. At this stage, each student began to focus on a question or problem, defining and describing it. Students were assisted in the process by the design team members. During this process, the design team leadership closely surveyed the focus and interest of each participating student to facilitate individual, small-group projects. Even though initial questions were redefined throughout the learning process, they naturally lead to the next stage in the process: investigation.

Step 2: Investigation. Step 2 also took place during the summer program and conducted in a similar manner as Step 1. Students began to collect information about their questions. This process included research using reading, observing, interviewing, or doing exploratory experiments. Even though the design team leadership assisted students in the investigations, it was important that students had ownership in the process.

Starting with the new academic year following the summer program, students engaged in the following three stages of their project design (Create, Discuss, and Reflect) iteratively. Students spent an average of four hours weekly on their projects until the early spring of the year. Each design team facilitated ongoing meetings and discussions during this process using the scheduled meetings and project’s social networking sites. Design team interaction was facilitated on an ongoing basis to assess project progress.

Step 3: Create. As students made numerous connections between the results of their investigations during this stage, they began the creative task of going beyond their previous experiences to create new ideas of how to answer their questions or solve their problems. Again, the design team members assisted the students during this process.

Step 4: Discuss. Students shared their investigations and new ideas with others. By comparing notes and discussing their results, they built a design community which increased the relevance of their projects.

Step 5: Reflect. Reflection consisted of taking the time to look back at the question, the inquiry research path, and the conclusions made. During this process, students decided whether a solution had been found and saw what further questions had emerged. Thus, the circle of inquiry began anew.

Seminar Meeting 3. A 3-h whole-group seminar meeting took place near the end of the school year where design teams rejoined to share their experiences related to the design activities. Project participants discussed strategies for disseminating their projects to peers across their district and throughout the region. The meeting also provided time for students to work on their project presentations at the following techno/career fair.

Techno/Career Fair. At this half-day-long exposition, participants showcased their projects and discussed their experiences with the community. The fair also served participating colleges and schools and the region’s business, industry, government, and university sectors to promote admission and career services related to IT/STEM fields. Community members from collaborating institutions and parents were invited to the fair.

Methods

Research Design

A mixed methods design was used in this study. As Gay et al. (2006) describe, mixed method research combines both quantitative and qualitative data collection and analysis in a single study. The use of descriptive, comparative, and interpretive components in this study required a combination of quantitative and qualitative research methods to appropriately answer the research questions. The first step of the study involved the dissemination of the pretest survey and questionnaire prior to the beginning of the program. The second step involved an ongoing collection of program participation data and artifacts from the project activities including content analysis of the program participants’ IT/STEM projects. The third step involved administering the posttest survey and questionnaire. The final step of the study included follow-up interviews with the study participants.

Participants and Setting

The school district involved in this study was a major urban school district located in Southeastern Michigan. It is the largest school district in the State serving nearly 66,000 students (mostly African American) throughout a major city area (Dawsey 2011). The district has a long history of challenging issues including shrinking population of students, financial instability, school closings, and a low high school graduation rate. When the present study was launched in 2008, it was reported that the high school graduation rate was 24.9 % in the district, worst in the nation among the largest school districts in that particular year (Mrozowski 2008).

Over a three-year period within two cohort groups (September 2008 to June 2011), the subjects for this study included 77 10–12th-grade high school students from four different schools. Each cohort participated in the study for two consecutive years with an overlap in 2009–2010. After the end of the major project activities, all students were invited to participate in follow-up interviews during the 2011–2012 academic year, and a total of 13 project graduates were interviewed, three from Cohort 1 and seven from Cohort 2.

The study was announced to all 10- and 11th-grade students in participating high schools at the beginning of the 2008–2009 school year (Cohort 1) and again at the start of 2009–2010 (Cohort 2), and volunteers were asked to participate. Four specific high schools were targeted because earlier in the process, the program had identified participating teachers from these schools creating relatively easy access to the student population. Based on STEM area teacher recommendations, the first cohort included 40 and the second cohort 37 students. Of those total 77 students, 40 (52 %) students completed all project activities including the cumulating final IT/STEM project; an additional 21 (27 %) students participated in all major project activities but did not complete the final project; and 16 (21 %) students left the program during the first year of their participation, mostly indicating personal reasons.

Instrumentation

A student technology-use survey (see “Appendix 1”) and a student questionnaire about STEM (see “Appendix 2”) were used in the study. The survey instrument consisted of two parts. The first part contained three sets of items that were measured on four- or five-point ordinal scales. The first set, which had 23 items, asked about the students’ self-perceived skill level in using various technologies including IT/STEM tools; the second set had 17 items asking about the frequency of technology usage; and the third set had three questions anticipating the respondent’s career choices with an additional open-ended question for career aspirations in the future. A total of eight other open-ended questions were also included in this first section asking questions about how students use technology at home and in schools. The second part of the technology survey included a hypothetical question in which students’ application of technology skills were tested.

The student questionnaire used in the study also had two parts. In the first part, the study participants were asked to respond to five open-ended questions about their understanding of IT use in STEM fields. The second part included 16 questions that were measured on five-point ordinal scales in order to ascertain students’ attitude toward IT/STEM. The external evaluation team reported successful and satisfactory use of both instruments for a number of years in different afterschool STEM-oriented programs confirming its reliability. Additionally, a panel of experts in IT/STEM areas reviewed and revised the instruments for content validity.

The researchers developed and used a guideline for participants’ IT/STEM projects.

The researchers also developed a protocol for the analysis of open-ended questions included in the student survey and the questionnaire. An additional protocol was developed for the follow-up interviews.

Data Collection and Analysis

The researchers collected various forms of data relating to each of the research questions by applying a combination of quantitative and qualitative methodologies. The quantitative data were collected from the pre- and post-surveys and the questionnaires. The qualitative data were collected from the analysis of the participants’ IT/STEM projects and follow-up interviews. The project final evaluation report prepared by the external evaluation team was also used, which included additional data collection and analysis. The research questions drove the data analysis.

Of those 77 participants, 43 answered the study survey both times–at the beginning of the program and at the end (56 %); 26 were from Cohort I and 17 from Cohort II. Similarly, 42 answered the study questionnaire both times; 26 were from Cohort I and 16 from Cohort II. Table 1 below presents the number of student participants completing the pre- and post-survey. Number of pre- and post-questionnaire responders by cohort and design team affiliation showed the same configuration except that one fewer student responded in the engineering group for Cohort 2.

Table 1 Number of pre- and post-survey responders by cohort and design team
Full size table

All students who completed the study instruments both times were included in the combined cohort analysis. The overall analysis also included three participants for whom pre/post-data are available but their teams were not identified. Assuming equal intervals between the values of the ordinal scales, repeated measures statistical models were fit to these paired data using SAS’ PROC MIXED, including a factor for the Cohort (I or II) and its interaction with the pre/post-repeated factor. A five percent Type I error rate was used to test for differences between the pre-program and the post-program responses. The quantitative survey data was collapsed across design teams and had too few in each subgroup to be analyzed by subgroups.

To analyze student responses to open-ended questions included in the student survey and the questionnaire, the researchers used an “emergent” design, analyzing and categorizing responses as they moved down the list of responses to a particular question then consolidating even more as needed until they had a set of categories of responses. In assessing an open-ended question from a pre/post-perspective, a 0–3 point scoring rubric was used to assess responses, defining the 0 and 3 point scores and making a qualitative judgment if the response was neither 0 nor 3, but in between. A score of 0 was given if there was no response, a response not related to the question, or nonsensical response. A score of 3 was given for responses that adequately addressed the core ideas of the question, including, but not limited to, definitions, reasons, and examples. If a response did not fit the 0 or 3 point score, a judgment was made as to whether it qualified as a 1 or 2 point score (taking into account what and how much was missing from the response).

The researchers organized and sorted the data collected through the follow-up interviews, following three repeating steps (reading/memoing, describing, and classifying) for analyzing the data (Gay et al., 2006) to identify/verify some of the study findings that emerged from the study survey and the questionnaire.

Findings

Gaining Skills

Participants in each cohort were asked to respond to a 23-item survey rating themselves on their skills in using specific technologies on a 0–4 point scale with 0 = “I do not use” and 4 = “Expert (I can teach others to use the tool).” As shown in Table 2, of the 23 items on the survey, 20 showed a statistically significant (α = 0.05) positive change from pre to post for combined cohorts. There were no statistically significant differences between the two cohorts on any item.

Table 2 Student technology skills
Full size table

The first 14 items on the survey (a–n) were related to common technology skills. Of those 14 items, 11 showed significant gains from the beginning to the end of the program. The three for which no increase was detected were: (c) Digital still camera, (f) iPod or other handheld device, and (g) Email. Even in the absence of an increase, on the average, the participants felt themselves to be independent users of these three technology skills with averages of 2.2 or higher on the scale of 0–3. The last 9 items on the survey (o–w) were related to advanced IT/STEM technology skills. All items in this category showed significant gains from the beginning to the end of the program.

Frequency of Use

Participants in each cohort were asked to respond to a 17-item survey rating how they use technology and the frequency of use. The survey included a 4-point scale with 1 = “I do not do,” 2 = “Sometimes I do,” 3 = “I do this often,” and 0 = “I don’t know about this.” As Table 3 illustrates, of those 17 items on the survey, 7 (items a, b, e, j, l, n, and p) showed a statistically significant (α = 0.05) positive change from pre- to posttest for combined cohorts. There were no statistically significant differences between the two cohorts on any item. The largest increase was for (p) Program robots using computer software which increased from 0.9 to 1.9.

Table 3 Technology applications and frequency of use
Full size table

The first 9 items on the survey (a–i) were related to students’ frequency of common technology use. Of those 9 items, 3 showed significant gains from the beginning to the end of the program (a, b, e). For the remaining 6 items, no significant change was observed over the course of the program, whereas the posttest mean score for each of these items was larger than the pretest mean score. Also, the pretest mean score for these items ranged from 1.6 to 2.6 indicating that participating students were already using such common technology applications when they entered the program.

The last 8 items on the survey (j–q) were related to students’ frequency of advanced IT/STEM technology use. Of those 8 items, 4 items showed significant gains from the beginning to the end of the program (j, l, n, p). Findings indicate that two main areas of impact regarding frequency of use included science- and engineering-related IT/STEM toolsets.

Data indicates that the program did not significantly impact students’ frequency of use for the remaining 4 items in the IT/STEM toolsets category (k, m, o, q) even though the posttest mean score was larger than the pretest mean score for each of these items. Of those four items, one was related to game design and the remaining three were related to use of IT in mathematical applications. This finding might relate to the fact that after the Capacity-Building activities (Year 1), a less than adequate number of Cohort 1 students was interested in forming a Math design team for the Design year activities (Year 2). Therefore, the design teams for Cohort 1 did not include a Math team. Similarly, not many students from Cohort 2 were interested in being part of a Technology design team after the first year activities ended, not forming a Technology team this time. The actual design of an interactive computer game (item o) was a Design year activity rather than a first year activity, and a relatively smaller number of students completed such a specific project during the life of the program. Similarly, more intensive work on data analysis and display (items k, m, q) was done during the second year with smaller groups of Math design team students. It is possible that such low enrollment in Technology and Math teams lowered the numerical values of the average score in the related survey items.

Participant growth in use of advanced-technology toolsets was also evident by the vast array of IT/STEM technology projects they created during the life of the program. A total of 20 IT/STEM projects were developed by participating student teams. Of those, six project teams from Cohort 1 participated in the 53rd Annual Science and Engineering Fair of Metro Detroit 2009–2010. Two teams received an Outstanding Award and the remaining four teams received Excellent Awards with their projects. A total of 4 projects from Cohort 2 participated in the 54th Annual Science and Engineering Fair of Metro Detroit 2010–2011. One Mathematics project received the 3rd place award, and another Math project received the Excellent Award. Two projects, one from Science and one from Engineering received Outstanding Awards.

Understanding IT/STEM

Participants in each cohort were asked to respond to five open-ended questions about their understanding of IT/STEM (shown in the Table 4 below) at the beginning and again at the end of their involvement in the project. Of the 5 questions, all showed a statistically significant (α = 0.05) positive change from pre to post for combined cohorts.

Table 4 Understanding of IT and its use in STEM Fields
Full size table

Data show that the change between pre and post does not appear to depend on the cohort with the exception of the first item—what is information technology? The cohorts differed in their pattern of responses to this one item. Figure 1 illustrates the difference in the responses of the two cohorts. Cohort I has the diamond-shaped symbol, Cohort II the square. When the four other items are pictured, the corresponding lines are basically parallel with the Cohort II line above Cohort I. In fact, when the cohort scores are averaged over time (pre and post) as shown in Table 4 above, Cohort II has higher averages than Cohort I for three of the items.

Fig. 1
figure 1

Questionnaire item 1—what is information technology?

Full size image

Attitude toward IT/STEM

Student participants in both cohorts were asked to answer a set of questions in order to ascertain their attitude toward IT/STEM. Table 5 below shows the results gathered at the beginning of the program and again at the end regarding participants’ attitude toward IT/STEM. No statistically significant differences were found in the amount of change in understanding between the two cohorts (α = 0.05).

Table 5 Attitude toward IT/STEM
Full size table

Statistically significant change in agreement registered for three of the statements (1, 5, and 15). Initial agreement with two statements was relatively high and dropped slightly by the end of the program: Item 5—When I work with other students on a science activity or math problem, we help each other figure out how to answer the questions; Item 15–I think mathematics is useful in our lives outside of school. The third item for which change in understanding is evident was the first item (Science is a lot of facts and procedures that I have to memorize). With an average of 0.6, agreement at the start of the program was close to 1.0, and dropped near to zero (0.2) but still significant positive change at the end.

Significant change in agreement was not registered for the remaining 12 statements tested in the program. Students presented relatively high degree of agreement with five technology-related statements (item 8, 9, 10, 11, and 16) at the beginning of the program, as mean scores indicate, leaving a little room for change. Significant change in agreement was not registered for four science-related items (item 3, 6, 12, and 14). In item 12—My science teacher often asks us to explain our thinking about our answers to science questions, there was a considerable yet not quite significant decrease in the level of agreement with the statement from Start to End.

Further analysis of the questionnaire items related to mathematics indicate that in item 2, there was a substantial, though not quite significant, decrease in the level of agreement with the statement from Start to End. Nonetheless, the level of agreement was still quite high (Mean 0.7), suggesting that students, on average, tend to think of mathematics as a subject requiring much memorization. At the high school level of mathematics study, this opinion may reflect reality. There is also a substantial, though not significant, decrease in the level of agreement with the statement in item 13. One could speculate that there is a connection between the results on item 2 and 13: The decrease in item 2 could indicate that students, on average, between Start and End were more open to the idea that mathematics is not solely an assortment of facts for memorization but a creative activity that requires careful thought and step by step justification. Therefore, between Start and End, the students may have realized, on average, that their teachers were not consistently asking for explanations of the mathematical thought processes (item 13). There was, at the Start, already a high level of agreement (Mean 1.0) with the statement in item 4 and only a small upward change registered and possible, given the high level of agreement at Start. There was no change in item 7, but a fairly low level of agreement for this type of question (Mean 0.7), possibly indicating that students have based their responses on their experiences with estimation, probability, and statistics.

The follow-up interview data seems to differ from the attitude survey findings presented above. Of those 13 students who participated in the follow-up interviews, all but one student highlighted that their participating in the program positively impacted or sustained their attitude toward IT/STEM. One participant stated that the program “increased awareness of math and technology more to learn than previously was aware of.” Similarly, another student stated “increased interest in both math and science.” One particular student said “the program reduced fear of math at higher level of education.” In sum, findings of the FI3T program related to changes in attitude toward IT/STEM indicate mixed results warranting further investigation in this area of the present study.

Career Aspirations

Participants in each cohort were asked to respond to a 3-item survey stating their desire for a career in an IT/STEM-oriented field on a 5-point scale with -2 = “Strongly disagree,” −1 = “Disagree,” 0 = “Uncertain,” 1 = “Agree,” and 2 = “Strongly agree.” In general, mild agreement was shown with regard to desiring a career in mathematics or science, and fairly strong agreement for a career that uses a lot of technology. For the combined cohorts, no change in agreement with the career-related statements occurred between the beginning and the end of the program (See Table 6). However, change in student agreement in desiring a career in mathematics differed by cohort. Table 7 shows that students in Cohort I started out uncertain about wanting a career in mathematics and became somewhat more negative toward it by the end of the program, while those in Cohort II started out positive and remained positive at the end. This outcome could be attributed to the fact that there was no Math design team for Cohort I, but there was one for Cohort II.

Table 6 Desire for a STEM-oriented field
Full size table
Table 7 Average response to “I would like a job or career in which I use a lot of mathematics.” by Cohort (−2 = SD, −1 = D, 0 = Uncertain, 1 = A, 2 = SA)
Full size table

Participants in each cohort were also asked to respond to an open-ended survey item stating what job or career they would like to have when they reach 30 years of age (Student technology survey, item 13). For the combined cohorts, a total of 37 students answered this particular question at the beginning and the end of their participation in the program. Data indicates the following results in student responses: 12 stated interest in the same IT/STEM-oriented career at the beginning and then again at the end of the program; 9 stated positive changes in their interest in an IT/STEM-related career after their participation; 5 became less interested in IT/STEM fields as their future job after their participation in the program; 8 presented career interest in other than an IT/STEM-related field at the beginning, and participation in the program did not change their decisions at the end; and 3 were interested in a non-IT/STEM career at the beginning of the program and stated changes in their interest to some other non-IT/STEM-related job at the end of the program.

Follow-up interview data also presented mixed results about the impact of the program on career choices. Of those 13 students interviewed, 8 indicated that the program sustained or strengthened their interest in pursuing a career in IT/STEM field. The remaining 5 thought differently, indicating limited impact of the FI3T program on their career decisions.

Discussion

Key Findings

The findings of the present study suggest that IT/STEM experiences facilitated in the FI3T program significantly impacted urban high school students’ common technology skills such as using computers, internet, productivity tools, and Web 2.0 tools. Perhaps more importantly, the program created significant positive impact on students’ IT/STEM technology skills (e.g., GPS, GIS, robotics programming). In most cases, the FI3T program also improved urban high school students’ frequency of common and advanced IT/STEM technology use when those technologies are available to them. Specific to the learning experiences gained in the FI3T program, two main areas of impact regarding frequency of use included science- and engineering-related IT/STEM toolsets. As the finding of this study suggests and other studies reveal, when low-income urban students are exposed to well-designed inquiry-based materials that draw upon IT skills in sophisticated ways, youth not only learn science better, they become IT-fluent (Edelson 2001; Songer et al. 2002).

Findings of the FI3T program suggest that the IT/STEM-learning experiences facilitated throughout the FI3T program also brought significant positive change to urban high school students’ understanding of what IT is and how STEM scientists and experts use IT in their innovative practices. For example, significant gains were recorded for better understanding of what scientists, engineers, or mathematicians do and how they use IT as they learn about and develop new concepts.

In general, findings of the FI3T program related to changes in attitude toward IT/STEM indicate mixed results requiring further investigation in this area of the present study. Findings suggest that the program increased awareness of math and technology. Similarly, increased interest in math and science was observed. For instance, the program helped participating students to realize that science is not a lot of facts and procedures that they have to memorize or that mathematics is useful in their lives outside of school. It appears that the impact in attitude changes is also related to a positive attitude students bring to afterschool programs. Given the fact that participants volunteered to join such afterschool programs, initial high interest and positive attitude are understandable and leave little room for change.

Findings related to the impact of the FI3T program on students’ desire for a career in an IT/STEM-oriented field indicate a mild agreement among student participants. In general, findings suggest that study participants have limited aspiration for a career in mathematics or science, but fairly strong aspiration for a career that uses a lot of technology. Answering a specific question related to future career aspiration, over 55 % of respondents indicated increased or sustained interest in IT/STEM-oriented fields at the end of their participation in the program. Over 13 % became less interested in such a future profession. The remaining 32 % were interested in other than IT/STEM careers at the beginning of the program and stated sustained or changed interest in some other non-IT/STEM-related jobs at the end of the program. One would argue that impact on over 55 % of study participants is a considerable effect. However, there seems a need to explain why such an extensive program did not influence the other half of the participants’ career aspiration toward an IT/STEM field. It might be the case that learning experiences provided within the FI3T program were insufficient to significantly impact high school students’ career aspirations, requiring additional interventions. In their future research, the authors of this present study are planning on extending FI3T project activities with an additional “internship” phase, Phase 3, (in addition to Capacity-Building and Design activities) in order to investigate the extent of such additional learning experiences on high school students’ desire for a career in IT/STEM fields. Or it might be the case that there are other external factors that impact high school students career choices. In a recent study, Scott (2012) highlights that students who attend STEM-focused high schools outperform their peers at similar institutions on standardized tests. However, the author also highlights the need for further investigation to track their progress to determine whether students who complete these programs chose to pursue STEM degrees at the post-secondary education. Current literature highlights some other dynamics why high school students are not attracted to STEM fields. Taking computer science and high school girls as an example, Pollock et al. (2004) point out misconceptions about the field, misconceptions about working styles of people successful in the field, lack of access to desirable role models, lack of interest in the field among their peers, and lack of confidence in the abilities perceived necessary for success in computer science. The overall findings of the FI3T program and the current literature suggest that further research is necessary to better understand the impact of STEM programs on career aspirations of high school students.

Limitation of the Study and Recommendations for Future Research

The present study has some limitations that need to be taken into account when considering this research and its contributions. The urban high school students that participated in this study are volunteers who were selected based on their initial interest in IT/STEM. These participants may not be representative of all students in the district where this study was conducted. In addition, the study findings reported in this research are based on students’ self-perceived skills, frequency of use, understanding, attitude, or career aspirations. As Marsden and Torgerson (2012) argue, the pretest/posttest design used in this study has limitations to directly address the possibility that regression, maturation, test effects or other possible confounders could account for the study findings. Therefore, the findings of this study should not be generalized beyond this group of students who participated in the study. Without an experimental study with a control group, the findings of the FI3T project should be considered as preliminary.

While the results are preliminary, the present study’s findings have implications for researchers and educators who are involved in designing and implementing extracurricular IT/STEM-learning experiences for high school students. The study adds to the field’s understanding of providing effective afterschool IT/STEM-learning experiences at the high school level. The study further suggests that a greater impact on high school students’ career aspirations for IT/STEM fields would most likely require additional experiences or different strategies. Further research seems necessary in this area. In a recent report, the National Science Board (2010) highlights that many of the opportunities for STEM-related activities are available in the form of informal, out-of-school enrichment activities rather than as an integrated part of a STEM curriculum. The report argues that these extracurricular activities are valuable to inspire interest in STEM, but “insufficient” by itself, given the fact that students spend a good amount of their time in the regular classroom. “Formal and informal education are mutually reinforcing and are most effective when synchronized” (p. 17).

As a next step, the authors of this study recommend a more rigorous research study to fully investigate the impact of the Community of Designers model and the IT/STEM-learning experiences that were implemented in the program on urban high school students’ IT/STEM learning. Impact of out-of-school IT/STEM activities on attitude changes and IT/STEM-related career aspirations are also recommended areas of further investigation.