Introduction

Significant forces in global communication, workflow, and education are converging to realign power, wealth, and work (Folkestad and Banning 2008). As Friedman (2007) explained, many forces have converged to cause a flattening or leveling effect of the world’s workforce, allowing many skilled workers from India, China, and numerous other nations to enter the workplace and compete for jobs that were traditionally held by few wealthy industrial nations. These significant changes are disruptive, causing workers around the globe to retool their skills in an effort to find employment.

Among policy makers and academics, these changes have been recognized and written about for many years. In 1983, in a report titled A Nation at Risk, the National Commission on Excellence in Education stated that, “Our Nation is at risk. Our once unchallenged preeminence in commerce, industry, science, and technological innovation is being overtaken by competitors throughout the world” (Gardner 1983).

More recently, in a report produced by the National Academies of Science titled “Rise Against the Gathering Storm” the authors’ summarized that “participants expressed concern that a weakening of science and technology in the United States would inevitably degrade its social and economic conditions and in particular erode the ability of its citizens to compete for high-quality jobs” (Augustine 2008, p. xi). In an effort to direct future “prosperity,” this highly visible Academy made specific recommendation about Science, Technology, Engineering, and Mathematics (STEM) education. Recommendations included establishing an aggressive undergraduate and graduate STEM scholarship program to attract the brightest scientists and engineers to institutions of higher learning (Augustine 2008). The current STEM focus appears to have originated around 2001 when Dr. Judith Ramaley a prominent educator, decided to change the term from Science, Mathematics, Engineering, and Technology (SMET) to STEM because science and technology support the other two disciplines and it simply sounded nicer (Sanders 2009).

Recognizing the longitudinal discourse, and recent focus and emergence of SMET/STEM we were interested in learning from an analysis of research conducted on SMET/STEM from its conception through current time. The overarching research question for this study was “What can we learn from the examination of doctoral dissertation abstracts that focus on STEM education from 1990 through 2010?” We addressed this question by exploring the following sub-questions: What are the attributes of the researcher and the research? What are the guiding frameworks for the research? What dissertations have been published in academic journals? What are the topics associated with the dissertations? A bounded qualitative meta-synthesis using document and template analysis guided the efforts to respond to these key questions. We present our account of these matters as follows: First, we discuss our meta-study framework. Second, we discuss our methods and procedures. Third, we present our findings. Finally we close with a listing of possible discussion items.

A Bounded Qualitative Meta-Synthesis Framework

Synthesis of research results is becoming a major strategy in literature reviews and establishing evidence based findings (Lipsey and Wilson 2001). Typically in synthesis work, a research topic or question is pursued by examining all the relevant research. An example of this approach is quantitative meta analysis (Glass 1976) where procedures using “effect size data [to] permit meaningful numerical comparisons and analysis across the studies” (Lipsey and Wilson 2001, p. 8) are used to establish the findings of the synthesis. Our study, however, uses a bounded qualitative synthesis framework (Banning and Kuk 2009) which diverges from the typical quantitative meta-analysis. This divergence is as follows: (a) the study is bounded by a search of a specific research genre—the doctoral dissertation abstract and bounded by a specific time period and therefore, more restricted than the typical quantitative meta analysis, (b) the study is framed within qualitative research not quantitative (Major and Savin-Baden 2010; Noblit and Hare 1988) and uses qualitative document analysis (Altheide et al. 2008) and template analysis (King 1998) as the specific research and analytic strategy for the study, and (c) the meta-study approach (Paterson et al. 2001) focus attentions beyond the typical look at findings to include an analysis of methods and theories.

Methods and Procedures

Establishing Data Set

Dissertation citations and abstracts were found by searching the ProQuest Dissertation and Theses™ digital database from 1900 through 2010 using the search terms education, science, technology, engineer, and STEM. In the 1990s, the acronym SMET preceded STEM, but due to the possible association with “smut”, STEM became the acronym of choice (Sanders 2009). We included both acronyms in our search efforts. However, we did not include dissertations that focused on a single discipline with in the STEM; either STEM or SMET needed to be within the title or abstract of the dissertation. All abstracts found by the librarian search professionals were reviewed by the two authors of this study. Through discussion, consensus was reached on inclusion of 101 abstracts. Foreign language dissertations and master theses were excluded from the study. The study search was terminated January 1, 2011.

The abstract used for the unit of analysis in this study contained the information captured by downloading the ProQuest™ data to Endnotes™ and then exporting this information using the “Show All Fields” filter. For the analysis, the following information was used: author, year, institution, title, and abstract.

Method of Analysis

The methodological framework for the study was qualitative document analysis (QDA) (Altheide et al. 2008). The purpose of qualitative document analysis is to analyze and interpret data from the examination of documents (Schwandt 2007). In this study, the documents were the doctoral dissertation abstracts related to STEM education. QDA is a qualitative content analysis which examines the text of documents for thematic structures (Altheide 1987) rather than counting the presences of predetermine codes as is the procedure in the classical content analysis of documents (Krippendorff 2004). Within the QDA framework, the coding of the dissertation abstracts utilized King’s (1998) template analysis. In this method both deductive and inductive coding approaches are used. Deductive codes were utilized for researcher and research attributes categories. The inductive coding was used in the analysis of the topics of the dissertation abstracts using the inductive coding strategy of the constant comparative method (Corbin and Strauss 2008). Each dissertation was given a descriptive/topical name and then the these codes were used to induce most abstract groupings which became the thematic structure for topics of the dissertation abstracts.

Trustworthiness

The final set of dissertations used in the study represented consensus by two research team members. To insure trustworthiness of the inductive coding process, the strategy of peer examination (Creswell 2007) was used by the manuscript authors to reach consensus in assigning topical codes to dissertation abstracts.

Findings

Researcher and Research Attribute Results

Table 1 presents our findings in response to the question of what are the attributes of the researcher and the doctoral research related to STEM education. Numbers and percentages in Table 1 were determined by using only the confirmed attribute assignments. Of the dissertations completed, 72% were authored by females and 28% were completed by males; 73% were completed by students receiving the Ph.D. and 24% were completed by students receiving the Ed.D. In addition, there was one each representing the D.B.A., Psy.D., and the D.Sc. degrees. Of the dissertation abstracts analyzed, 47% were quantitative and 30% were qualitative. Seventeen percent were mixed methods dissertations and seven of the dissertation abstracts did not reveal the methodology of the study.

Table 1 Distribution of researcher and research attributes

Table 2 shows the distribution of dissertations by year from 1990 through 2010. The first dissertations to meet our search parameters appeared in 2003. The year 2009 was the most productive year with 24 dissertations.

Table 2 Distribution of dissertation by Year

In regard to the degree granting institution, 70 different institutions awarded degrees with sixteen institutions awarding two degrees. Table 3 lists the five institutions that awarded three or more doctoral degrees during the study’s time period. With six degrees granted, the University of Southern California awarded the most doctoral degrees.

Table 3 Listing of institutions awarding over three degrees or more

Guiding Frameworks Results

Nearly all research is built upon or guided by previous work. In this study, the theory component of the meta-synthesis model (Paterson et al. 2001) led to the following question of the data: what were the guiding frameworks important enough that the researcher included their mention in the abstract of the study. To answer this question each abstract was examined for the presence of a guiding framework and then subsequently was coded by the guiding framework classification system used in this study.

The guiding framework classification system used in this study is comprised of three components: theoretical guiding framework; instrument based guiding framework, and a literature based guiding framework. The theoretical guiding framework is defined within this study as any use of theory as guiding the research. A range of theory levels (Schwandt 2007) were used. The levels include the following: level of grand theory—theories that are very abstract, formalized, and general (Mills 1959); theories of middle range (Merton 1968) that are less abstract, less formalized, and less general than grand theory, but based on empirical generalizations; and substantive theories (Glaser and Strauss 1967) which are less abstract, formal, and general than middle range theories and are often specific to a particular phenomenon or group. The mention of a particular model was also regarded as falling within the rubric of a theoretical guiding framework (Jaccard and Jacoby 2010). The instrument guiding framework is defined as framework that is built on an existing research instrument, for example, a specific published survey or inventory (Banning and Kuk 2009). Finally, the literature guiding framework is defined as research that invokes mention of key background literature in the abstract. Table 4 presents the results of applying this classification system to the 101 abstracts of the study. Seventy-six (75%) of the abstracts did not mention a guiding framework for the study. Twenty-six percent of the abstracts mentioned a guiding framework with 17% of the abstracts mentioning a named theory or model.

Table 4 Guiding frameworks noted in abstracts

Publishing of the Dissertations

To answer the question of how many of the study’s STEM education related dissertations were eventually published in academic journals, two sources were searched. These were the WEB of Science ® and the Academic Search™ Premier database. Web of Science® provides coverage of over 10,000 journals and conference proceedings and The Academic Search™ Premier data base includes over 8,000 journals. Of the 97 dissertations in the study, only five (Fadigan 2003; Moriarty 2006; Moye 2009; Norman 2008; Rutherford 2007) were found to be published in academic journals and an additional two (Obarski 2007; Oware 2008) were published in conference proceedings. Dissertations in the study from late 2009 and 2010 would not be expected to be in journal print at the time of the present study.

Topical/Thematic Results

Using the method of constant comparative analysis (Corbin and Strauss 2008) each dissertation topic was given a descriptive code and then compared and contrasted with all other assigned codes. From this list of codes a thematic categorical structure was induced that organized the dissertation topics into two major categories: Recruitment and retention issues with sub codes representing two major groups—underrepresented and the non-underrepresented; and STEM academic focus issues with sub codes of instructional issues, program evaluation, role of STEM graduate students, and mentoring. The diversity group of interest and setting were noted for each of the sub codes where appropriate. Tables 5, 6, 7, 8, 9, 10, 11, 12 display the results of the inductive coding procedure and the appendix is organized by table topic and setting to give greater detail and specific links to the dissertations. While some dissertations related to several topic areas each dissertation was assigned only to the most representative topic.

Table 5 Recruitment of the underrepresented
Table 6 Recruitment of the non-underrepresented
Table 7 Retention of the underrepresented
Table 8 Retention of the non-underrepresented
Table 9 STEM instructional issues by setting
Table 10 STEM program evaluation by setting
Table 11 STEM graduate students by setting
Table 12 STEM mentoring by setting

Discussion

The discussion focuses on the issues of quality, the use of the currently poplar pipeline metaphor to describe conditions related to underrepresented retention, and the instructional activities of K-12. While it is difficult to judge the quality of the research from examination of abstracts, nor was it the goal of this study, two important items for discussion emerged. One is the quality of the abstracts and the second is the lack of guiding frameworks. In many cases the abstracts were not as useful as they could have been in terms of providing important information about the study. This issue of abstracts lacking important information is not specific to those in this study and has become an agenda on the national research scene. Kelly and Yin (2007) call for the strengthening of structured abstracts for education research. They suggest improving the writing of abstracts to enhance their usability to develop and support arguments. They note the abstract should address the following areas: Background and content, purpose/objective/research questions of the study, setting, population/participants, intervention/program, research design, data collection and analysis, findings/results, and conclusions/recommendations. This need for improvement is especially germane to synthesis studies such as this current study where the abstract may be the major source of data.

Before discussing the second quality issue, a cautionary note needs to be stated. An abstract of a dissertation is not the dissertation. However, the lack of a guiding framework (Ravitch and Riggan 2012) to be mentioned in the abstract suggests its lack of importance. Seventy-five percent of the abstracts mention no guiding framework. The most often mentioned theory/model was Weidman’s (2006) conceptual framework of undergraduate socialization. For example, Tinto’s (1987), conceptual framework that focuses on retention issues in higher education, was notably absent from the abstracts.

Finally, a comment on the issue of non-publication of the dissertation research is offered. This study did not seek an answer to why so few dissertations were published in academic journals. The lack of motivation to publish and the lack of acceptance by a journal are two obvious possibilities. This finding, however, does suggest that there is important research being conducted in STEM education that remains relatively hidden.

QDA revealed that forty percent of the dissertations were on the topic of STEM recruitment and retention and within this group nearly 80% targeted issues of diverse populations (see Tables 5, 6, 7, 8). The authors raise two issues in the form of questions related to these findings. First, is the magnitude of studies that focus on underrepresented groups warranted? And second, if so, is the current utilization of the pipeline metaphor helpful? In regard to the first question the findings of Riegle-Crumb and King (2010) regarding gender and racial/ethic differences in STEM field should be noted. They summarize as follows:

The authors analyze national data on recent college matriculates to investigate gender and racial/ethnic disparities in STEM fields, with an eye toward the role of academic preparation and attitudes in shaping such disparities. Results indicate that physical science/engineering (PS/E) majors are dominated by men, but not, however, disproportionately by White men. After accounting for high school preparation, the odds of declaring a PS/E major are two times greater for Black males than for White males, and Black females are closer than White females to closing the gap with White males. The authors find virtually no evidence that math attitudes contribute to disparities in choice of a PS/E major. Finally, in contrast to PS/E fields, biological sciences draw relatively equitably from all groups. (p. 656)

If there remain significant issues in recruiting, then our findings suggest two important considerations. First is the question of who is doing the research. Seventy of the dissertations were completed by women. Of the 20 dissertations that focused on women, 85% of these were authored by women. This finding raises the question of “marginalization.” Is the topic of STEM education and underrepresentation “women’s only issue?” A second consideration is regarding the use of the pipeline metaphor. Of the 40 dissertation abstracts focusing on the recruitment and retention of underrepresented groups, 25% invoked the metaphor of the pipeline. In a recent paper by Svinth (2008) the usefulness of the pipeline metaphor in STEM discussions is questioned:

Pointing at the many weakness of this quite popular metaphor, the paper will question the usefulness of the metaphor when it comes to understanding why women at a disproportional rate leave science academia. The paper argues that the metaphor oversimplifies the highly complex issue of retention and exit from science careers and thus does more damage than good. To ensure a more thorough understanding of how the representation of women in science academia develops and accordingly proper political actions, it is time to give up the notion that the ‘leaky pipeline’ metaphor communicates the problem. (p. 1)

Svinth (2008) notes the following shortcomings: The metaphor implies that PhD degrees and careers in science academia are the goals of education; it considers only the ‘push’ effect not the ‘pull’ effect; it is unidirectional; it is value laden, and it does not consider the entry problem.

The work of Metcalf (2010) supports the point of view presented by Svinth:

I argue throughout that the pipeline model has a limited view of retention that is based upon socially constructed ideas about what constitutes ‘valid’ scientific and engineering work and who counts as ‘real’ scientists and engineers. (p. 1)

Metcalf presents the “chilly environment metaphor” as a more useful alternative. Pawley (2007) speaks specifically to the usefulness of this metaphor in addressing retention:

A Chilly-climate based model … suggests that leaks are caused by a ‘chilly environment’ that discourages people already under environmental stress (again, women and people of color) from remaining. Programs that attempt to stem these leaks provide metaphorical ‘sweaters’ (survival tools for underrepresented populations to better withstand the chilly environment) or train their white, male counterparts on how to ‘turn up the thermostat’ by implementing, for example, parent-friendly tenure procedures, gender-neutral hiring protocols, or the much-maligned idea of ‘sensitivity training’.

(p. 7)

The Pawley (2007) metaphor suggesting that retention can be improved by providing tools to the underrepresented for warming up the chilly environment or by turning up the institutional thermostat is in concert with looking at program interventions from an ecological perspective (Felner and Felner 1989). Within the ecological intervention model there are three broad strategies: those that focus on the characteristics of persons or participants, those that focus on conditions of the environment or program and those that address both person characteristics and environmental conditions. Using this model to examine the thirty-one dissertation abstracts addressing the retention of the underrepresented; twenty-one addressed person characteristics, two addressed environmental conditions, and eight focused on both person characteristics and environmental conditions. These results suggest that the primary intervention strategies for improving retention for the underrepresented are focused more on the student than making changes in the environment (turning up the thermostat).

The foregoing leads to the last item of discussion. Where is the focus on changing the STEM learning environment occurring? Of the 36 dissertation abstracts focusing on STEM instruction, only 30% were conducted within the community college and college/university settings. Innovations in conditions of instruction were more likely to be found in the K-12 setting. These included items as instructional delivery methods, curriculum revisions, development of instructors, and the use of technology in the classrooms.

Finally, a note regarding the limitations of the study is needed. Only dissertation abstracts were used, not full dissertations. This limits the full discussion on the content of the dissertation, for example, the traditional look at “what works” in meta-analytic studies is missing. The focus is more on the researcher and the research attributes and broad topical categories than what the research promotes in new understanding and application of findings. However, the analysis of abstracts can give guidance to those seeking this understanding and application.

Summary

The overarching question addressed by this study was: What can we learn from the examination of STEM/SMET doctoral dissertations abstracts. One hundred and one abstracts were examined. Recruitment and retention of the underrepresented groups emerged as the major research topic particularly at the post-secondary level. Instructional topics were most likely to be addressed within the K-12 setting. Discussion of findings focused on the issues regarding the quality of the abstracts, the lack of publication, and the continued use of the pipeline metaphor and the lack of instructional innovation at the post-secondary level. Caution was noted that the study examine only dissertation abstracts not full dissertations, but the methodology used in this study gives an understanding of past research and issues that future research might address.