Standards are being raised throughout the education profession as part of the “audit culture” (Blyth et al. 2010). This includes research-intensive universities where pressure to publish more—and more of what really matters—has increased. As publication pressure has increased, more and more journals, books, and other publication outlets have emerged. In our search for outlets where we could reasonably expect to publish our own research, we identified nearly 300 journals. Of course, these journals are not equal in importance or in the potential to influence practitioners, teachers, or other researchers. Thus a critical aspect of the scholarship process is choosing where to submit manuscripts. One key factor in this decision process is the relevance of the journal to the topic of the manuscript and its timeliness in completing the review and publication cycle, as these factors allow the author to communicate the message of a paper to a particular target audience in a timely and influential way. Once a pool of relevant and appropriate journals has been selected, a second important factor in choosing where to publish is the quality of the journals. In fact, and perhaps to the detriment of scientific inquiry, funding agencies and promotion committees often emphasize the importance of “quality” publications in “top-tier” journals (Bird 2007; Blyth et al. 2010) even above publishing to a specialized target audience (Lavie 2009).

The question of which journals are top tier is extremely challenging and precarious. Many scholars cannot agree as to what makes one publication outlet stronger than another. Without clear criteria, many scholars guess at where they should submit their work, hoping others will approve of their decision. In the absence of clear criteria, funding agencies and promotion committees, which are often comprised of scholars from disciplines unfamiliar with the publications of other scholars’ fields, are forced to guess at how to evaluate a scholar’s portfolio. This can lead to inaccurate evaluations about what matters and what is important, causing them to overvalue or undervalue a scholar’s contribution.

This situation exists at our university, as we imagine it does at many others. We have observed that, unless scholars can clearly articulate their rationale for journal selection, they may be judged inaccurately by scholars who review work outside of their own field, such as those on university promotion and review boards. For example, many scholars from other disciplines rely almost exclusively on quantitative citation indices, which often do not index large portions of educational journals, leaving our colleagues at a disadvantage. Neither is the field of education alone in this misrepresentation. Scholars in the social sciences, the humanities, technological sciences, law, and other fields have all expressed similar concerns. At the encouragement of our own College of Education, we have developed a proposed framework for discussing the merits of publication outlets in our field. This framework does not offer hard answers (e.g., Journal X has a rating of Y and thus is Z% better than another journal), but we believe it creates a language and a set of theoretical criteria for explaining the choices we make about where to publish. While applying these criteria is still challenging because there is little agreement about what the data sources should be for each, they have still proven helpful in assisting us to explain our scholarship to others in a way that allows them to understand the relative merits of different types of publication outlets. In this article, we present this framework by first exploring the perils of evaluating journals based on a single quantitative measure (e.g., the Impact Factor rating of the Institute for Science Information’s [ISI]). We then discuss previous efforts to evaluate scholarship using multiple criteria and describe emerging technologies that create new considerations. Finally, we present a framework consisting of three key criteria for evaluating scholarship (rigor, impact, and prestige) along with the kinds of data that theoretically could be collected as evidence for each of these criteria. We present these ideas in the spirit of discussion and realize that applying these criteria is more difficult in practice than in theory; thus, we welcome feedback from the academic community on how to refine these thoughts.

The Problem with Impact Factors

Initially developed in 1964 (Taris 2006), impact factor (IF) is a statistical measure of the average number of citations of articles in a given publication. In 1978, Hirst suggested using IFs to create a list of “core” journals in any discipline. Recently Hirst’s suggestion seems to have gained momentum, with funding agencies and promotion committees increasingly requesting impact factors or other comparable quantitative measures of journal quality as determinants of quality (Weingart 2005). Some have used the IFs to rank journals from the “best” to the “worst” (Wagner 2009). Other scholars have rejected these methods of interpreting IF data and have questioned the validity of ISI’s impact factor specifically, as well as the merits and pitfalls of similarly quantitative measures of journal quality (Amin and Mabe 2003; Opthof 1997; Seglen 1997). The criticisms of IF and other comparable quantitative measures typically fall into four categories: (1) the wide variance between disciplines, (2) the potential for a quantitative measure to be manipulated or skewed, (3) the validity of evaluating a journal’s impact when considering the value of an individual researcher, and (4) the bias in citation metrics for and against certain types of research.

First, impact factors and other quantitative measures vary widely by discipline. This can create challenges when promotion or funding committees, comprised of scholars from many different disciplines, need to evaluate the importance of scholarship outlets. Without understanding this wide variance, scholars from other disciplines may unfairly critique a scholar’s publications if they are not all published in journals with high ISI impact ratings. For example, an ISI Journal Citation Report in the category of “Education & Educational Research” returns only a single journal out of 139 with an IF greater than three and only nine other journals with an IF of 2.0 or higher (7.2% of total education journals). In contrast, over 40% of the journals returned in the category “Chemistry, Analytical” and nearly 90% of the journals in “Developmental Biology” are listed with impact factors of 2.0 or higher (see Table 1). Because of this wide disparity among fields, many researchers have argued that IF is not a valid measurement for journals in fields such as nursing (Melby 2005), communication (Levine 2010), medicine (Barbui et al. 2006; van Driel et al. 2007), developmental psychology (D’Odorico 2001), and social work (Furr 1995). An example of the problem this situation creates is evident in the authors’ field (i.e., education), where only 26% of the 2,000+ articles indexed in the Education Resources Information Center (ERIC) are even included in the ISI (Corby 2001), making IF particularly problematic for interpreting educational research quality. By this measure, even sources included in ISI are misrepresented, as possible citations from omitted journals would not be included in their overall rating. Thus, under-representation of publications is a multi-faceted problem.

Table 1 Discrepancy between relative ISI impact factor scores for different fields of study
Full size table

Second, citation metrics can be manipulated and skewed (Lavie 2009). For example, several scholars have cautioned that IF penalizes journals that publish articles that are innovative, question traditional theories, or represent certain subcultures (Boor 1982; Nkomo 2009; Wicks 2004). In addition, citation metrics can be greatly affected by self-citations (Foley and Sala 2010), geographic or linguistic origin of the research (Rey-Rocha et al. 2001), the number of articles a journal chooses to publish in a year (Rousseau and Hooydonk 1996), multi-authored articles (Sala and Brooks 2008), the quantity of publications (Haslam and Laham 2010), and even the editorial policy of the journal’s editor (Sievert and Haughawout 1989).

Third, the validity of journal citation metrics for evaluating individual researchers has been debated. Although impact factors represent the total impact of a journal, citations vary widely among papers within a single journal, making a statistic of impact for average articles largely uninterpretable (Lavie 2009). Most researchers would not accept the mean of a dataset as the only measure in their own research, instead requesting to know also the range and variance (at a minimum). Yet over-reliance on IFs as indicators of publication quality commits the same error. Further, the Internet and electronic databases allow articles published in journals with smaller impact ratings and even books or other non-periodical outlets to be found and accessed by others (Lavie 2009). Tyrer (2008) concluded that, while IFs can provide a reasonably good evaluation of a journal, they do not provide a valid assessment of the quality of individual papers or researchers.

Fourth, an emphasis on citation metrics is biased against certain kinds of research. For example, in education only 18.9% of 1042 education journals have been reported as indexed in the Thomson-ISI database (Fairbairn et al. 2009). In our analysis of the journals indexed in Scopus, another database indexing scientific journals, only 4% of the 17,500 indexed journals represented the social sciences. This poor representation can lead to gaps in scholarship, for example, against innovative new methods of research dissemination. Leading open-access journals, such as Open Learning, the International Review of Research in Open and Distance Learning, and others are often too new to appear in citation indices. Bias also exists against entire communities of inquiry, such as research on distance learning. For example, none of the four journals(presumably, four of the top journals in this field of inquiry) included in Lee et al. (2004) review of distance education research are listed in the Thomson-ISI database, even though some have been around for at least two decades. Perhaps more surprising was that our search did not find a single journal with the word “distance” in its title in the ISI index. Clearly, if there is pressure to publish in journals with high impact ratings, scholars will be discouraged from researching how to improve distance education, which is shocking when we consider the rapid rise in distance education technologies and models, particularly in higher education.

The ability of impact factors to discriminate against certain kinds of important research is not limited to distance education, but has been cited by other scholars as well; and this discrimination can manifest itself in various ways. Haslam and Laham (2010), for example, found that a researcher’s personal impact rating was strongly associated with publishing more articles, rather than publishing articles of higher quality. Rousseau and Hooydonk (1996) and Sala and Brooks (2008) found that there was a direct relationship in their disciplines between the quantity of articles in a journal and that journal’s impact factor, which could persuade a journal to publish “news, letters, responses to previously published items, commentaries, and other short testimonies” (Sala and Brooks 2008, p 1,139), as well as shorter research articles instead of lengthier, in-depth research studies, in an attempt to raise impact factors. In addition, Levine (2010) reported that specialty journals are often rising in their impact factors, while broad, established national journals see their ratings going down. Finally, citation metrics can be biased against high quality longitudinal research that requires a longer gestational period (D’Odorico 2001).

These realities could lead scholars to find it more worthwhile to publish short, quick, and perhaps less important work in narrower journals, leading to higher impact factors but perhaps less real growth in the scholarly discussion within a discipline. As an example, van Driel et al. (2007) lamented that family medicine journals often have lower impact factors because their interventions take years to produce results. “By nature, the scientific impact factor of family medicine research is low, but its potential to influence the quality of our lives and to contribute to equity is high” (p. 401)

An additional concern is that impact factors can be biased towards trendy or popular topics that will be picked up and cited by others, rather than new, significant changes in direction that may take years to become accepted. In fact, Boor (1982) worried that “the journal’s impact factor is likely to be reduced by its publication of articles that do not conform to current customs, fads, and fashions” (p. 975). Similarly, because the ISI Social Science Index is not updated annually, scholarship on emerging technologies or ideas may be difficult to evaluate.

Recent Efforts to Evaluate Scholarly Publication Outlets

Does this mean that journal impact ratings are not useful? Not at all. These ratings still provide a valuable piece of information about the potential quality of a journal. We believe, however, that impact factors are only one piece of the puzzle and thus are not valid if considered in isolation from other criteria. Neither are we the first to propose that there are problems with an exclusive dependence on impact factors. Understanding some of the many ways in which others have addressed scholarship evaluation may highlight pertinent issues. In this section, we will describe several major initiatives that have emerged recently to evaluate academic quality: the ERA initiative in Australia, the RAE effort in the United Kingdom, the SCImago Page Ranking based on the Scopus database, and the use of emerging technologies such as Google Scholar.

One example of the use of multiple indicators to determine the overall quality of a publication outlet is the Excellence in Research for Australia (ERA) initiative. ERA is a cross-disciplinary effort to evaluate research in Physical, Chemical, and Earth Sciences as well as Humanities and Creative Arts. The initiative examines nearly 21,000 publication venues based on (a) a journal’s IF (if it has one); (b) the Esteem Percentage, a measure of the perceived importance of a journal; and (c) the composition and qualifications of the editorial board. These indicators are evaluated together to result in a single composite score (Fairbairn et al. 2009). The goal is to get a more holistic understanding of a particular venue. Perhaps unwittingly, by merging all these indicators into a single metric, ERA masks the diverse ways in which a publication might or might not contribute to quality scholarship. For example, a publication with a high impact factor, average esteem, and average editorial board might receive a score similar to a publication with low impact but is highly respected with a highly touted review board. A composite score masks the variation gained from using multiple indicators, when many times it is important for a scholar to know about a journal’s strengths and weaknesses in these various areas.

The Higher Education Funding Council for England has battled with the question of measuring academic quality for nearly two decades. The Research Assessment Exercise they used to judge the quality of research in the United Kingdom’s system of higher education was highly criticized for its narrow definitions and the difficulty it created for academics to judge scholarship outside their fields. In response, the Council has recently proposed a re-worked Research Excellence Framework (Gordge 2011). This new framework redefines the impact of scholarship in terms of academic, economic, and output impact. Such redefinition seeks to establish and reward socially responsible research. In so doing, however, it marginalizes entire disciplines (Bekhradnia 2009). The RAE exercise points to the need for a framework to be simultaneously robust enough to identify quality through rigor and impact, flexible enough to be applied by several disciplines, yet simple enough that evaluators hailing from outside disciplines can reasonably evaluate the publication quality in a field not their own.

SciVerse Scopus (hereafter called Scopus, available at http://scopus.com) is another effort to provide citation information and journal rankings. Based upon Elsevier’s database and managed by an external advisory board (SciVerse Scopus 2011), Scopus accepts research and practitioner journals, as well as periodical books, after evaluating the titles according to the publication’s policy, content (i.e. readability and impact), frequency of being cited, regularity in publication, and online availability. Scopus claims to index more than 18,500 titles from 5,000 publishers from around the world, including 1,800 open access journals. The Scopus database provides the basis for the SCImago Journal Ranking (http://www.scimagojr.com/), which ranks journals and countries by their comparative academic prestige based on the Google PageRank algorithm (González-Pereira et al. 2011). While Scopus together with the associated SCImago Journal Ranking provides valuable information, it suffers the same weaknesses as the ISI database and its impact factor. First, it focuses on citations as the main indicator of publication quality. Second, these citation statistics vary widely between disciplines and can be skewed or biased. Finally, many of the non-citation criteria used by Scopus for accepting titles into the database are suspect (e.g., the “quality of the journal homepage”, and “cited references in Roman script,” see SciVerse Scopus 2011).

Modern technologies provide additional ways to think about publication importance. For example, Publish or Perish (PoP, see Harzing 2011) uses Google Scholar citations to create impact metrics that reflect simple reports of citation counts to more complex statistics showing the influence of self-citation, immediacy, and other issues. This technology indexes a wider range of publications than ISI. Handbooks, for example, and theory or practitioner publications (e.g., Educational Technology Magazine) are not indexed in the ISI, yet they can still play an important role in disseminating scholarship. Because Publish or Perish does not discriminate against these publication outlets, it is easier to find data on their potential impact. As an example, the citation metrics for some leading handbooks compare favorably with top tier journals. It is also possible within PoP to limit searches to specific years, thereby getting a time-sensitive view that can be especially important for newer publications or research agendas that may not yet be well indexed in traditional citation databases.

New technologies for calculating citation metrics such as Publish or Perish are not free of bias themselves. The increasing availability of, and access to, online databases of information has begun to alter scholars’ citation patterns, influencing future research. In a study on citations in the field of economics, Depken and Ward (2009) found that the easy availability of JSTOR (Journal Storage—a popular academic publication database) correlated with a decrease in references to articles not found in JSTOR. As new technologies emerge that enable scholars to archive and mine data in new ways, so, too, should our understanding of the resultant benefits and perils expand.

We can draw several key conclusions from these various efforts to evaluate academic quality. First, no single metric should be used in isolation, but rather ought to be used in connection with other metrics to give a more holistic understanding within a scholarly field of research. Second, any framework for evaluating publication outlets needs to be simultaneously sensitive to specific disciplines, but easy to interpret by those outside of that discipline. Third, there is a need for a framework that is flexible enough to capture the changing face of scholarship, including technological advances. Finally, there is a need for indicators of quality to be timely, reflecting the current state of the field. We now present a framework which proposes three criteria that could be triangulated to provide a more holistic understanding of the importance of academic publications.

A Framework for Evaluating Academic Publications

We argue that high quality publication outlets demonstrate three characteristics. First, they are rigorous, i.e., discerning, critical, and selective in their evaluations of scholarship. Second, they have impact on others in that they are read, cited, and used. Third, by being prestigious, they are well known to other scholars and practitioners, increasing the prestige of the authors they publish and bringing more light and attention to their work and their institutions. These three criteria—rigor, impact, and prestige—have the potential to create a more holistic assessment of the value of a body of scholarly work.

Rigor

High quality journals are rigorous, meaning they are more critical in their reviews, are more discerning about what they will accept and publish, and apply higher standards for judging quality research than other journals. They question all aspects of an academic study, including theoretical foundations, participant sampling, instrumentation, data collection, data analysis, conclusion viability, and social impact. They make decisions about the quality of research on its own merits: i.e., through blind review by distinguished and experienced peers and editors. Being published in a rigorous journal lends credibility and acceptance to the research because it indicates that the author(s) have successfully persuaded expert scholars of the merits of the article.

When evaluating the rigor of a journal, authors often consider the acceptance rate as a key indicator. However, judgments based solely on acceptance rates must be made with care because journals calculate their rates differently. Additionally, a lower-tier journal may receive lower-tier quality manuscripts and accept very few of them, resulting in a low acceptance rate but still poor quality publications. Despite these issues, the journal’s acceptance rate may be documented as one measure of rigor. Other indicators of rigor might include a policy of double blind peer review, the number of reviewers, and the expertise and skill of these reviewers and the editorial board, who determine how discerning, rigorous, and selective the journal will be. Editors are especially of primary importance, as they resolve contradictory reviews and make final determinations of scholarship quality.

Many indicators of rigor are currently already documented and ought to be considered when evaluating the quality of a publication outlet. For example, acceptance rates, review policies, and the number of reviewers may be found on the journal’s website or through bibliographic sources such as Cabell’s Directories (http://www.cabells.com/). It is much harder to document the rigor of the reviewers and editors, and this is ultimately a subjective interpretation. Like all subjective decisions, the best method of verification would be to seek opinions of other qualified scholars in the field to confirm or deny your own.

In collecting evidence of the rigor of a publication outlet, we believe the following questions might be useful:

  • How does the acceptance rate compare with other journals in this specific discipline?

  • How is the acceptance rate calculated, if known?

  • What type of peer review is used? Is it editorial, blind, or double-blind? How many reviewers are used to make decisions?

  • What is known about the quality of the reviewers and editorial board? Are they recognizable to other experts in the field and known for their insights into the research? How rigorous would outside experts believe these reviewers and editors to be?

Impact

Impact refers to how extensively individual manuscripts and publications are referenced by other publications and how much they contribute to the scholarly progress of a discipline. In this article, we are referring only to impact on research and theory development, not on actual practice. Undoubtedly impact on practitioners is an important quality of good scholarship, as it could be argued that true impact is only felt on the practitioner level. However, we do not address practitioner impact because this framework is focused on criteria for evaluating academic research and theory publications. We can conceive of the possibility of another framework being developed to guide the evaluation of how much impact an academic has on actual practices, with different evidence being presented and analyzed; but that is beyond the scope of this article.

In evaluating the academic impact of a publication, in addition to the ISI Impact Factor, authors might also review the citation statistics provided by Scopus, SORTI Esteem or Q Scores (i.e., a ranking of journals within specific disciplines), as well as Eigenfactor, Immediacy, h-index, and Cited Half-life Scores, which are other indicators of impact based on statistics that represent attempts to avoid some of the bias in the traditional IF. Because these metrics, available through either ISI, Scopus, Publish or Perish (Harzing 2011), or Google Scholar Citations are affected by how extensively a journal is indexed in particular databases, it is important to triangulate impact statistics from multiple venues. For example, while ISI has been reported to only index 26% of educational articles indexed by ERIC (Corby 2001), we have found Google Scholar to typically index most major educational journals, including those that are not indexed in ISI. In addition, Google Scholar indexes non-academic publications and handbooks, which are still often valuable but not indexed in ISI or Scopus. Thus we believe that Publish or Perish, which calculates citations in Google Scholar, is often more meaningful and accurate in its impact ratings for our discipline. This may not be the case for every discipline. As the major citation databases were originally invented to provide a picture of citation metrics in the hard sciences, fields such as chemistry and physics seem to be better indexed in the Thomson-ISI.

Additionally, a journal’s circulation, its publisher’s effectiveness and reach, or the availability of the journal on the Internet indicate its potential for impact (although potential may not be realized). Emerging social networks such as Mendeley (http://mendeley.com) and Academia.edu provide statistics that indicate how often individual manuscripts are searched for or saved to other scholars’ citation databases. Analytic data from social networks, search engines, and publisher downloading statistics could provide an interesting estimate of how much a publication or author is read or sought out by others.

Some non-peer-reviewed outlets have greater impact than those that are peer reviewed. For example, publication in a widely read and cited practitioner outlet can have high impact. In addition, a Publish or Perish search reveals that some highly cited books are more highly cited in Google Scholar than many top journals. Thus, while peer review would be a prime indicator of the rigor of a journal, non-peer-reviewed outlets may be able to show high impact, indicating they still have value. This also shows the need to triangulate findings from all three criteria.

In collecting evidence of the impact of a publication outlet, we believe the following questions might be useful:

  • Is the publication indexed in ISI or Scopus? If so, what is the impact rating (ISI) or citation count, h-index, and SCImago Journal Ranking (Scopus)?

  • What are the impact ratings according to Publish or Perish? Here we believe it is useful to use the same time window as that used by ISI or Scopus. So for example, if you typically use the 5-year ISI Impact Factor, then it would be wise to also limit your Publish or Perish search criteria to the last five years to retrieve comparable statistics.

  • What is the open-access policy of the publication outlet? Outlets that embrace open-access delivery have the potential to have more impact, as the articles are more easily found through Internet search engines. However, the open-access nature of a publication outlet is only an indicator that it has potential for greater impact, not that it has necessarily achieved this impact.

  • What is the circulation of the publication outlet? This is also only an indicator of the potential for impact, as many journals are packaged and sold as bundles to libraries, increasing circulation but not necessarily impact. However, greater circulation does indicate the potential for higher viewership and greater impact.

  • Is there any indication that the publication has impact on other scholars? For example, is the book widely adopted as a text for university courses? Is there evidence that the journal is frequently used to influence policy or other research?

Prestige

Prestige is a qualitative judgment about the respect a scholar receives for publishing in a particular outlet. Because it is more qualitative, it is more difficult to evaluate in a promotion dossier or grant application and is perhaps largely a theoretical exercise where scholars honestly question the perceived prestige of a journal where they are considering publication. A possible indication of the prestige of a journal is whether other researchers recognize the journal when asked and whether their intuitive perception is that the journal is of high quality. For example, in the overall field of education, publishing in the Review of Educational Research or the Review of Higher Education is highly regarded because these are prestigious journals, sponsored by major professional organizations, and well known among educational scholars from all disciplines.

More quantifiable and objective measures of prestige might be rigorous surveys of scholars in a discipline to gauge their perception of a publication outlet. As an example, several studies have surveyed researchers in educational technology about publications they recognize, read, and respect (e.g. Holcomb et al. 2003; Orey et al. 2010; Ritzhaupt et al. 2011). These studies provide valuable information on the relative prestige of a publication outlet. Other indicators of prestige may be whether the publication outlet is officially sponsored by a large national or international professional organization, whether the publisher is reputable, and whether the editor and editorial board are well known and respected.

Often prestige alone is used to evaluate the quality of a journal, but this can be faulty since journals rise and fall in relative quality and because prestige is often so subjective. Thus many journals that were highly prestigious 10-20 years ago might still be well known even though their rigor and impact have fallen, and new journals that are perhaps not yet well known may still be publishing high quality research. Prestige, then, can be only one indicator of the quality of the journal to be considered in relation to the other indicators.

In collecting evidence of the prestige of a publication outlet, we believe the following questions might be useful:

  • Are there any published studies investigating the popularity or respectability of publications in this field? If so, is this specific publication outlet listed?

  • How recognizable is the publication outlet to other respected scholars? What is their opinion of its importance?

  • Is the publication published by a well-known publisher? Sponsored by a major professional organization?

  • How well known and respected is the editorial board to other scholars in the field?

Applying the Criteria

In making and then defending our own decisions about where to publish our work, we have attempted to apply these criteria qualitatively—using the metrics and data to inform an inductive decision based on evidence from all three categories. We have found that those outside our field have found it easier to understand our choices because we can justify them by providing data about the relative rigor, impact, and prestige of a particular publication outlet in comparison with other publication outlets in the discipline. This framework has also been helpful within our School of Education, where multiple departments are housed, but where we often need to explain to each other the relative importance of different publication outlets within our specific disciplines. As we sought a framework that would encompass all of the scholarship being conducted within the School, the principles of rigor, impact, and prestige have proven flexible enough to provide a common language that all departments could use, even though the specific pieces of evidence important in each of their disciplines were unique and nuanced.

The following are a few examples of how these criteria could be applied in describing a variety of different publication outlets. Using publications in our own field, we demonstrate how this framework might be used (see Table 2). We have masked the names of the journals to focus our discussion on the framework and evaluation criteria, not the specific ranking of individual journals.

Table 2 Application of the proposed framework to publications in the field of educational technology
Full size table

Decisions on publications such as those represented by #1 and #4 are fairly straightforward. We can see from this chart that Publication #1 scores high in all three categories. As such, we would consider it a top-tier venue for publication. Indeed, we would be hard-pressed to find a scholar in our field that would argue with this evaluation for this journal. On the other end of the spectrum, publication #4 scores relatively poorly in each category, resulting in our own interpretation of a lower-tier outlet for publication.

The difficulty may come in scoring publications #2, #3, and #5. The rigor of #2 appears to be fairly staunch, but it is reviewed only by the editor. However, in relation to its peers, this journal seems to have strong citation numbers. This particular journal is often left out of consideration of measures of prestige because of its lack of blind peer review (Ritzhaupt et al. 2011). However, the leaders in the field regularly use this publication outlet as a venue for publishing new ideas and theories, and consequently this publication is one of the most read in our field (Holcomb et al. 2003). Taken individually, each of the measures we used to rate this publication could be problematic for an external review panel unfamiliar with our field. Taken together, we might rate rigor as mediocre, impact as high, and prestige as high, resulting in an upper, mid-tier publication.

Publication #3 paints a different picture. It has a respectably stringent acceptance rate, but the number of times each article is cited in Google Scholar is low. This may be due to the fact that this publication is viewed as a practitioner journal within our field; and, as such, practitioners are more likely to apply the theories than they are to cite them. Also, in addition to regular research articles, this journal publishes many non-research articles and columns, geared towards informing the members of our professional association. These shorter pieces are indexed in Google Scholar and likely bring down the overall ratio of citations per paper. Finally, this particular journal enjoys high prestige as demonstrated in a survey of important journals in the field, ranking in the top-10 overall. Combining these criteria, our qualitative judgment would be to rate this as a lower, mid-tier publication.

Finally, publication #5 presents an interesting case. It is actually a handbook in the field. As such, it lacks key indicators often used to interpret its worth by those outside the field (i.e. ISI impact factors and acceptance rate). Yet, it is edited by a renowned group of scholars, and the number of times each chapter is cited in Google Scholar is nearly as high as the average number of citations per article in our highly regarded publication #1, demonstrating the high impact of this handbook. It also enjoys great prestige in the field and is used by both novice and experienced scholars. As such, we would rate this as a top-tier publication.

Conclusions

We emphasize that these ideas constitute a proposed theoretical framework for how scholars could make and justify, to those from other disciplines, decisions about where they choose to publish their research. In practice, scholars would still need to engage various sources of data and make sound and well-reasoned arguments for the quality of their publication choices. Even though final judgments about journal quality remain a subjective decision, the framework responds to several of the needs that we identified in current efforts to evaluate the academic quality of publication venues. It is flexible enough to allow for multiple and varied sources of data within the categories of rigor, impact, and prestige. As such, the framework allows for the timely inclusion of new metrics as novel ways of measuring academic quality emerge or evolve. The inclusion of multiple indicators allows the framework to be applied to different disciplines. Finally, it is impossible to use the framework while depending on a single metric as an indicator of quality, which may help scholars avoid this dangerous trap. We do not advocate joining the many indicators into a single metric as that would mask the diverse ways in which a publication contributes to quality scholarship. We also emphasize that this framework provides a common language that can benefit scholars in justifying their publication decisions and assist promotion committees in knowing what questions to ask about a candidate’s publication record. Instead of simply asking what a journal’s impact factor is, we hope that committees would seek or request information on the rigor, impact, and prestige of a candidate’s publication record, leading to a more holistic and accurate assessment.

We welcome discussion about whether these three criteria are the most useful and accurate in evaluating educational technology publication outlets or whether additional criteria might be added to the framework. Engaging in this discussion is critical. If we cannot clearly articulate the criteria for determining the quality of our publication outlets, then others (i.e., promotion committees and funding agencies) will have to draw their own conclusions using metrics and criteria that may be less useful or even inapplicable to our disciplines. Also, we emphasize that we believe these criteria should be applied flexibly, qualitatively, and intelligently in making decisions about scholarship quality. We do not recommend using these criteria uncritically to generate a ranking of journals that “count” and “do not count” since all of these data points can be skewed, manipulated, or changed from year to year. Still, by intelligently triangulating multiple data points, we can make more holistic judgments on the quality of publication outlets and share a terminology for discussing our publication decisions.