Since United States (U.S.) postsecondary institutions proliferated in the mid-1800s and greatly expanded throughout the 1900s to comprise over 7000 unique postsecondary institutions today, these colleges and universities have increased their competition over students, faculty members, staff, and resources (Hazelkorn 2015). From this competition, a plethora of college and university ranking systems have been created and employed to measure various success indicators of an institution, including the first U.S. News & World Report ranking in 1983 (Boyington 2014) to Times Higher Education’s world university rankings in 2004 (Times Higher Education 2018) to Money magazine’s recent value-based ranking system in 2017 (Money 2017). Beyond institutional ranking systems, many systems have begun ranking individual academic programs and disciplines within institutions, such as the Quacquarelli Symonds (QS) World University Rankings by Subject (Quacquarelli Symonds 2018), the Center for World University Rankings’ (CWUR) Rankings by Subject (Center for World University Rankings 2018), and U.S. News & World Report’s ranking of graduate programs (U.S. News & World Report 2017). The latter is the subject of this study’s focus.

Extant research has documented the influence of the U.S. News & World Report rankings on how administrators view peer institutions (Bastedo and Bowman 2010), how hierarchical resources are allocated from administrators, faculty, alumni, and out-of-state students (Bastedo and Bowman 2011), and how prospective students make their choice to attend an institution (Hazelkorn 2015). However, for as the U.S. News ranking system continues to be, there exist several gaps in literature, that if filled, may lead to a better understanding of the popular ranking system.

First, no extant research has examined specific academic program U.S. News rankings—such as the ranking of graduate education programs—to learn which U.S. News assessment criteria best predicts such a ranking. For institutions seeking to improve the ranking of specific academic programs, and for other stakeholders to understand how an academic program achieved its ranking, it is important to delve deeper into how specific U.S. News assessment criteria influences an academic program’s ranking.

Second, no extant research has addressed how the Internet—the most popular source of pre-college information for prospective students (Burdett 2013; Daun-Barnett and Das 2013)—is related to an institution’s or an academic program’s U.S. News ranking. As Internet technologies have advanced, so have the tools with which websites can be measured, and thus, evaluated for their size, popularity, and overall effectiveness. Internet measurement software such as SEMrush allows researchers to measure website—including institutional.edu websites—to learn which websites are the largest, most popular, and features the highest degree of search engine optimization (SEMrush 2018).

As a result, educational researchers now have the ability to measure an institution’s website size, popularity among other websites, and how much money an institution invests in their website and web-based marketing by employing web analytics software applications such as Google Analytics. Researchers have since employed these web analytics applications to analyze how colleges and universities pay for search results placement on popular search engines such as Google, and how an institution’s website size may influence its ranking (Alsmadi and Taylor 2018; Krrabaj et al. 2017).

Ultimately, through descriptive analyses, multiple regression, and tests of multicollinearity, this study will answer three previously unanswered questions regarding U.S. News & World Report graduate education rankings and Internet characteristics.

  1. (1.)

    How large and how popular are the institutional.edu websites of highly-ranked graduate education programs according to the 2018 U.S. News & World Report rankings?

  2. (2.)

    Which U.S. News graduate education program ranking criteria best predicts ranking?

  3. (3.)

    Do Internet characteristics predict U.S. News graduate education program rankings better than U.S. News ranking criteria?

By answering these questions, researchers will better understand how specific U.S. News assessment criteria predicts ranking and how measurable institutional.edu Internet characteristics influence these rankings, potentially helping to explain the stratified nature of the U.S. higher education system.

Literature Review

Educational rankings influence institutions of higher education institutions as they impact student choice (Clarke 2007), the institutional mission (Collins and Park 2016), overall strategy (Lynch 2015), recruitment and admissions (Locke 2014), and public relations efforts (Hazelkorn 2015). Rankings have also been shown to be influential policy instruments that can assess the performance and effectiveness of institutions (Bowman and Bastedo 2009; Salmi and Saroyan 2007; Sponsler 2009), as well as influence the decision-making of institutional leaders (Gnolek et al. 2014). The past 20 years has seen an intensity on the influence of college rankings, especially those provided by U.S. News.

Since 2000, the influence of rankings as it relates to students’ college choice has increased by 92% (Eagan et al. 2017). This growing attention has led to both an increasing backlash from many colleges (Thacker 2005) and a number of recent empirical studies on the various effects of undergraduate and graduate school rankings (Griffith and Rask 2007; Martins 2005; Rindova et al. 2005; Sauder and Lancaster 2006; Volkwein and Sweitzer 2006). The study of rankings and student behavior is not new, as economists and higher education researchers have been exploring this phenomenon for decades (Bastedo and Bowman 2010, 2011; Bowman and Bastedo 2009).

Over the years, institutions of higher education have evolved to operate more like economic enterprises than traditional education spaces. Competition, accreditation processes, influence of government and politics, and the decrease in public funds has helped to exacerbate the shift towards academic capitalism in higher education (Bok 2009; Slaughter and Rhoades 2004). The Internet, however, is an underexplored yet important mediating factor when it comes to influencing the economic enterprises of institutions of higher education. For example, Slaughter and Rhoades (2004) argued that as colleges and universities decreased their focus on knowledge and more on profit-oriented activities, the Internet served as a new frontier to develop, market, and sell research products, educational services, and consumer goods. Furthermore, the authors highlighted how institutions of higher education over the years have used state and public resources to leverage themselves on the Internet to market their institutions as places for new investments and infrastructure (Slaughter and Rhoades 2004). As of yet, only Alsmadi and Taylor’s (2018) and Taylor’s (2018) studies have measured institutional Internet characteristics, with the latter focusing on the Internet characteristics of historically-Black colleges and universities (Taylor 2018). Ultimately, this study seeks to fill an important gap in the literature and explore the relationship between institutional rankings and Internet characteristics, the latter of which have only become measurable in recent years.

Methods

The following sections will detail how the research team identified a sample, collected data, analyzed data, and addressed the limitations of the study.

Data Sources

The research team derived data for this study from three sources: 2017–2018 U.S. News & World Report ranking of national universities and graduate education programs (U.S. News & World Report 2018), the Integrated Postsecondary Education Data System or IPEDS (National Center for Education Statistics 2018) and SEMrush. SEMrush is a search engine analysis program using Google Analytics data to examine paid advertisement strategies, keyword grouping and management, pay-per-click effectiveness, and search engine optimization of websites hosted around the world (SEMrush 2018).

Sample

The research team agreed that analyzing highly-ranked institutions was a priority in this study, as extant research suggests highly-ranked and/or elite institutions often use rankings to justify institutional decisions (Bastedo and Bowman 2010, 2011; Hazelkorn 2015). As a result, the research team delimited this study’s population to the top 100 institutions according to U.S. News & World Report national university rankings. From here, the research team explored these 100 institutions and learned every institution published a website in 2017 (e.g., www.harvard.edu). Once these websites were located, the research team analyzed each institution’s website to learn if these institutions housed a college or school of education. After exploring all 100 institutional websites, the research team learned that 77 institutions housed colleges or schools of education and graduate education programs. Of these 77 institutions, 69 institutions reported graduate education program data to U.S. News for the 2016–2017 academic year, reflected in the 2017–2018 U.S. News graduate education program rankings. As a result, 69 institutions comprise this study’s sample.

Variables

After the research team identified the sample, the research team analyzed each college or school of education’s website through SEMrush (e.g., http://www.tc.columbia.edu/). Once the research team gathered this data, the team employed IPEDS to gather institutional variables including institution type (public or private) and institutional endowment. The research team extracted several Internet characteristics for from SEMrush, including 2017–2018 monthly averages of unique organic keywords, web traffic by visitor, and cost of web domain hosting. Specific to paid adwords per application season month, SEMrush analyzed the number of paid adwords per month, the amount of web traffic by visitor generated by paid adwords per month, and the cost of paid adwords per month. This database is available upon request. Definitions of these Internet characteristics can be found in the sections that follow.

Webpages

A webpage is a page linked to a larger website (e.g., “https://www.harvard.edu/about-harvard/harvard-glance” linked on www.harvard.edu). Each website (e.g., www.harvard.edu) includes a certain number of webpages organized under different directories. Webpages are one measure of Internet size.

Organic Keywords

Organic keywords are number of unique keywords or search terms bringing Internet users to the institutional website (e.g., a user performing a Google search for “Harvard economics,” seeing www.economics.harvard.edu in the search results, and then clicking on the link). Organic keywords are measures per unit of time (e.g., per month) and represent one measure of Internet popularity and search-engine optimization.

Traffic Volume

Traffic volume is the average number of users visiting an institutional website per unit of time (e.g., per month). Traffic volume is one measure of Internet popularity.

Search Cost

Search cost is the cost paid by institutions to link organic keywords in Internet search results placement over a period of time (e.g., one year). Search cost is one measure of Internet investment.

Backlinks

For a particular website, a backlink is the presence of one’s hyperlink on another website. Backlinks are the number of institutional hyperlinks on other Internet websites (e.g., www.harvard.edu linked on https://www.usnews.com/best-colleges). Backlinks are one measure of Internet popularity.

Data Analysis

All IPEDs and SEMrush data were merged and then uploaded to STATA for a stepwise regression analysis. In the regression, the research team decided that institutional characteristics were important to include in the model, these characteristics being overall U.S. News national university ranking, endowment size, and institutional type (public or private). These variables were included in the model as extant research has demonstrated that private, well-endowed institutions are better positioned to purchase institutional supports (Hazelkorn 2015), although no extant research has addressed how institutions purchase website improvements. Moreover, the research team decided to control for institution type and overall U.S. News & World Report national university ranking to consider the context and relative prestige of each institution before analyzing specific graduate education program rankings. All variables with wide variance and large in scale (i.e., endowments in the millions or billions of dollars) were logged to define the regression models. All logged variables are labeled as such in each table of findings.

In addition to institutional characteristics, the research team analyzed all 11 metrics used by U.S. News & World Report (2017) to produce graduate education program rankings. These metrics included (1) overall score (out of 100), (2) peer assessment (out of 5), (3) administrator assessment (out of 5), (4) average admitted doctoral student GRE verbal (maximum 170) and (5) quantitative scores (maximum 170), (6) doctoral admission rate (maximum 100), (7) doctoral students per faculty member (continuous variable), (8) doctoral degrees granted per faculty member (continuous variable), (9) funded research (continuous variable; in dollars), (10) funding research per faculty member (continuous variable; in dollars), and (11) overall graduate student enrollment (continuous variable) (U.S News & World Report 2017, pp. 103–105).

Findings

Descriptive statistics of Internet characteristics of institutions in this study (n = 69) can be found in Table 1 below:

Table 1 Website characteristics of the top 69 graduate education programs in the 2018 U.S. News & World Report whose institution is also ranked in the national university top 100 rankings
Full size table

A descriptive analysis reveals that, on average, public institutions publish larger websites (by webpages), spend more on their websites, and have more popular websites (by organic keywords, traffic, and backlinks) than private peers. Although public and private institutions published similarly sized websites and are comparably popular considering backlink data, public institutions in this sample were much more popular on the Internet considering organic keywords and traffic. These findings will be discussed in greater detail in the Discussion and Implications section of this study.

Results of the multiple regression analysis using U.S. News (USN) ranking criteria can be found in Table 2 below:

Table 2 Multiple regression predicting U.S. News & World Report graduate education program rankings using U.S. News & World Report graduate education program ranking criteria, with semipartial correlations explaining R-squared
Full size table

The first regression model included all USN criteria used to calculate graduate education program rankings, with overall graduate education program ranking serving as the dependent variable. Results in Table 2 suggests peer assessment (t = − 2.21, p = 0.03) best predicts graduate education program ranking aside from overall score (out of 100), with all variables producing an R-squared of 0.88. After a test of multicollinearity to minimize error and better understand R-squared, peer assessment was also responsible for the largest semipartial correlation (0.0124). These results suggest USN graduate education program peer assessment scores may be the most accurate metric to predict overall USN graduate education program ranking. This is an important finding and will be addressed in the Discussion and Implications section.

Results of the multiple regression analysis using institutional characteristics and Internet metrics can be found in Table 3 below:

Table 3 Multiple regression predicting U.S. News & World Report graduate education program rankings using overall U.S. News ranking, demographic information, and internet characteristics, with semipartial correlations explaining R-squared
Full size table

The regression model in Table 3 sought to learn what effect—if any—institutional characteristics and Internet metrics have on USN graduate education program rankings. Findings in Table 2 suggests overall USN national university ranking (t = 2.76, p = 0.00) and public institution sector (t = − 2.60, p = 0.01) best predicted USN graduate education program rankings in 2018. These two variables were also responsible for the highest semipartial correlations after a test of multicollinearity, as overall ranking and public institution sector comprised 0.0444 and 0.0395 of the R-squared value of the model (0.60). It is important to note this model produced a lower R-squared (0.60) than the first model (0.88), although the first model included more variables, likely producing an inflated R-squared value. Additionally, institutional endowment (t = − 2.03, p = 0.05) also predicted USN graduate education program ranking, meaning that larger endowments were associated with a better USN graduate education program ranking in 2018. Finally, considering overall USN ranking alongside institutional and Internet characteristics, no single Internet characteristic predicted USN graduate education program ranking.

Results of the multiple regression analysis using USN criteria, institutional characteristics, and Internet characteristics can be found in Table 4 below:

Table 4 Multiple regression predicting U.S. News & World Report graduate education program rankings using all criteria, with semipartial correlations explaining R-squared
Full size table

The model in Table 4 included all prior variables to learn how these variables predicted USN graduate education program ranking. Results in Table 4 suggest overall score (t = − 2.83, p = 0.00) best predicted USN graduate education program ranking. This finding is logical, as the higher or better a program is ranked, the higher their overall USN score. However, considering all USN graduate education program assessment criteria and institutional characteristics such as sector, overall USN ranking, and endowment, three Internet characteristics best predicted USN graduate education program ranking in 2018: webpages, keywords, and backlinks.

First, webpages (t = 2.55, p = 0.01) and backlinks (t = 2.30, p = 0.03) negatively predicted graduate education program ranking, meaning that the larger a graduate education program’s website was, the lower or worse that program was ranked in 2018. Similarly, the more backlinks a graduate education program had in 2018, the lower or worse that program was ranked. Inversely, keywords strongly and positively predicted graduate education program ranking (t = − 2.75, p = 0.00), meaning that the more organic keywords which led to the graduate education program’s website, the higher or better ranked that program was in 2018. Keywords were also responsible for the highest semipartial correlation value across any variable in the model (other than overall score) at 0.0124. This is a very unique finding, as larger graduate education program websites did not predict a higher or better USN ranking, but organic keywords did predict a higher or better USN ranking.

Given Table 4’s inflated R-squared value, the team performed a variable inflation factor (VIF) analysis to determine multicollinearity and refine the model. After VIF, the team removed funded research (VIF = 32.32), peer assessment (VIF = 16.34), and overall score (VIF = 11.21). Results of this multiple regression analysis, having removed multicollinear variables, can be found in Table 5 below:

Table 5 Multiple regression predicting U.S. News & World Report graduate education program rankings after removing multicollinearity, with semipartial correlations explaining R-squared
Full size table

After removing multicollinear variables, several USN graduate education program ranking criteria predicted ranking, including administrator assessment (t = − 2.39, p = 0.02), doctoral student admission rate (t = 3.30, p = 0.00), and funded research per faculty member (t = − 4.89, p = 0.00). These findings were logical, as higher administrator assessments and higher amounts of funded research per faculty member were related to a higher or better USN graduate education program ranking evidenced by negative critical t values. Similarly, lower (more exclusive) doctoral student admission rates predicted a higher or better USN graduate education program ranking, evidenced by a positive critical t value. This was another logical finding, as institutional exclusivity has been a hallmark of highly-ranked institutions of higher education for decades (Griffith and Rask 2007; Sauder and Lancaster 2006).

Although doctoral student admission rates and funded research per faculty member were the strongest predictors of USN graduate education program ranking in 2018, webpages, keywords, and backlinks were still predictive of USN graduate education program ranking. Akin to the results in Table 4, results in Table 5 indicate larger websites (t = 2.61, p = 0.01) and a greater number of backlinks (t = 2.10, p = 0.04) negatively predicted USN graduate education program rankings in 2018. This finding suggests a larger and more popular.edu website may not be important if a graduate education program is seeking to improve its ranking: In this case, “bigger” is not “better.” This finding will be elaborated upon in the Discussion and Implications section of this study.

However, organic keywords strongly predicted a higher or better USN graduate education program ranking in 2018. Organic keywords are number of unique keywords or search terms bringing Internet users to the institutional website, and organic keywords are considered a measure of Internet popularity and search engine optimization. Organic keywords predicting a higher or better ranking could mean one of two phenomena. First, it could have been the case that higher or better ranked graduate education programs in 2018 were simply more popular on the Internet, evidenced by the volume of search terms (organic keywords) from Internet users trying to access the graduate education program’s website information. Second, it could have been that higher or better ranked graduate education programs published smaller websites that were highly optimized for search engines, allowing Internet users to enter many combinations of search terms (organic keywords) to locate the website and access its information. Ultimately, considering organic keywords, “bigger is better” for highly-ranked USN graduate education programs in 2018.

Discussion and Implications

As the first study of its kind to analyze U.S. News & World Report graduate education program rankings alongside institutional and Internet characteristics, there are several notable findings to spur theoretical discussion and opportunity for future research.

First, results in Table 1 regarding the Internet characteristics of graduate education programs is noteworthy, as this study is the first of its kind to analyze the web presence, visibility, and investment of education program websites. Table 1 suggests public graduate education programs published larger websites, experienced more web traffic, were more frequently searched for given organic keyword totals, spent more on their website, and were more visible on the Internet than private graduate education programs in 2018. Without a guiding hypothesis or extant research to inform these findings, it is difficult to understand why public graduate education programs invested more heavily in Internet technologies and were more popular on the Internet than private peers. Perhaps public graduate education programs are larger and feature more certification or teacher’s licensure pathways, leading to the necessity to invest in program websites and web visibility. Here, future research should examine the web investment of all educational institutions to learn why websites grow, when they grow, and how an institution’s or academic program’s website leads to other areas of growth, such as endowment or ranking.

Second, several results of this study are consistent with research focused on institutional ranking systems and the importance of institutional reputation to determine rankings (Bastedo and Bowman 2010, 2011). For example, this study found peer assessment was the best predictor of U.S News & World Report graduate education program ranking provided the results in Table 2. Theoretically, this result is important in the context of academic capitalism forwarded by Slaughter and Rhoades (2004). If peer assessment best predicts ranking, and elite institutions are best positioned—financially and socially—to remain atop the rankings (Hazelkorn 2015), these elite institutions can continue to assess their peer institutions in ways that solidify ranking systems and perpetuate academic capitalism (Slaughter and Rhoades 2004). Year after year, the same institutions are atop the overall national university rankings and graduate education program rankings, and this phenomenon may come as no surprise, given peer assessment best predicts ranking in graduate education contexts. Ultimately, elite institutions may be able to assess their peers in ways that solidify rankings and marginalize less elite, less wealthy institutions.

Yet, the wealth of an institution does not tell the entire story in terms of graduate education program rankings. Results from Table 5 indicate that institutional endowment is not a good predictor of U.S. News & World Report graduate education program ranking, yet funded research per faculty member is a good predictor. Here, it seems the overall wealth of the institution may not be as important as how specific graduate education programs raise—and subsequently spend—research dollars. Here, educational researchers should continue to explore how faculty members and research centers procure grant and institutional funding to bring prestige—and a better ranking—to their school or college of education. Similarly, results in Table 5 suggest academically-elite doctoral students are strongly related to better graduate education program rankings. As such, educational researchers should continue to investigate how elite graduate education programs recruit academically-elite doctoral students to increase their ranking and prestige among the education community.

However, Slaughter and Rhoades’ (2004) notion of academic capitalism may be less pronounced in graduate education contexts, or the overall endowment of an institution may not affect the finances of a graduate education program or college of education. Here, future research should evaluate the endowment size of schools and colleges of education, compare those endowments to overall institutional endowments, and explore the relationship between institutional wealth and the wealth of schools and colleges of education.

Finally, results from the refined regression model in Table 5 suggest “bigger is not better” regarding the size of graduate education program websites. As larger websites require an investment to write webpages and create content, this study suggests graduate education programs seeking to improve their ranking could be overspending on their website. Certainly, there is an investment to be made when publishing an institutional website, as human resources are required to analyze the effectiveness of a website and search-engine optimize its content for various educational stakeholders. Certainly, larger websites are more expensive to publish than smaller ones. Similarly, it is expensive to analyze organic keywords and tie those keywords to institutional webpages, as this process requires tech-savvy web developers who know how to optimize web content for specific audiences, such as prospective graduate students seeking a certain graduate program or a research assistantship with a desired faculty member.

Yet, measures of Internet popularity either did not predict graduate education ranking (traffic) or negatively predicted ranking (backlinks). From here, it may be important for graduate education programs to understand that the largest, most popular websites may not reap the most benefit. In this study, the highest and best ranked graduate education programs published smaller websites with a higher degree of search engine optimization. This meant that resources may have been invested in graduate education program websites to maximize the relationship between search terms and the institutional website, producing not more traffic, but specific traffic from Internet users who were searching for specific webpages. Ultimately, educational researchers should continue to investigate how institutional websites are developed, which strategies institutions are using to drive Internet users to their website, and whether Internet characteristics transcend more traditional U.S. News & World Report ranking criteria, such as student selectivity, alumni giving, and faculty prowess (Morse and Brooks 2017).

Bok (2009) and Slaughter and Rhoades (2004) asserted institutions of higher education have strategically used the Internet to promote institutional brands as a method of competition over faculty members, students, alumni, degree plans, prestigious awards, athletics, and many more facets of an institution. If popular ranking systems such as U.S. News & World Report, Forbes, and Times Higher Education publish yearly rankings which catalyze competition, it seems necessary to begin to consider the Internet as an important institutional mode of competition, one that can now be measured and weighed against other institutions. Here, researchers should investigate how Internet characteristics may influence other ranking criteria, such as peer assessment or funded research. Without further analysis, it is difficult to discern how institutional websites may be influencing other ranking factors.

In no uncertain terms, given the ability to measure Internet characteristics, the analysis of institutional websites and online marketing strategies has special implications for the study of higher education and ranking systems in general.

Conclusion

As Internet technologies continue to advance, educational researchers will be tasked with learning how to measure Internet characteristics and augment extant research focused on the role of academic capitalism in the 21st century institution of higher education. However, the results of this study suggest “bigger is not always better” considering the size of an institutional graduate education program website. From here, researchers should reevaluate how institutions—and graduate education programs—are ranked and how these programs allocate resources to improve their standing within the academic community.

The work of Bastedo and Bowman (2010, 2011) have demonstrated how influential ranking systems can be and how these ranking systems lead to institutional decision making in current and future contexts. As the first study to analyze graduate education program rankings, it is important to expand this study to other fields to learn how ranking systems produce nuanced decision-making strategies and stratification among public and private institutions and programs. As a result, the educational community—and other research communities—will better understand how 21st century technologies can be manipulated and invested into amplify or mitigate the effects of academic capitalism. If “bigger” is indeed not “better” in technological contexts, researchers should investigate this finding and better understand the role of the Internet in an increasingly technological society.