Introduction

Higher education ranking systems (HERSs) are ubiquitous. Initially driven by a desire for global comparisons regarding higher education quality (Dill and Soo 2005), HERS are now also used for classifying, evaluating and holding institutions accountable across an ever-increasing range of issues and concepts (Marginson and Van der Wende 2007; Stolz et al. 2010). HERs now function as a mean of measuring not only perceived quality but also performance (Collyer 2013) and universities expend considerable resources on analysing and promoting specific ranking schemes that they believe will show their institutions in the best light.

Contemporaneously, in an era of mass higher education, governments of nation states increasingly are holding higher education institutions answerable to a wide range of issues. This has given rise to performance measures for accountability/governance, affordability, access and equity (Conner and Rabovsky 2011). Of these, equity in higher education has been a perennial issue for decades and has come under even greater scrutiny internationally, following the adoption of the sustainable development goals (SDGs)—specifically SDG4 relating to the provision of equitable and inclusive education for all—and the Education 2030 Framework for Action in 2015 (Chien and Huebler 2018). In Europe, this has translated into equity being integral to the Bologna Process in the form of ‘social dimension’, subsequently adopted as a goal by the European Higher Education Area (EHEA) (European Commission/EACEA/Eurydice 2015). In Australia, this has seen the inclusion of an ‘equity participation’ measure in proposals for performance-based funding (Wellings et al. 2019).

It is therefore unsurprising that some HERs have moved to incorporate aspects of equity in their rankings’ methodologies. The Washington Monthly College Guide and Rankings (US) is one example of a college ranking system that considers issues of equity explicitly in its calculation. The ranking considers three aspects of equity: Social Mobility (recruiting and graduating low-income students); Research (producing cutting-edge scholarship and PhDs); and Service (encouraging students to give something back to their country) (Washington Monthly 2016). Also in the USA, the Social Mobility Index (SMI) is an explicit effort to shift policy focus away from historical conceptualisations of higher education prestige to encourage institutions to compete around factors which improve access. The SMI considers five main variables: tuition fees (the higher the fees, the lower the ranking); economic background of students; graduation rate; early-career salary outcomes for graduates; and endowment or donations to the college (the higher the endowments, the lower the ranking) (CollegeNet, 2017). In 2017, The Equality of Opportunity Project published its (US) college-level data on the percentage of students from lower-income families who reached higher income quintiles by their early 30s (Equality of Opportunity Project 2017).

However, from a methodological research perspective, other than some broad statements regarding intent, and information outlining what data were collected and how it was incorporated into the subsequent ranking, there has to date been no full-developed, systematic attempt to construct an equity-specific HERS. That is, one which (i) commences with a conceptual consideration of how higher education equity might be defined, (ii) considers what the constituent elements of higher education equity are and how they might be measured and (iii) undertakes a systematic, research-driven approach to constructing a methodology to inform such a ranking. This paper sets out the findings of a research project undertaken in Australia to construct just such a ranking system. The Australian higher education sector is not atypical; it is a mass higher education sector with a mixture of public and private institutions, with a dual focus on quality and equity. As in many other countries, there are explicit social justice policies that aim to increase the representation of students from social groups historically under-represented in higher education. Universally, the concept of equity in higher education is understood and acknowledged and therefore the results from this case study have wider relevance.

Research method

The first stage of the project involved developing a conceptual framework for constructing a theoretical approach to higher education equity. Here, we built upon our earlier work (de-identified) which formulated the following approach. First, a broad series of domains needs to be derived, in which higher education equity could be said to exist or occur. These are a higher-level classification. Second, sources of information relating to higher education equity performance are identified within these domains. These provide data from which indicators (see following) can be generated. Data sources can be quantitative, qualitative or a mixture of both. Third, indicators can be extrapolated from the data sources. Indicators provide the critical means by which some aspect of higher education equity can be measured. All three of these elements—domains, sources and indicators—can be objectively defined and measured. Ultimately however, equity can be applied with different theories of justice in mind and with different understandings of the wider ramifications of the distribution of education (Cameron et al. 2018). Likewise, ranking systems themselves are subjective in many regards: aggregation methods vary, data and their sources have differing degrees of reliability and decisions on what to include and exclude from the final ranking system are significant.

The conceptual framework, and its constituent elements of domains, data sources and indicators, was devised in consultation with key higher education stakeholders, internationally. Stakeholders included higher education researchers and experts with knowledge and understanding of ranking systems and/or equity in higher education; higher education equity practitioners; senior executives in higher education institutions, including those charged with equity policy; and higher education institutional planners. Interviews with stakeholders were conducted in February, March and April of 2018. The interviews were directed by open-ended questions, designed to explore the nature of higher education equity, what actions institutions could take to address it and what evidence they could provide to demonstrate success. In total, 31 stakeholders provided in-depth feedback and this was used to provide both suggestions for, and expert opinion on, potential indicators.

The next stage of the project was to assess each data source and indicator for suitability in a functional HERS. ‘SMARV’ was the project team’s systematic approach to assessment of ranking elements, and is derived from SMART (Specific, Measurable, Accountable, Relevant and Timely)—a mnemonic acronym first used by Doran (1981) for organisational management practice but subsequently evolved and adapted for use in diverse settings. In respect to measuring performance in public and/or not-for-profit organisations, SMART-like objectives can assist in providing a clear definition of tangible results to be accomplished, accompanied by an indication of the specific measures that will be used to evaluate success or failure in achieving them (Poister et al. 2014). Our adaptation of the criteria was focussed on identifying those indicators of higher education institutional practice with the potential to have a positive impact on equity outcomes. In this context, the SMARV acronym stood for indicators that were:

Specific—targeting a particular area through which higher education equity could be improved.

Measurable—the indicator uses data which are reliable and will measure performance against the objective.

Accountable—the indicator relates to a performance measurement where it is possible for the higher education institution to influence the outcome, even if not entirely.

Relevant—the indicator relates to an area of improvement that is relevant to higher education equity.

Value—that which the indicator measures adds value to the final HERS.

The final stage of the project involved developing a methodology for constructing the actual ranking. A weight-and-sum approach was adopted and the rationale for this, arising from the research project, is explicated in the relevant section of this paper. Rankings were generated using data from Australia’s 37 public universities. In 2017, these 37 institutions enrolled nearly 95% of all domestic bachelor-level students in Australian higher education system (Department of Education and Training 2018).

Constructing and assessing a framework for measuring higher education equity performance at the institutional level

In order to rank higher education institutions according to their equity performance, a workable definition of ‘higher education equity’ must be advanced. The concept of equity is connected with principles of ‘justice’ and ‘fairness’ (Raphael 1946), which are socially constructed. Hence, the idea of ‘equity’ has definitional latitude. For example, commutative equity requires a society to treat all individuals equally, whereas distributive equity requires individuals who are disadvantaged to be treated differently, in recognition of their needs. Higher education institutions have at times struggled with the latter approach, due to a belief in the overriding principle of merit; yet research has established that merit-based processes fail to take into account necessary accommodations for those who have faced discrimination and have systemically accumulated disadvantages (Liu 2011). Further, whilst it can be argued that ‘each advance in the participation of persons from under-represented groups is a move forward, regardless of whether the participation of the middle class is also advanced’ (Marginson 2011 p. 35), these may not affect the overall composition of the student population. Consequently, the idea of ‘proportional fairness’ has been examined, where institutions seek to improve equity outcomes relative to the specific context of the institution, its history, student demographics and resources (Pitman 2014).

Across many international jurisdictions and domains, higher education equity is founded upon two basic indicators of success: access and affordability. This is reflected in the United Nations’ International Covenant on Economic, Social and Cultural Rights, which requires that education must be accessible in three respects. First, it should be non-discriminatory. Second, it should be physically accessible. Third, it should be economically accessible (UN Economic and Social Council 1999). Whilst indicators relating to access have been the foundation of higher education equity policy and practice, consideration should also be paid to its other dimensions. In 2009, a report released by the Cabinet Office in the UK observed that access to higher education had not, in and of itself, addressed the continuing inequitable access to professional employment, which remained heavily skewed towards persons from higher socio-economic circumstances (Cabinet Office Strategy Unit 2009). This statement reflects a conceptual need to incorporate additional, ‘downstream’ indicators of success, such as employment outcomes, to the notion of higher education equity. In the USA, publications such as Education Pays by the College Board regularly report the positive correlation between higher education attainment and such outcomes as earnings, social mobility, health factors and civic engagement (The Pell Institute and PennAhead 2015). Therefore, implicit higher education equity aims now extend to a realisation of these, post-graduation socio-economic benefits. At the ‘upstream’ end of the higher education experience, a range of variables influence higher education participation, including educational aspirations, academic preparation and school achievement (Gemici et al. 2014; Perna and Swail 2001). Elsewhere, Boyadjieva and Ilieva-Trichkova (2018) explore higher education equity through the lens of the ‘common good’, encompassing principles of ‘social commitment’ to higher education, along with more traditional indicators of access, availability and affordability.

Interviews with stakeholders indicated a high level of congruency with extending the concept of higher education equity to cover actions and activities that occurred both pre and post higher education. From this broad platform, six domains were identified as being relevant to higher education equity performance measurement. Within these domains, a total of 33 potential indicators were identified. In some cases, multiple data sources were identified for a single indicator and in this instance, each was subjected to the SMARV assessment. This resulted in a final list of five suitable indicators for use in a HERS. Figure 1 shows all 33 indicators assessed, with the 5 selected for inclusion in the HERS highlighted.

Fig. 1
figure 1

:The six domains for measuring higher education equity performance at the institutional level

Full size image

Domain 1: aspiration

Aspiration is a complex construction, encompassing issues of identity, social expectation, preferences, understandings of certain possibilities and the capacity and resources to realise these aspirations (Gale et al. 2013). Research has established that, along with financial costs, a key barrier to higher education participation by low-SES students is a perception by many in the target group that higher education is neither appropriate nor of value to them (Dow et al. 2010).

Universities can play an important role in their local areas and regions in nurturing higher education aspirations for equity-group students. A prime example of this is the provision of outreach activities; most notably aspiration-raising programs targeted senior secondary school students (Austin and Heath 2010; Perna and Swail 2001). Evaluations have established the effectiveness of such activities (e.g., Gale et al. 2010; Hahn et al. 1994). Further, Australian universities receive funding for outreach and aspiration-building programs through a national funding system—the Higher Education Participation and Partnerships Program (HEPPP)—so a ranking system needs to at least consider the extent to which institutions impact on aspiration. However, providing metrics for aspiration faces formidable technical challenges, both in terms of the disentanglement of factors affecting aspiration but also the systems required to collect data on it. Theoretically, it would be possible to collect data in several ways. Longitudinal surveys are a regular feature of social data collection, such as the Longitudinal Surveys of Australian Youth, or the United Kingdom Household Longitudinal Study. Data is also collected by many individual institutions to gauge the effectiveness of their outreach activities. However, in the case of Australia and elsewhere, there is no widespread nor systematic collection of data of the aspirational activities undertaken by Australian higher education institutions, though informal evaluations occur regularly at the individual, programmatic level. Therefore, no reliable indicators for the Aspiration domain were available for this study.

Domain 2: academic preparation

Prior academic achievement is the primary indicator of subsequent academic success (e.g. Gemici et al. 2013). It is an important consideration when exploring equity issues because academic potential has been shown to be influenced by socio-economic factors (e.g. Department of Education Employment and Workplace Relations 2010; Lim et al. 2014). Within secondary schooling systems, tertiary-specific academic programs are sometimes provided. Furthermore, higher education institutions frequently run their own tertiary-preparation programs. The most common are enabling programs, which attract higher than average enrolments from equity-group students (Hodges et al. 2013; Pitman et al. 2016).

A 2011 review found that approximately 50% of students enrolled in all enabling courses were identified as being from several equity groups such as Indigenous students, regional and remote students and low SES status students, compared with 30% of all domestic undergraduate enrolments (Lomax-Smith et al. 2011). Once again however, reliable data that adds value do not yet exist for the domain of Academic Preparation. Therefore, no reliable indicators for academic preparation were available for this study.

Domain 3: access and participation

As identified earlier in this paper, access and participation remains one of the cornerstones of higher education equity policy and practice internationally as well as in Australia. Since the early 1990s, Australian higher education policy formulated at the national level has focussed on relative targets or goals (e.g. Bradley et al. 2008; Department of Employment Education and Training 1990). Although no relative goals currently exist, the underlying principle that higher education participation should aspire towards proportional representation remains in place.

In Australia, there are several, high-quality data sources from which access and participation indicators can be constructed. These include offers to study, acceptances and deferrals of offers and enrolments. However, these indicators are mostly measuring the same elements of access and participation, with minor variances. Therefore, inclusion of one renders the others redundant. Of all indicators, it was determined that the most relevant was enrolment data, where the specific equity group’s enrolment was measured as a proportion of the control student population.

It may also be important to consider not only which institutions students enrol in, but which degrees. In Australia, positive employment outcomes are more highly correlated with the course the student studied, than the institution they graduated from (e.g. Graduate Careers Australia 2014; de-indentfied), with the influence of degrees in STEM (Science, Technology, Engineering and Mathematics) and ‘elite’ courses such as Medicine being particularly noticeable. The examination of access and participation patterns in such courses contextualizes overall trends in participation (see for instance, Bastedo and Gumport 2003), but is problematic in a ranking if higher education institutions vary significantly in terms of overall courses offerings and in some cases, course delivery. Whilst enrolment in elite degrees is an important consideration for higher education equity policy, by definition, an institutional ranking system will not be able to properly capture its influence, compared with a ranking focused on a single type of course and degree (e.g. undergraduate STEM enrolments).

In Australia, a recent study on graduate earnings by Koshy (2016) finds no evidence for an earnings premium among ‘elite’ universities, which they suggest indicates that ‘…these effects have diminished over time due to some form of convergence in the reputations of Australian higher education institutions, perhaps through the movement of newer institutions into traditional course offerings such as Medicine and Law, which account for a significant share of the wage differentials seen between graduates’ (p. 8).

On the basis of these arguments, in this study only an indicator for higher education enrolments was included for the Access and Participation domain.

Domain 4: first-year experience

As the name suggests, the First Year Experience domain relates to ‘surviving’ the first year of study, the criticality of this having been well-established by research (e.g. Luzeckyj et al. 2011; Southgate et al. 2014). The first-year experience encompasses aspects such as establishing a sense of belonging, adjusting to studies, managing finances, succeeding in subject and of course, continuing to the second year and beyond. Many groups of students under-represented in higher education can feel a stronger sense of alienation in the first year, compared to other students (Reay et al. 2009). Support programs and pastoral care are essential for many students, as they struggle to balance personal and work commitments with study, as well as establish a sense of purpose and belonging (Bexley 2008).

Many stakeholders consulted throughout this project reinforced the importance of first-year retention, with some describing it as the most important equity indicator of all. In Australia, retention is defined as any student who commenced a course in 1 year and continued in the next. The Government also calculates success rates, referring to the proportion of units passed in the first year divided by all units attempted. Success is an important consideration for a number of reasons; chief among them being that (a) higher success rates set up a positive feedback loop for the student and improve future study outcomes and (b) higher success rates reduce both the amount of time and cost of studying higher education, by avoiding repeating units.

Another means by which the first-year experience might be assessed is through a national student ‘experience’ survey, in which more than 200,000 students participate annually. However, it was determined that a ranking system would not prioritise student satisfaction over first-year retention and success. Thus, any data collected through the surveys would have little if any effect on the final rankings. Therefore, indicators for first-year retention and first-year success were included for the First-Year Experience domain.

Domain 5: progress during higher education study

In Australia, equity group students regularly report lower levels of completion from higher education studies, compared to the general cohort (e.g. Department of Education 2015; de-identifed). The same pattern is found in other nations’ higher education systems (e.g. Johnes and Taylor 1989; Office for Fair Access 2014; Strayhorn 2010).

The Government regularly conducts cohort analyses to analyse completion rates (Department of Education 2014, 2015) and the associated reports provide valuable data for the higher education sector. Completion can be assessed at multiple points, i.e. 4, 6, 8 or 9 years after the cohort enrolled. There was a majority view in the stakeholder feedback process that completion should be assessed at the further end of the spectrum, i.e. at the 9-year point. This was due to the perception from some stakeholders that certain groups of students take longer to complete and taking an earlier point of comparison would bias against some institutions.

Other aspects of completion were also considered for indicators; namely time to completion; and completed share of ‘elite’ degrees. The major objection to time to completion is that ‘timely’ is a subjective construct (Higher Education Standards Panel 2017). Also, using a measure of student satisfaction was also considered. However, the reservations for using an ‘elite’ degree indicator and/or a student experience indicator in this domain were the same as those expressed for using them in the Access and Participation, and First-Year Experience domains (see above).

Therefore, only an indicator for degree completions was included for the Progress during Higher Education Study domain.

Domain 6: graduate outcomes

A recent study of Australian graduates found most equity group students experienced below-average graduation outcomes, whether in terms of median salary, full-time employment or securing permanent or open-ended contracts (Pitman et al. 2017). Again, the pattern is repeated in other countries (Britton et al. 2016). Further studies in higher education are also an outcome directly attributed to undergraduate success. Overall, these graduate outcomes are shaped by student academic performance, course undertaken and a range of other factors shaping job choices and opportunities (Li and Dockery 2015). In Australia, research suggests the lower level of participation of students from low socio-economic backgrounds in postgraduate education may reflect ‘thin’ undergraduate educational experiences, with a greater proportion enrolled part-time, and in external and multi-modal modes of study (Bell and May 2016). For this reason, graduate outcomes needs to be identified in the broadest sense possible.

Two broad spheres of graduate outcomes were identified: employment and further study. Within these two spheres, it was possible to construct multiple indicators, delineated by factors such as how soon after graduation the measurement was taken; whether graduate was full-time, part-time or ‘under’ employed; whether the job was relevant to the degree studied; salary level; or whether the postgraduate studies were in an elite field. Data for all these indicators were available through two student surveys: one taken less than 12 months after graduation, and the other more than 3 years after graduation. For the less-than 12-month survey, the advantage is that contact details for graduates were relatively current and there tended to be more responses to the survey. For the greater-than 3-year’s survey, the primary advantage was more time for employment outcomes to be realised; however, fewer survey participants responded. For this exercise, the less-than 12-month data was preferred, to improve sample sizes. Furthermore, a relatively broad measure of graduate outcome was preferred, denoted the ‘earning or learning’ indicator. As the name suggests, this indicator measured whether, within 12 months of graduation, the student was either employed, undertaking further studies, or both. Therefore, an indicator for earning-or-learning was included for the Graduate Outcomes domain.

Generally, outcomes measures are favoured by policymakers. However, they require the future development of longitudinal collection instruments to track student progress and outcomes, as noted in the Australian context by Koshy et al. (2016).

Stage 3: developing a methodology for constructing a HERS

The methodology for constructing a HERS has to address three key challenges: (1) defining the equity population, (2) accounting for selection influence and (3) selecting a weighting method.

Defining the equity population

A ranking involves the merging of multiple measures into a single summary measure (Poister et al. 2014). In Australia however, equity status is defined across five distinct groups: (1) students from a low socio-economic background (low-SES), (2) indigenous students, (3) students from regional and remote areas, (4) people with disability (PWD) and (5) students from non-English speaking backgrounds (NESB). This means that important information would be lost in the aggregate. Furthermore, there is the issue of volume. For example, there are over 140,000 low-SES students in higher education, compared to only a little over 7000 students from remote Australia (Department of Education and Training 2018). Equity groups with low, raw numbers, could therefore have a disproportionate effect on the aggregate ranking of the university. Consequently, a composite ranking could allow an institution to hide poor performance in one or more areas, by better achievements elsewhere. Therefore, a separate rank was constructed for each equity group.

Accounting for selection influence

Institutional performance in relation to student equity is shaped by the availability of equity students to a given institution and also trade-offs between inclusive enrolment practices and academic performance. These issues were both present in our analysis. First, in Australia, institutions’ outcomes in relation to access are somewhat determined by their location in a given state or territory, whereby their equity enrolment profile will, in part, reflect the distribution of equity groups in their local population as institutions typically draw students from their local state or territory. Institutions in states and territories with larger equity populations have a natural tendency to enrol higher numbers of equity students (Phillimore and Koshy 2010). Not accounting for this would reduce the valency of the overall ranking. Our solution was to include two measures in the ranking: the Access Rate and the Access Ratio. The Access Rate is the participation rate for the institution, while the Access Ratio adjusts this for the relative population share of the relevant equity group in the institution’s home state or territory.

The second adjustment pertained to remaining indicators. Here, there was a need to balance a comparison between outcomes among equity students with those of the overall student population in their institution and those of equity students elsewhere. Essentially, in Australia as elsewhere, there is an inverse relationship between higher education equity participation and higher education equity progression. Our own research for this project confirmed this relationship. We found that, with a few exceptions, institutions with equity representation between 40 and 50% had retention rates between 75 and 90%, whilst those with representation between 70 and 85% had lower retention rates between 65 and 77%. Prior research has demonstrated the positive correlation between prior academic preparation and tertiary academic performance (Cherastidtham and Norton 2018). To account for this effect, two ratios were included in the ranking for these indicators:

Ratio 1: where the basis of comparison is with other institutions (e.g. ‘how well does University X retain students from Group Y, compared to how other universities retain their students from Group Y?’)

Ratio 2: where the basis of comparison is the overall institutional rate (e.g. ‘how well does University X retain students from Group Y, compared to all other students in University X?’)

The same rationale was applied to the indicators for student success, completion and graduate outcomes.

Selecting a weighting method

Finally, a weighting method had to be adopted to combine the indicator set into a single index number. A weight-and-sum approach was preferred and adopted. The weight-and-sum approach is a well-accepted ranking method (Soh 2017) and is popular due to its transparency and computational simplicity. This approach is the most commonly used for HERS (e.g. QS, 2014; de-identifed). Broadly, the approaches taken to weighting can be classified in two categories: the expert-decided method and the data-decided method (Podinovskii 1994). This project tested both broad approaches.

The expert-decided method, also known as the direct explication method (Zeleny 1982), was operationalised using the opinions of 31 stakeholders, including higher education researchers; experts with knowledge of ranking systems; higher education equity practitioners; senior executives in higher education institutions, especially those charged with equity policy; and higher education institutional planners. Each stakeholder was asked to consider the various elements of higher education equity, across the five domains, and identify, hierarchically, which indicators should be weighted and how. Interestingly, there was a great degree of diversity in this regard, resulting in no clear consensus. Five distinct approaches emerged, which we categorised as follows:

Abstainers: those who believed it was not possible to be prescriptive when measuring equity performance.

Conscientious objectors: those who were opposed to using a HERS to measure equity performance.

Equal weighers: those who did not prioritise one indicator over another.

Participators: those who prioritised access and participation over other indicators.

Retainers/completers: those who prioritised retention and completion over other indicators.

We also examined the data-decided method; notably, a principle component analysis (PCA) approach was adopted. The goal in PCA is to find the relative relationship (represented as weights) between indicators within the index (Nardo et al. 2008). This suited this study, since we were seeking to observe the maximum performance from each institution, although the PCA approach works best when dealing with comparisons across a large number of institutions and therefore the relative small number of Australian universities was a limitation, which would not necessarily be the case in a European or American ranking.

Drawing on the approach outlined in the section above, the specific approach taken for developing the weight-and-sum rankings are listed below. Note that this process was undertaken separately for each of the five equity groups. First, raw performance scores (RPS) for each indicator were developed, using the ratios described above, for each university. Next the RPSs were normalised on a scale of 0 to 100 to generate normalised performance scores (NPS), using the proximity-to-target transformation (Hsu and Zomer 2016; Lewis et al. 1994):

$$ \mathrm{NPS}=\frac{{\mathrm{RPS}}_u-\operatorname{Min}\left({\mathrm{RPS}}_a\right)}{\left(\operatorname{Max}\left({\mathrm{RPS}}_a\right)-\operatorname{Min}\left({\mathrm{RPS}}_a\right)\right)}\times 100 $$
(1)

where RPSu is a university’s raw score for any given measure, RPSa is a vector of all universities raw score for any given measure, Max(RPSa) and Min(RPSa) are the highest and lowest value among universities for any given measure, which are also the targets for best and worst performance respectively in this project. Then, each normalised performance score was multiplied by its final measure weight (weighted importance) to generate a weighted performance scores (WPS). An overall score for each university was calculated by summing the WPSs and finally, universities were ranked based upon the overall score. For the sake of transparency, we do not normalise the WPS. Rather, the reported score is the weighted average of component NPSs (0 to 100). For this reason, the top institution in the WPS ranking will have a score below 100, except where it is the top institution in each underlying NPS.

Multiple weighting formulas were applied, to reflect the three expert-decided weighting formulas (i.e. equal weighers, participators, retainers/completers). A fourth formula was derived from the PCA analysis.

Findings from the rankings process

Below, we outline four ranks, all for the low-SES group of students, the largest equity group in Australia. This group has been chosen to illustrate the findings from our study because the results here were not atypical to those of the other groups. Furthermore, although jurisdictional definitions vary, many nations recognise and target similar groups of students for attention. For this group, four ranks are provided, each illustrating the effect of the four broad approaches to weighting equity performance.

Equal weightings ranking

The equal-weightings approach weighted each indicator identically, meaning participation was considered equal to retention, success, completion and graduate outcomes. Similarly, within-institution and cross-sector comparative performance was weighted equally (see Fig. 2). For the purposes of this study, this rank was considered the ‘baseline rank’. This does not infer that it is the optimal or benchmark rank; however, it provides a useful comparator to illuminate the effect of prioritising one element of equity performance over another.

Fig. 2
figure 2

Low-SES equity performance ranking based on equal weightings, 2018.

Full size image

The baseline rank reveals that, when all things are weighted equally, universities can achieve equity goals in quite different ways. For example, five of the top ten-ranked universities were located in regional Australia or were located in urban locales accommodating larger populations of low-SES persons. These institutions achieved their high rank primarily by enrolling relatively high numbers of low-SES students. The other five universities in the top ten-ranked universities belonged to Australia’s ‘Group of Eight’ (Go8)—relatively old, prestigious, research-intensive institutions, enrolling disproportionate numbers of high-SES students. Historically, the Go8 have significantly underperformed in regard to widening access and participation; however, with this methodological approach, they achieved a high ranking due to their superior performance in the retention, completion and graduate outcomes of their equity students.

Participation ranking

With this rank, the two participation indicators were weighted three-times higher than the other indicators. Consequently, the top ten was dominated by universities with high participation rates of equity students. Regional universities fared well, taking up seven of the top ten positions. Only four universities from the baseline ranking retained a position in the top ten. Generally speaking, universities with high performance in retention, success, completion and graduate outcomes were unable to compensate for the weightings bias in the top half of the ranking. However, in the bottom half, compensation was more possible. For example, despite having a participation rate almost double that of Monash University, Edith Cowan University was ranked nine places lower than Monash. This highlighted one of the issues concerning the use of rankings to measure equity performance: namely that a ranking system does not measure the performance differential between two ranking places and, consequently, may magnify or understate actual performance. Also, due to the absence of a control or benchmark, a ranking system can mask aggregate underperformance (Fig. 3).

Fig. 3
figure 3

Low-SES equity performance ranking based on participation-bias weighting, 2018.

Full size image

Retention/completion ranking

With this rank, the two retention and two completion indicators were each weighted one and a half-times higher than the other indicators. This resulted in rankings somewhat similar to the equal-weighted rankings method, with seven universities appearing in the top ten of both rankings. Regional universities fared worst, with none appearing in the top ten of the Retention/Completion ranking. Six of the top ten positions were occupied by Go8 universities, including all the top five positions. Again, the use of rankings tended sometime to magnify actual performance. For example, Charles Darwin University’s difference in retention rate to Southern Cross University was approximately 3.5%; however, this contributed to it being 12 places lower in the rank, out of 37 universities. Thus, the gap between actual performance was magnified using a ranking system (Fig. 4).

Fig. 4
figure 4

Low-SES equity performance ranking prioritising retention and completion, 2018.

Full size image

Data-decided weighting (PCA)

Using the PCA method, the national retention ratio measure accounted for more than half (52.7%) of the variance in outcomes. More specifically, the PCA approach placed the most significant weight first in retention (national comparison), then completion (national comparison) and then on success (national comparison) (see Fig. 5). While this data-driven approach is interesting, from a policy perspective, it could lead to perverse outcomes, due to the emphasis on highly correlated variables that may not properly reflect the picture of equity intended. As shown in Fig. 6, the upper echelon of the ranking is dominated by Australia’s ‘Group of Eight’ universities. At the other end of the spectrum, the university with the highest proportional enrolments for low-SES students (Central Queensland University) is ranked only 21 out of 37. Consequently, if a PCA or similar approach was used to determine equity performance rankings, then institutions enrolling relatively few, but high-achieving, equity students could be perceived as performing better than institutions enrolling many more, but lower-achieving, equity-group students.

Fig. 5
figure 5

PCA-derived weightings contribution—Low-SES

Full size image
Fig. 6
figure 6

Low-SES equity performance ranking based on PCA analysis, 2018

Full size image

Conclusion

The evidence from this study shows that if policymakers wish to construct a HERS to measure higher education equity, it needs to proceed from a sound conceptual basis. That is, rather than generating rankings from existing data, policymakers need to achieve consensus on what higher education equity means in their jurisdiction, which indicators are required to measure performance and what data need to be collected to inform these indicators. By incorporating a data-driven methodology as one of the options in our study, we revealed how a failure to do this can result in perverse rankings outcomes. Ideally therefore, this requires those constructing the methodology to move from a position of ‘valuing what one can measure’—the underlying basis of many HERS—to ‘measuring what one values’. In the case of higher education equity, this means capturing actions and outcomes that occur before and after higher education studies, such as outreach, academic preparation and transition to employment or further studies.

Even then, using a HERS to measure higher education equity performance is problematized by several factors. Based on the evidence from this study, whilst it may be possible to achieve consensus on the broad dimensions of higher education equity, it is far more difficult to quantify which indicators should be used to measure performance and even further, which indicators should be prioritised over others. Some stakeholders prioritise access and participation, others retention and completion and yet others a neutral position. Each approach significantly affects the final rank. When participation is prioritised, universities enrolling high proportions of equity-group students benefit, even when the subsequent higher education student experience is sub-optimal. Conversely, universities that limit their engagement to a smaller number of relatively high-achieving equity-group students would probably rise higher in a ranking system focussing on retention and completion. Our expert-decided method could not fully resolve these tensions, and while there are alternatives, such as the analytical hierarchy process or AHP suggested by Saaty (1990), these tend to be data heavy. Furthermore, a HERS may distort actual equity performance, especially where the number of universities being ranked is relatively small. A minor shift in performance in just one indicator can result in an institution moving upwards or downwards by an exponential factor. This can give the impression that the difference in performance between two institutions is far greater than it is. Also, in a situation where aggregate performance is sub-standard, a HERS could give the impression that some institutions were achieving excellence when this was not the case.

Our study also revealed that adopting a systematic, theory-driven approach to ranking equity might result—at least initially—in an incomplete and therefore inoperable HERS. This was the case for the Australian higher education sector, where out of 33 potential indicators, only 5 were deemed to be appropriate for use. Consequently, the resulting HERS was not suitable for operation nor informing public policy. However, it was a useful means for exploring how various aspects of equity (e.g. access, retention, graduate outcomes) and differing institutional profiles (e.g. higher access/lower retention or lower access/higher retention) affected the final ranking.

In turn, this may help stakeholders clarify the wider dimensions of higher education equity. Our study shows that there are multiple ways in which a university can achieve higher education equity depending on institutional profile, historical legacy and the overall policy environment. Thus, while a single rankings index has the advantage of focusing stakeholder attention, the complexity of measuring institutional performance in regard to higher education equity may be better understood through the complementary use of a wider range of indicators in respect to specific measures of disadvantage (e.g. Low SES or Regional) and domains (e.g. Access versus Outcomes), to generate a stronger assessment of institutional performance.