Introduction

The ability to generate novel knowledge and integrate it into innovative processes, products, and services plays a crucial role in sustaining socio-economic development within the current knowledge-based economy. Recognizing the pivotal role of research in fostering innovation and growth, various nations have invested in national research systems, making the improvement of their effectiveness and efficiency a top priority in their policy agendas.

A substantial body of literature has examined the scientific competitive standing of nations, highlighting the significance of research performance in the broader context (Aksnes et al., 2017; Allik et al., 2020; Bornmann & Leydesdorff, 2011; Harzing & Giroud, 2014; King, 2004; Li, 2017; May, 1997; Tijssen et al., 2002). Furthermore, the quality of human capital is identified as a vital factor for sustained growth (Aghion and Howitt, 2008; Romer, 1990; Lucas, 1988). Scientific knowledge accumulation contributes to the enhancement of educational and technological capabilities, positioning universities as key entities in the research system. Governments actively pursue the development and strengthening of higher education systems to globally compete for talented individuals and resources.

To assess and enhance the macroeconomic efficiency of research systems, governments conduct periodic evaluations of research institutions, often linking them to performance-based funding. Public funding tied to research performance serves as a competitive mechanism to incentivize continuous improvement in organizational efficiency. While there is a common agreement on the importance of incentive systems to foster research performance, an active debate is still ongoing regarding the methods to assess research performance. Critics of world and national university rankings have long argued against their indicators and methodologies (Billaut et al., 2010; Dehon et al., 2010; Liu & Cheng, 2005; Sauder & Espeland, 2009; Van Raan, 2005).

The dissatisfaction with evaluation methods among stakeholders of research systems has become so strong and widespread that it led the European Commission to promote the establishment of the Coalition for Advancing Research Assessment (CoARA), now consisting of over 600 research institutions, with the aim of reforming research assessment. CoARA envisions research assessment that “recognizes diverse outputs, practices, and activities, maximizing research quality and impact primarily through qualitative judgment supported by the responsible use of quantitative indicators”.

Reactions from evaluative scientometricians were swift (Abramo, 2024; Ioannidis & Maniadis, 2023; Torres-Salinas et al., 2023). Our personal position on the matter is that there is no one-size-fits-all methodological approach to conducting research assessment. The choice of methodology depends on various variables, including the objectives of the evaluation, scale, research disciplines, expected accuracy level, budget, data availability, and last but not least, context. We consider the contribution of scientometrics to research policy and management similar to that of medical imaging diagnostics in clinical medicine. The physician uses the results of imaging diagnostics together with other investigations deemed necessary to formulate a treatment. The same applies to the decision-maker in the field of research. Qualitative judgment can hardly do without quantitative assessment. The work illustrated in this manuscript is a case in point. It is unlikely that a qualitative analysis could provide the same information to the research decision-maker, despite the limitations and assumptions of the bibliometric method employed.

This study focuses on the microeconomic efficiency of research organizations, particularly their discipline portfolio management. Universities are viewed as “multi-business” organizations in the higher education sector, engaging in various scientific disciplines. The challenge for university managers (rectors) is the strategic management of their discipline portfolios, involving decisions on which disciplines to enter, dismiss, and invest in based on competitive standing. The study introduces a methodology to assess whether research organizations specialize in disciplines where they hold a competitive advantage, emphasizing the importance of aligning competitive standing with disciplinary specialization.

The operational methodology involves measuring scientific specialization indices and relative research productivity for each organization in each research field. The study aims to answer questions related to the efficiency of research organizations in choosing disciplines and exploiting their competitive advantages. The Balassa specialization index and fractional scientific strength indicator are applied to assess relative specialization and research productivity.

The application of the methodology is demonstrated using the Italian higher education systemFootnote 1 as a case study. The choice of the Italian case is primarily driven by the availability of input data (cost of labor and capital) and output data (research output disambiguated at the individual level), which we use to measure the research performance of universities. In Italy, 98 universities have the authority to issue legally recognized degrees, with over 90% of faculty employed in public universities largely funded by the government (around 56% of total income). All professors are required to engage in both research and teaching. During the period under investigation, Italy ranked 8th globally in both the number of publications and citations. Italian scholars contributed 15.9% of total EU publications and received 19.3% of total EU citations. Despite a decrease in the number of academics, there has been significant growth in both the number of publications and their scholarly impact (Abramo & D’Angelo, 2023).

The results aim to reveal the efficiency of the discipline portfolio choices at both the organizational and system levels. The methodology can be extended to other countries contingent on data availability, providing valuable insights for university leaders and policymakers responsible for research system efficiency.

While existing literature explores scientific performance and research specialization separately, this study represents a novel attempt to examine the link between the two. The subsequent sections present the methodology and data, showcase the analysis results, and conclude with considerations for future work.

Data and methods

To evaluate the efficacy of organizations in selecting research fields for focused research activities, it is essential to quantify competitive advantage and specialization indexes in each field. These metrics facilitate the ranking of fields based on both indicators at each university, and the degree of similarity between the two rankings can be assessed using the tau-b Kendall correlation statistics (Conover, 1999; Kendall, 1938). This correlation coefficient attains a value of 1 when there is perfect agreement between the two rankings, signifying maximum efficiency in the research organization’s selection of scientific domains for investigation. A correlation coefficient of 0 indicates no correlation between rankings, while a value of − 1 signifies maximum inefficiency, with one ranking being the exact reverse of the other.

To gauge the competitive advantage of a research organization in each field, we compare its research productivity with that of all other observed organizations in that field. We define the productivity of researchers in a field as the output value per euro spent on research. Additionally, we assess relative research field specialization by comparing the organization’s share of research expenditures in the field with that of all organizations.

To conduct these assessments, access to the following information is required: (i) the research staff in each organization; (ii) their classification per research field; (iii) their individual cost; (iv) the cost of resources other than labor devoted to research in each field, and (v) the research output in each field.

In Italy, the MUR maintains a database of university personnel, providing detailed information on each professor, including name, gender, affiliation, field classification, and academic rank at the end of each year.Footnote 2 Professors are classified into 370 scientific disciplinary sectors (SDS) grouped into 14 university disciplinary areas (UDAs).Footnote 3 Data on salary costs for research personnel are available from the DALIA databaseFootnote 4 maintained by the MUR. However, the cost of resources other than labor at the discipline level is scarcely available globally. Still, for this study, we assume it to be similar to data available in NorwayFootnote 5 and invariant across professors.Footnote 6

Publications indexed in the Web of Sciences (WoS) serve as a proxy for the total output of research activities. The bibliometric dataset is obtained from the Italian Observatory of Public Research (ORP), a database developed and maintained by the authors. This dataset is derived under license from WoS, utilizing a complex algorithm for affiliation reconciliation and author disambiguation.Footnote 7

Due to the limited coverage of bibliometric repertories in the arts and humanities and several social science fields, the analysis is confined to STEM disciplines, encompassing 205 SDSs in 10 UDAs. To ensure statistical significance at the field level, only university-SDS pairs with a minimum of five observations (i.e., five professors in the SDS of the university) are considered. For each university-SDS pair, two indicators are calculated: a proxy of research productivity, the Fractional Scientific Strength (FSS), representing relative competitive advantage, and the research field specialization index (SI), which is elucidated in the subsequent subsections. To ensure accuracy in impact measurement, a minimum two-year citation window is allowed, and the observation period spans from 2015 to 2019, with citations counted as of December 31, 2021.

Measuring the competitive advantage of universities

We consider research laboratories as productive entities with production factors consisting of i) researchers (L); tangible resources (such as scientific instruments, materials, etc.), and intangible resources (like prior knowledge, social networks, etc.) (K). Researchers generate knowledge, which is documented in publications (Q) to facilitate its dissemination. The value of publications varies based on their impact on future scientific advancements, commonly referred to by bibliometricians as scholarly impact, measured through citation-based metrics. Productivity, a key indicator of the efficiency of any production system, is operationalized through several simplifications and assumptions. Initially, scientific productivity is measured at the individual level using the FSS,Footnote 8 defined as:

$${FSS}_{p}= \frac{1}{\left(\frac{w}{2}+k\right)}\bullet \frac{1}{t}\sum_{i=1}^{N}{c}_{i} {f}_{i}$$
(1)

where:

w = average yearly salary of the professor (we halve labor costs, assuming that 50 percent of professors’ time is allocated to activities other than research).

k = average yearly capital available for research to the professor.

t = number of years of work by the professor in the period under observation.

N = number of publications by the professor in the period under observation.

\({c}_{i}\)= impact of publication i (weighted average of the field-normalized citations received by publication i and the field-normalized impact factor of the hosting journal)Footnote 9;

\({f}_{i}\)= fractional contribution of professor to publication i.

As for the cost of labor, w, data concerning salary for research personnel were obtained from the DALIA database,Footnote 10 which is also maintained by the MUR. As for the cost of capital, k, we relied on Abramo et al. (2020).Footnote 11

The productivity of universities, which are heterogeneous in the research fields of their staff, cannot be directly measured at the aggregate level. So, after measuring the productivity of individual professors (Eq. 1) we normalize individual productivity by the average of the relevant field (SDS). At the aggregate level then, the yearly productivity FSSA for the aggregate unit A (SDS, UDA, Department, etc.) is:

$${FSS}_{A}= \frac{ 1}{RS}\sum_{j=1}^{RS}\frac{{FSS}_{{P}_{j}}}{\overline{{FSS }_{P}}}$$
(2)

where:

\(RS\)= number of professors in the unit, in the observed period;

\({FSS}_{{P}_{j}}\)= productivity of professor j in the unit;

\(\overline{{FSS }_{P}}\)= average productivity of all productive professors under observation in the same SDSs of professor j.

A value of \({FSS}_{A}=1.20\) means that the university’s unit A employs researchers with average productivity of 20% higher than expected.

In this way, we can measure the productivity of the university at SDS, UDA, and the overall level.

Measuring the research field specialization of universities

We draw from international trade theory, adapting the concept and measure of production specialization to our context. To assess the universities’ degrees of specialization in each field, we use the Balassa index (Balassa, 1965). It shows whether a university specializes in a specific field relative to other universities. Named PFTC the total cost of the production factors L and K, employed by the university i in SDS j, in the observation period:

$${PFTC}_{ji}=\sum_{z=1}^{{M}_{j}}{t}_{z}\left(\frac{{\text{w}}_{z}}{2}+{k}_{z}\right)$$
(3)

where Mj is the number of professors of university i in SDS j; and \({t}_{z}\) the number of years on staff of professor z in the observation period, the specialization index \({SI}_{ji}\) of university i, in the SDS j is:

$${SI}_{ji}=\frac{{PFTC}_{ji}}{{\sum }_{j}{PFTC}_{ji}}/\frac{{\sum }_{i}{PFTC}_{ji}}{{\sum }_{j}{\sum }_{i}{PFTC}_{ji}}$$
(4)

The higher the value of \({SI}_{ji}\) compared to one, the more specialized the university i is in SDS j. If \({SI}_{ji}\) is less than one, it means that no specialization is involved in j for university i. At a more aggregate level of UDA, the index can be easily calculated by applying Eq. 4 with j referring to UDA instead of SDS.

Results

In order to answer the research questions, we measure for each university the degree of similarity of the SDS rankings where the university does research for the two indicators just described. For instance, we present the case of the University of Rome “Sapienza,” the largest nationally (and in Europe) with over 3500 professors on staff on 31/12/2021. In this study, the analysis dataset is limited to STEMs, where we count 2905 professors on staff during the observation period, in 180 different SDSs.

Figure 1 shows the scatterplot of the 147 SDSs with at least five professors on staff. The two indicators show values greater than unity in only 22 SDSs (accounting for 15 percent of the total). In 48 SDSs (32.7 percent), both indicators are below 1, but the most crowded quadrant is the second one, with 60 SDSs showing values of FSS less than unity and SI greater than unity simultaneously. The two rankings show virtually zero correlation (Kendall’s tau-b = − 0.025), allowing us to state that at the overall level, this university does not specialize in research fields where it holds a competitive advantage.

Fig. 1
figure 1

FSS and SI distributions for 147 SDSs of University of Rome “Sapienza”

Full size image

Repeating the analysis for all universities in the dataset, we obtain what is shown in Table 1. Kendall’s correlation coefficient is positive and significant only for the University of Sannio (tau-b = 0.800, calculated for only 5 SDSs). In particular, this university shows very high productivity (FSS) in ING-INF/05 (Information processing systems), ranking at 86th national percentile, and very low in FIS/01 (Experimental Physics) ranking at 17th national percentile. At the same time, the specialisation index recorded for the first SDS is the highest among the five active in the university, while for the second it is the lowest.

Table 1 FSS and SI correlation, at university-level
Full size table

At the bottom of the list of Table 1, we find the University of Urbino “Carlo Bo”: the correlation recorded for the values of the two variables measured on its 13 SDSs is negative and significant (tau-b = − 0.487). In this University, FIS/01 (Experimental Physics) is the top ranked SDS by productivity (top at national level by FSS) but the bottom ranked, among the 13 active SDSs in the university, by SI. Conversely, the University registers the worst FSS performance in GEO/05 (Applied geology), an SDS with the second highest SI (7.261), immediately after GEO/02 (Stratigraphic and sedimentary geology) registering a value of SI equal to 8.057.

Overall, 30 universities show positive correlation coefficients (in no case significant, apart from Sannio) against the 31 universities that show a negative value. However, in no case it is significant apart from the University of Urbino, the University of Calabria (tau-b = − 0.200), and Cattolica del Sacro Cuore (tau-b = − 0.208). In these three cases, we can say that these universities concentrate their research activity in fields with relatively modest performance. The middle zone of the distribution is very dense, with 35 universities (almost half of the total 63), and a correlation coefficient within the range (− 0.1; + 0.1). We deduce that, apart from a single university, Italian universities do not specialize in fields where they hold a competitive advantage. Scrolling the top of Table 1, we notice the presence of “Scuole Superiori” which are known to be exceptionally brilliant in terms of scientific performance. We then ask whether there is a correlation between “selection efficiency” and research productivity. In other words, whether the best-performing universities are also the most efficient in choosing fields in which to focus their research activities. The scatterplot in Fig. 2 reports the position of each university by overall productivity (FSS) and degree of similarity for ranking (FSS-SI Kendall correlation). To assess the FSS-degree of similarity ranking correlation, we apply the Somers’ D statistics, which reveal no correlation (Somers’ D index = 0.0036, P >|z|= 0.972).

Fig. 2
figure 2

Scatterplot of overall productivity vs FSS-SI correlation for universities in the dataset

Full size image

It would thus seem that not even the most productive universities are particularly careful in choosing those research fields in which they can exploit their competitive advantage. Except for only one university, among others characterized by a low number of observations (5 SDSs only), there is no correlation between the fields in which universities specialize and the relative research productivity; indeed, in some cases there is an inverse correlation. However, there may “locally” be fields in which this occurs. To test it, we consider fields rather than universities as the unit of analysis. The question we then ask is: which fields are characterized by greater (lesser) selection efficiency?

At the operational level, for each university, we sort the SDSs in which it is active by FSS and SI and measure the respective percentiles (100 = top).Footnote 12 An SDS with a 90 percentile by FSS means that only 10% of the SDSs in which the university is active perform better. Similarly, an SDS with the 90 percentile by SI means that it is more specialized in only 10% of the SDSs in which the university is active. At this point, for each SDS, we can construct a scatterplot by the above two percentile rankings of all universities active in that SDS and apply correlation statistics.

For instance, we report the case of Applied Technological Pharmaceutics (CHIM/09), in which 24 universities (with at least 5 professors) conduct research. Figure 3 reports the scatterplot of the data and shows a weak but significant correlation, with Kendall’s tau-b values of 0.294 (Prob >|t|= 0.0471). There are eleven universities in the first quadrant, with both FSS and SI percentiles at least equal to 50. In the third quadrant, there are six universities with both indicators below the 50th percentile. In the second quadrant, the position of the University of Salerno stands out, showing an FSS percentile of 24.3 compared with an SI percentile of 73.0. In the fourth quadrant, on the other hand, we have the opposite anomaly, that of the University of Florence, which in this SDS shows relative productivity at the top 3 percentile (FSS percentile of 97.0) against, however, a very low specialization percentile of 13.9. Conversely, in Biochemistry (BIO/10) the correspondence between the two dimensions is much less evident. As shown in Fig. 4, in this SDS, only four out of 44 universities are positioned in the first quadrant. The set of universities (16 in all) is far more numerous in the third quadrant. Of the remaining 24, 21 universities are in the fourth quadrant, characterized by a high relative value of performance (FSS percentile greater than 50) and low relative value of specialization (FSS percentile greater than 50), resulting in an overall correlation value close to zero in fact (Kendall’ tau-b = − 0.074).

Fig. 3
figure 3

FSS and SI percentiles distributions for 24 universities in CHIM/09, Applied Technological Pharmaceutics

Full size image
Fig. 4
figure 4

FSS and SI percentiles distribution for 44 universities in BIO/10, Biochemistry

Full size image

Repeating the analysis for all SDSs under observation yields the data shown in Table 2. For the sake of significance, we limit the analysis to SDSs (173 out of the total 205) with at least five observations, i.e., at least five national universities with at least five professors in the SDS under consideration from time to time. Overall, in only 12 SDSs (accounting for 7% of the total), we obtain positive and significant correlation values (Kendall tau-b): 4 SDSs are in the Agricultural and veterinary sciences and 3 in Chemistry. At the same time, the data indicate 7 SDSs in which the correlation is significant but negative, mainly in the Medicine area.

Table 2 Number of SDSs showing a significant (p-value < 0.1) Kendall tau-b correlation between FSS percentile and SI percentile, by UDA
Full size table

Discussion and conclusions

Because of its abstract nature, knowledge evaluation is challenging for scholars, practitioners, research managers, and policymakers. Over time, bibliometricians have tried to propose, apply, validate and improve indicators and approaches to identify the strengths and weaknesses of research systems at macro (country), meso (institutions), and micro (individuals) levels. There is a particular interest in how institutions and countries perform in scientific disciplines and what determinants explain why a country presents a specific competitive advantage in one field over another (Braun et al., 1995; Horta & Veloso, 2007; King, 2004; Kozlowski et al., 1999).

In this study, we investigated a complementary aspect of the microeconomic efficiency of research organizations and the macroeconomic efficiency of research systems i.e., whether research institutions specialize in scientific domains where they hold a competitive advantage. We measured the scientific specialization index of each Italian university and their research productivity in each field. Measuring research productivity (defined as an output-to-input ratio) is a formidable task because of the lack of input data. Benefiting from Italian structural advantages concerning input metadata availability, we have operationalized the measurement of a proxy of productivity unparalleled worldwide.

We applied Kendall’s statistics to assess the degree of similarity between the rankings by research productivity and specialization index of Italian universities in each research field. Findings reveal that, with only one exception, Italian universities do not specialize in fields where they hold a competitive advantage. In particular, the data show the presence of three universities that concentrate research activity in fields where they even have a relatively modest performance.

This result, in part, anticipates the answer to the second research question we initially posed, whether the organizations best at doing research were the ones best at selecting the fields to concentrate their research activities. The answer is negative: the analysis finds a complete absence of correlation between the overall productivity of universities and their ability to concentrate in the fields in which they are best at doing research.

The final in-depth study to answer the third research question aimed to identify the fields in which the most productive organizations concentrate their research activities. As was to be expected, even in this case, few positive exceptions emerged from a rather apparent general phenomenon: in only 12 fields out of the 173 analyzed, there is, in fact, a significant positive correlation between the productivity of universities and their degree of specialization in the field. These fields fall mainly in two disciplines, Agricultural, and veterinary sciences and Chemistry. In seven other fields, the correlation is significant but negative, indicating the paradox of a greater concentration of research by lower-performing universities.

In interpreting the main findings of the analysis, it should not be forgotten that universities play the primary role in higher education in addition to research. This aspect implies a necessary diversification in research activity. Delivering degrees in, e.g., engineering implies giving courses and hiring relevant professors in mathematics, physics, and others. In an efficient research system, one expects that universities excelling in engineering research, while performing low in mathematics, deliver degrees in engineering, leaving to others those in mathematics, but not the other way around. The evidence from the analysis is both suggestive and counterintuitive. However, this evidence may find an explanation considering the peculiarities of the Italian academe, which was for years a scarcely competitive higher education system.

It was only in 2009, following the first national research evaluation exercise, that Italy began allocating a small portion of its public funding to universities based on research performance. Initially, this was around 7%, but law 98/2013 set a minimum share of 16% for 2014, with a mandated annual increase of 2% up to a maximum of 30%. Despite this, the system remains somewhat erratic. While the funding is awarded based on the average research performance of individual professors, the financial rewards are given to the universities, which are not required to distribute the money according to individual performance, that is not communicated to them. This lack of obligation or specific incentives allows universities to hire or promote professors in fields where they may not necessarily excel.

The reasons underlying the revealed low efficiency in the choice of disciplines to concentrate research in, are partly to be found in the interplay of the management culture that has dominated for years in the academia, and Italian labor laws. Universities have developed in a non-competitive environment, whereby public funds were allocated to them on the basis of size and type of disciplines, and professors’ salaries were (and still are) not linked to performance. Rectors are elected by both academic and non-academic staff, fostering please-all management practices to assure re-election. Resources for recruitment have been allocated internally to the various departments more on the basis of their negotiation power than inspired by the principle of efficiency (Civera, D’Adda, Meoli, Paleari, 2022). Furthermore, the effectiveness of recruitment and career progress has been undermined by amply diffused favoritism practices which often prevail on merit-based selection (Abramo et al., 2014a, 2014b, 2015; Durante, Labartino, Perotti, 2011; Gerosa, 2001; Zagaria, 2007). In general, non-competitive environments do not favor the application of management science theory in running organizations. We then doubt the mastering of business portfolio management techniques among the government bodies of Italian universities.

Even where the willingness to change the bad practices of the past where there, following the university performance-based funding recently introduced by the government, good-willing rectors find the current labor law a formidable obstacle to dismissing inefficient professors, and pursuing discipline portfolio efficiency. While possible in theory it reveals hardly realizable in practice.

Inefficient diversification strategies by universities translate into inefficient research systems at national level. The recent introduction of performance-based funding linked to the national research assessment exercises has been an important initial step by the government toward the strengthening of a competitive environment, a harbinger of continuous improvement along the knowledge production dimension. Additional incentives are needed to stimulate efficiency also along the discipline portfolio management dimension. We interpret the recent introduction of extra-financial rewarding for “excellent” university departments as assessed by the national research assessment exercises, an important step by the government toward this direction. The government’s direct allocation of resources to the best performing disciplinary departments in each university, might counterbalance the internal political power influence in determining the disciplinary areas in which to recruit scientific personnel. A concern remains though about the VQR methodological failures in assessing research performance and, consequently, about the correct ranking of university departments (Abramo & D’Angelo, 2015).

An additional intervention that we deem effective would be the communication to each university of their diversification strategy efficiency, and recommend that the government allocation of resources be based not only on the efficiency of production but also of discipline portfolio management.

We conclude the work by reminding the usual limits and assumptions embedded in all bibliometric approaches. Firstly, the knowledge generated may not always be reflected in publications, and bibliographic databases like WoS, utilized in this study, may not encompass all published works. Secondly, assessing the impact of publications through citation-based metrics is a predictive rather than definitive measure, and citations only verify scholarly impact while overlooking other forms of impact. Thirdly, we do not account for variations in capital available to individual researchers. Finally, the results could be impacted by the classification schemes used for publications and professors. These constraints highlight the necessity for caution when interpreting data obtained from scientometric methods. However, we do not expect these limitations to disproportionately affect any particular Italian university, thereby preserving the reliability of the study’s findings.