Introduction

Nowadays the great interest is devoted to the global educational system, so there is a strong competition in this field between universities. In order to structure and estimate the performance of world universities the leading ranking agencies develop and constantly improve their methodology for university ranking. This increased interest to the positioning in the global university ranking make governments of the several countries develop and implement various programs in order to increase their universities competitiveness around the world. For example, Japanese government launched “Super Global Universities” program in 2014, “Brain Korea”, China’s 211 and 985 plans and Russian “5–top 100” project. Unless all of the programs mentioned above have got different length, all of these governmental initiatives could help ensure that the country will become attractive for the world’s knowledge production and innovation. In this paper we will focus at the 1 particular program “5–top 100”, not all programs like it.

Globalization of the economy gave rise to the new phenomenon for the Russian higher education system, which noticeable effect has appeared recently. This phenomenon is “university rankings”. The internationalization of higher education has led to the need to compare universities in different countries by a variety of indicators. Previously Russia did not pay much attention to the university rankings, because there was no need to do that.

The government program for increasing university competitiveness “5–top 100” was launched in 2013 in accordance with the Decree of the Government of Russian Federation No. 211 (Decree of the Government of Russian Federation No. 211 2013) and the Presidential Decree of the Russian Federation No 599 “On measures of implementing the state policy in the sphere of education and science” (Decree of the President of Russian Federation No. 599 2012). The projected duration of the program is 7 years. The name of the program clearly reflects its main aim: “to maximize the competitive position of the selected group of leading universities in the global market of educational services and research programs” (Program “5–top 100” official site 2015). The program ‘5–top 100” supports 15 best universities carefully selected from 54 universities in Russia, “with a desire to see at least 5 of them enter the top 100 of the leading global university rankings by 2020” (Kremer 2003). The program “5–top 100” is one of the most ambitious programs because it aims at changing university environment, enabling and supporting strong international collaboration in order to increase Russian university attractiveness around the world, tight timeframes, funding amount, and strategy plan. We would like to mention that program “5–top 100” is not the only government program in Russia for supporting its educational and research sector.

There were other government programs for strengthening the university research: for example, ‘national research universities” program (launched in 2008) that assume competition between universities (mainly in applied sciences). This program helped universities to obtain the modern laboratory equipment and improve the quality of its applied researches. A year later the other government program (creation the 5 “federal universities” on the basis of the regional educational and research center) was launched. The brief review of these government programs can be found in (Schiermeier 2010).

The Russian education system is well known for its focusing on the fundamental science. The program “5–top 100” is designed to incorporate the tradition of strong fundamental science with the newest tendencies in international higher education. Also, there is one feature that is specific to the Russian academic world, where “both faculty and administrators enjoy… their work” (Lukman et al. 2010).

The amount of funding available for the Program “5–top 100” is about 1 bln EUR up to 2016. The participating universities differ from each other according to their geographical region, specialization, competence, number of students, involvement in joint international research projects. The science in Russia was historically concentrated around the Russian (earlier Soviet Union) Academy of Science represented by various institutions, so the main science centers were Moscow, Saint Petersburg, Novosibirsk and Vladivostok. Nowadays the program “5–top 100” supports universities from different regions to develop the university of new format and implement the concept “research institution”. After each year of participation, universities report to the coordination center of the program “5–top 100” on their results and plans for further development. The following year funding amount directly depends on the previous year results. The results include the following key performance indicators:

  • position in the world rankings (general and major subject rankings);

  • number of publications of all types in the Web of Science and Scopus databases per faculty (excluding duplicate items) for three preceding years, including the reporting year;

  • average citation rate per faculty derived from the total number of Web of Science and Scopus publications for 5 previous years, including the reporting year;

  • percentage of international academic staff members among the total academic staff members, including Russian citizens with PhDs from foreign universities;

  • percentage of international students among total enrollment;

  • Unified State Exam average score (graduates admitted for full-time programs: bachelor and specialist);

  • share of revenue from extra-budgetary activities in the total income.

The objective of this paper is to evaluate the effect of the program ‘5–top 100” on the publication activity of the Russian universities. So our null-hypothesis H 0 was as following (we divided it into 3 parts for clarity):

  1. 1.

    the program “5–top 100” influences significantly not only the publication activity of the participating universities in Russia;

  2. 2.

    the results began to improve before the program launch as result of the expectation theory;

  3. 3.

    the best results (both quality and quantity results) are shown by Focused group universities as these universities.

As the program “5–top 100” launched many participating universities begun develop the publications supporting programs in order to stimulate their publication activities. Most of the universities (for example, Novosibirsk State university, Kazan federal University etc.) offer the researchers the linguistic and other support services. But other universities (for example, Moscow institute of physics and technology) proposed and introduced the other type of publications supporting program: the authors of publication are rewarded depending the number of authors, the journal impact factor that this paper was published in and the “threshold” impact factor that characterizes the research area of the paper. We believe that these publications supporting programs helps universities to achieve their goals regarding publication output growth and average NCI growth, because the researchers had got an opportunity and motivation to publish their papers in the journals with higher impact factor than previously, so the university as a whole gains more scores in the ranking for its publications.

Given paper does not aim at the consideration the relationship between the finance support provided by the program “5–top 100” and the universities results in the world rankings. The top world rankings use the more complicated methodology than the given paper. The world rankings methodology uses not only citation data (but citation data has the significant weight), but also the normalized parameters reflected the universities activity. So we can say that our task is more narrow. But the comprehensive analysis contains the conclusions on the influence of the program “5–top 100” on the universities results in the top world rankings will be very interesting. We would like to add, that most of the program “5–top 100” participants are ranked in the top world rankings. Among Russian universities ranked in the top world rankings there are more of the program “5–top 100” participants than not participants (most of them are in the reference group in the given study).

This paper contains 3 Sections: Methods, Main results, and Discussion. Introduction open the paper. First section describes the methods applied for the analysis and other important notes. Second section is devoted to the presentation of main results. In third section, we discuss results obtained and present several ideas on the further research development. The Conclusion with reference list completes the paper.

Methods

We used only the open resources and databases available at subscription. The quantitative information was extracted from the SciVal database (SciVal database 2015). SciVal is one of the Elsevier Research Intelligence solutions, that helps researchers and research managers analyze the results of the university and and research groups, evaluate research strategy and performance. SciVal is analytical tool based on the Scopus publication database.

Time period—from 2008 to 2014 (5 years before the program and 1 year of the program operation).

We analyzed the publication activity of the selected universities as their results. The main indicators are supposed to be:

  • number of publication (all number of publications and number of articles separately). Type “publications” includes all types of publications according to the SciVal, type “article” corresponds to the appropriate type in SciVal;

  • field-weighted citation impact (here and after FWCI) for all publications and for articles only;

  • publication share in top-10 % of highly-cited publications in the world (not subject area-specific);

  • publication share in top-10 % most reputational journals (not subject area-specific) according to the SJR classification.

FWCI is a dimensionless value normalized for science field, so it is widely used for analyzing the quality of publications. This indicator is meaningful itself and does not depend on the year. The world mean value of FWCI is 1 in any year in any research field (so the indicator is called “field-weighted”). In contrast, the number of publications (scholarly output) depends dramatically on the university size, university competence, and specialization. Focused and Comprehensive universities usually have the higher values of FWCI than other university categories, but Comprehensive + universities publish more papers due to their size and wide range of competencies. Thus the following indicators were analyzed by university groups: total number of publications (scholarly output), number of articles (scholarly output: articles only), FWCI for all publications, and FWCI for articles only. We analyze the gross values for the university publication output, because the leading ranking agencies take into account gross values estimating the university output in qualitative terms.

For the number of publications we calculated three growth rates in order to compare dynamics according to different periods:

  • before the program launch \(\sqrt[4]{{\frac{{{\text{Value}}_{2012} }}{{{\text{Value}}_{2008} }}}} - 1;\)

  • “transitional” rate \(\frac{{{\text{Value}}_{2013} }}{{{\text{Value}}_{2012} }} - 1;\)

  • growth rate within the first year of the program \(\frac{{{\text{Value}}_{2014} }}{{{\text{Value}}_{2013} }} - 1.\)

For the period before the program launch we calculate the average growth rate using the concept of geometric mean (compound annual growth rate). This indicator is widely used in investment analysis for calculating growth rates, more information can be easily found in (Anson et al. 2011).

We divide all universities into groups according to QS methodology. Originally QS methodology proposes the method of dividing of all universities according to the Faculty Area: “four categories based on the institution provision of programs in the five broad faculty areas used in the university rankings. Due to radically different publication habits and patterns in medicine, an additional category is added based on whether the subject institution has a medical school” (QS methodology 2015). The basis for dividing universities into groups is presented in Table 1.

Table 1 Groups of universities (QS methodology 2015)
Full size table

All participating universities were assigned to special groups in accordance with their specialization narrowness. We assign the category to each university analyzed as it is indicated at official QS website (see Table 2).

Table 2 Grouped universities that participate in “5–top 100” program
Full size table

We would like to comment on this split method: as the only university—Novosibirsk State University—originally should be attributed to the Full Comprehensive group due to QS methodology, we decided to attribute it to the Comprehensive group. This change is necessary in order to make groups representative of the university of the similar competence. Hence we analyzed only Focused and Comprehensive universities, Specialist and Full Comprehensive were not in our sample.

There are two heterogeneous groups Reference group and “5–100” group. The universities for Reference group were selected according to the following criteria: Russian universities not participating in the program “5–top 100”, ranked under the QS BRICS 2014, and having the comparable (with universities participating in “5–top 100”) number of publications within the time period of interest (The group “5 -100” comprises all 15 universities participating in Program “5–top 100”).

We compared the group characteristics across different groups as well as with country and world means for better understanding the university group position in the science world. For example, if we obtained FWCI of Focused group universities being higher than the Reference group, Comprehensive group and world mean, we could say that universities from Focused group have ranked first.

Also the interval estimate analysis was performed and it turns out that ranges for “5–top 100’ group is usually lower than for each of the residual group. The ranges for each of the group before the program were relatively low, then in the transitional period the ranges rapidly increased and in the first year of the program ranges decreased, but not to the initial level. One can explain this phenomena by deep modernization processes in the universities before the program and something like fluctuation adjusting processes in the first year of the program. Later we do not lead the dispersion ranges in order to not confuse the reader. The dispersion ranges calculated are relatively broad, and the ranges for different groups overlap, but the mean values provided are different. It means that the staff activity of the universities from the different groups was similar at the all considered time periods, especially at the time period 2008–2012. So we can say that before the program launch the activity of universities was similar.

Main results

Scholarly output

Firstly, we thoroughly analyzed the growth dynamics of number of publications for selected groups of universities. Secondly, we calculated average annual growth rate for the period 2008–2012 before the program “5–top 100”, growth rate for “transitional year” (2012–2013) and growth rate for the first year of program. Tables 3 and 4 illustrate these generally expected results.

Table 3 Scholarly output: all publications
Full size table
Table 4 Scholarly output: articles only
Full size table

One can easily notice that before launching the “5–top 100” program the average annual growth rates of publication of all types of universities was at the moderate level: the leader was Focused group with 27 %, the following group was Comprehensive group with 16 %, Reference Group had only 7 % of growth rate. All growth rates were above the world and country mean values. The absolute number of publications for Comprehensive group is much greater than Focused, whereas the growth rates are substantially less.

If we look at the trends of all groups, we may state that there is a positive growth rate of publications throughout the whole time-period of interest except the world mean. This tendency can be easily explained: before the program launch, universities demonstrated their internal natural growth rate (higher than country and world means), and within the transition year participating universities were involved into competitive race and made many efforts to improve their results. The excess of all publication growth rate 2013–2014 for Comprehensive group universities was in 1.26 times over Focused group and really in 7.9 times over Reference group and did not turn out to be a very surprising result. It should be mentioned that within the first year of the program “5–top 100”, the results of participating universities increased much more significantly than Reference group results and country and world means.

The dynamic of publication growth by years of the university rates is presented in the Figs. 1 and 2.

Fig. 1
figure 1

Scholarly output (all publications) dynamic

Full size image
Fig. 2
figure 2

(SciVal database 2015)

Scholarly output (all publications) trend

Full size image

Table 4 illustrates the growth rates of the number of articles only (without reviews, conference papers, notes etc.). The main tendencies discussed in the paragraph above are the same, but absolute values of the growth rates are less among all groups of universities. We may assume that universities can go two ways: increase the article and review numbers (articles and reviews are considered to have the highest quality among all publication types) or for example conference papers as the easiest way of accelerating publication growth. Based on these assumptions we may conclude that universities from Comprehensive group reached the high growth rates of all publications by increasing the number of not only articles.

Other not quite usual result associated with results of Reference group of universities is that there is a point of inflection of the articles quantity growth rate in 2013: the growth rate increased from 4 % in 2008 to 20 % in 2013 and then fell to 10 % in 2014. Country and world means show the same tendency.

Figures 3 and 4 represent the dynamics of the number of articles published each year for all groups.

Fig. 3
figure 3

Scholarly output (articles only) dynamic

Full size image
Fig. 4
figure 4

(SciVal database 2015)

Scholarly output (articles only) trend

Full size image

Field-weighted citation impact (FWCI)

FWCI is an indicator of research quality since this indicator is dimensionless and normalized for the research area. FWCI enables us to compare publications from different fields and aggregate research quality of the university. We estimated universities at the competence group level as described in part 1.

Figures 5 and 6 present results of the analysis of dynamics for FWCI of all publication types (Fig. 5) and FWCI for articles only (Fig. 6).

Fig. 5
figure 5

FWCI: all publications

Full size image
Fig. 6
figure 6

Dynamics of FWCI of research papers

Full size image

We can understand from Fig. 5 that FWCI (all publications) steadily increases for Comprehensive and Focused groups with FWCI for Focused group prevailing over Comprehensive group. Reference group demonstrates unusual dynamics near the point 0.6. All Universities participating in Program “5–top 100” have possessed the higher FWCI rate than country mean since 2011.

Figure 6 presents the dynamics of FWCI for articles only. Focused group published articles with higher FWCI than Comprehensive and Reference groups. All analyzed groups have demonstrated better results than country mean since 2011. The group “5–100” had FWCI (articles only) in 2014 higher than the world mean (=1). The Reference group’s FWCI is constant and fluctuating around the value 0.4.

The FWCI for articles only is more reliable indicator than FWCI for all publications because it is unlikely to reflect statistical outliers. FWCI for articles only demonstrates the trend of quality increasing year by year. The world mean is given for meaningful comparison and shows the ideal FWCI. The growth rates of FWCI by groups are different and can be compared due to the variance in university competencies.

High-quality publications (top-10 %)

In order to assess the quality of publications by competency groups we analyzed two more indicators: the share of all publications of the group in top-10 % most highly-cited publications around the world (Fig. 7) and in top-10 % most reputational journals (by SJR, Fig. 8). Y-axis in Figs. 7 and 8 is percentage share from all group publications.

Fig. 7
figure 7

Fraction of publications in the world top-10 % highly-cited publications

Full size image
Fig. 8
figure 8

Fraction of publications in the world top-10 % most reputational journals

Full size image

The analysis presented in Fig. 7 supports our assumptions. Focused group of universities has the higher share of highly-cited papers due to the participation in mega-collaborations. In 2014, this indicator for Focused group became equal to Comprehensive’s one and country mean owing to science globalization.

The same conclusions may be derived from the analysis of publication percentage share in top-10 % most reputational journals. Focused group still demonstrates better results comparing to Comprehensive and Reference groups. Over the time-period considered the positive trend is shown by Group “5–100”, but the world mean is significantly better.

Discussion

We have analyzed only two indicators of publication activity in time: number of publications and FWCI. However, many other indicators as well as complex indicators can be used. For example, the percentage of publications of particular university in top-1 % instead of top-10 % of the most highly-cited papers in the world may be used as the indicator of paper quality. As well as natural indicators of quality, international collaboration impact by science area can be used if we assume that science is a very cooperative area. Such indicators reflect knowledge transfer, so only collaborations of top-scientists can conduct research of high quality.

In this paper, we consider only number of publications and FWCI to be the “red-flags” of quality and quantity of scientific research, because these indicators are clearly understandable and are directly influenced by government increasing competitiveness program.

As we mentioned in part 2, universities may choose different ways in order to meet the program “5–top 100” requirements and obtain funding for the following year in the number of ways:

  1. 1.

    Increasing quality of research projects by inviting leading scientists in the field of university competence, recruiting high-quality academic and research staff (including international), founding new and perspective laboratories, attracting international students, and launching international and competitive educational programs. These joint efforts will give rise to quality and quantity of publications, ratio of international academic and research staff to all and international student involvement in education and research in Russia. This way is perspective but requires substantial amount of time to implement and support development activities (for example, several of the government program for increasing competitiveness of Asian universities that we mentioned above expect valuable results only to 2050).

  2. 2.

    Increasing only the formal indicators, which is far less time-consuming than way described above, but is not quite honest. For example, universities may increase the number of publications in so-called “trash journals” or publish more conference papers and notes, which are non-reviewed papers. According to this point of view, the presence of the affiliation of such universities in the highly-cited articles brings the necessary numbers to the university profile and requires far less efforts. However, in long-term perspective this way is actually the end road.

We intentionally do not make assumptions about the way that universities chose, it can be idea for further research, and such research can be used as basis for university evaluation under the program “5–top 100”. Moreover, there is another interesting area for further research. We have compared the universities participating in Program “5–top 100” only with Russian universities. For example, the research based on comparing the influence of program “5–top 100” on Russian universities and other development programs abroad may be meaningful and useful. There is a wide range of educational programs in developing countries that are to reform educational system (similar to Program “5–top 100”), so the brief initial analysis provided in (Povalko 2014) can be extended. One more way is assessing performance of universities using new ranking techniques, comprising not only research-oriented indicators, but also social, industrial and environmental (Sokolov 2015).

Conclusion

The hypothesis tested was described in part 1. Based on the results obtained we may conclude that our hypothesis is not rejected. The program “5–top 100” demonstrates the positive influence on the publication activity of the universities evaluated. Universities from Reference group may participate in different supporting and development government programs, but they would like to compete with other universities including universities participating in “5–top 100” in order to strengthen their current position in the educational and research areas in the world.

All participating universities should present their very ambitious strategies up to 2020. Owing to the program, participating universities started working with industry more closely, providing the students with knowledge and skills required by international labour market. For universities to be very competitive, close collaboration with venture scientific and production associations, funds and industry corporations is essential (Gogolev 2015).

This research was carried out in Scientific Research Institute—Federal Research Centre for Projects Evaluation and Consulting and financed with the support of the Ministry of education and science of the Russian Federation according to the State Task No. 2015/N7 project No. 3253 “Organization and monitoring the innovative activity of the subjects of the Russian Federation”.