Introduction

In recent years, Chinese universities have achieved great development in terms of the number of researchers and academic research funding (Xu 2008). China is one of the rapid growing economies of the World (Guilln 2003) and the great volume of research coming out of this country necessitate the measurement of the research performance of Chinese universities by comparing them with international standards for the purpose of assessing the quality and impact of the research.

In this paper, we measure the research performance of Chinese universities by using bibliometric indicators. Bibliometrics is a group of methods, such as citation analysis, to quantitatively analyze academic literature (De Bellis 2009). It has been widely used to explore the impact of universities’ research outcomes in order to measure research performance. However, most of the researches using bibliometric indicators usually focus only on research performance in certain specific areas, such as the work done by (Ding et al. 2013; Fu et al. 2014; Yang et al. 2013), or they only evaluate Chinese universities’ research outcomes but do not bring other top universities of the world into the comparison, such as the work done by (Cao et al. 2010; Fu and Ho 2013; Moed 2002).

The goal of our work is to find out the research strengths and weaknesses of Chinese universities when compared with the top universities in the world. Our findings can help Chinese universities identify their weakness and know how to improve by learning from the other top tier universities, e.g. by investing more funding on international collaboration research projects. For means of comparison, we measure the academic research performance Chinese universities against those universities from two regions; the US and the Europe across various subject areas in order to see the gap. We first adopt a list of existing indicators to measure the recent research performance of Chinese, American and European universities in terms of both the quantity and quality of publications and citations, and then deeply analyze the difference. We also compare the internationalization of the research of Chinese, American and European universities. Lastly, we propose an overall score to measure the research performance of these universities.

The paper is organized as follows: in “Related work” section, we present the existing related work on bibliometric methods, in particular for measuring academic research performance. In “Data and methodology” and Result and discussion sections then present our data and methodology with results and discussion. Concluding remarks and future work are outlined in Concluding remarks section.

Related work

This section will discuss existing related research on bibliometric study particularly on which measuring research performance. The review of some important previous works grouped in the thematic order is as follows:

Moed et al. (1985) first presented the results of a study on the potentialities of bibliometric data as a tool for university research policy formulation. The authors concluded that the use of bibliometric data can provide monitoring for university research-management and science policy, despite the fact that there are problems in data collection and handling.

Liang and Wu (2001) proposed a model to prove how important the indicator of publication count is in comparison with quality or impact indicators. Extended negative exponential function proves to be an appropriate model to fit the rank-frequency distribution here. The Scopus database we have used in our research matches their criteria to select databases.

Raan (1996) gave an overview of the potential and limitations of bibliometric methods for the assessment of strengths and weaknesses in research performance based on the following methods: research performance assessment and monitoring scientific developments. The advanced analysis of publication and citation data shown in this article provides insight into the position of actors on the research front in terms of influence and specializations. We have selected some of these indicators in our bibliometric study as well.

Moed (2010) presented a new indicator of journal citation impact, denoted as Source Normalized Impact per Paper (SNIP). It measures a journal’s contextual citation impact based on the frequency at which authors cite other papers in their reference. SNIP is defined as the ratio of the journal’s citation count per paper and the citation potential in its subject field. It aims to allow a direct comparison of sources in different subject fields. The citation potential varies not only between subject categories but also between journals within the same subject category. For instance, journals covering emerging topics have higher citations than research published on classical subjects. SNIP adjusts for such differences; therefore, we have adopted the SNIP metric for measuring the research performance in our study to ensure that citations from journals with low number of references are weighted correctly.

Leeuwen et al. (2001) have presented evidence that the value of impact indicators of research activities strongly depends upon whether one includes or excludes research publications in SCI covered journals written in languages other than English. Additional materials were gathered to show the distribution of SCI papers among publication languages. In our measurement, we have excluded all the self-citations and used the SNIP values to measure the level of journals, which can avoid the issue of many authors publishing articles in SCI journals with low impact factor as the journals’ language is Chinese.

Zitt and Bassecoulard (1998) focused on determining the internationalized nature of science by studying the distribution of the authoring and citing countries of a journal in their study. These authors conducted the experiment that determines how experimental internationalization indexes and the SCI for science indicators are sampled. This information has been very useful for our experiments in this research.

Tijssen et al. (2009) presented the results of a statistical analysis carried out among the world’s major providers of science-based information and services to the business sector. The statistical data was derived from university and industry co-authored research publications (UICs). The various UIC rankings highlight measurement issues and reveal the differences depending on the selected UIC indicator. The UIC indicator offers an interesting tool for carrying out domestic and international comparisons of research universities.

Hassan and Haddawy (2013) introduced a new quantitative measure to determine the international scholarly impact of countries. They presented a case study to illustrate the use of a proposed measure in a particular area and found that the international scholarly impact of the countries is not necessarily bound to publication output.

Cao et al. (2010) focused on measuring academic output in terms of overall publication activity and the influential strength of certain disciplines of the selected universities using Essential Science Indicators (ESI) database in their research. The authors used bibliometric indicators, such as number of publications and citations to devise a framework for their research. However, the ESI database is smaller than SCI and Scopus and so it might not fully reflect the performance of universities in China. In addition, although this research discussed the gaps between the quality and quantity of research output, it did not address the gap between Chinese universities and world-class universities; a gap which we aim to fill through our research.

Moed (2002) focused on using bibliometric techniques to measure research activities in China. This research was based on data extracted from the Science Citation Index (SCI). One of the main contributions of the author is to split ISI journals into Chinese and non-Chinese journals based on the publishing language of the journal in order to measure Chinese universities’ research performance. However, this approach is not comprehensive as some Chinese journals can also be very good journals, e.g. Journal of SoftwareFootnote 1 and thus we believe that research performance must be measured on the basis of quality, i.e., publications in high impact journals and conferences.

Fu and Ho (2013) studied the publication outputs based on Science Citation Index Expanded (SCIE) from 1974 to 2011 for the top two Chinese universities; Tsinghua University and Peking University, using h index and highly cited articles. However, since the comparison was limited to only two universities, the findings cannot be generalized. Ding et al. (2013) evaluated the productivity of China in the field of pharmacy in relation to ten representative countries using citation analysis. The authors also selected the USA and European countries and found that there is huge gap between China and developed countries. However, their research only focus on research performance in certain specific areas, which is not sufficient to provide comprehensive suggestions to developing countries.

More recently, Yang et al. (2013) studied municipal solid waste manage in China based on the publication output in SCI. Similarly, Fu et al. (2014) further evaluated China’s scientific output of chemical engineering based on SCIE database. But their researches only focus on certain areas which cannot show the overall research performance of Chinese universities.

All of the above-mentioned researches focused only on a selected discipline and do not compare Chinese universities with other world-class universities across disciplines. Thus the goal of our work is to compare the research outcomes of Chinese universities with top universities in the world across disciplines based on the publication quantity, quality and international impact.

Data and methodology

This study covers seven research areas in science and technology that most of the universities in the world are currently working on: Agricultural & Biological Sciences, Biochemistry, Genetics and Molecular Biology, Chemistry, Computer Science, Engineering, Environmental Sciences and Physics and Astronomy. The data available for these disciplines over the time period of 2007–2010 was extracted from the Scopus database which contains more than 5,000 international publishers and at least 19,500 peer-reviewed journalsFootnote 2; offering about 20 % more coverage than Web of ScienceFootnote 3 (Falagas et al. 2007). For our measurement, we picked the top 20 universities from China, the US and Europe. These universities were chosen for the period from 2007 to 2010 having the highest number of publicationsFootnote 4 (Details can be found at Supplementary Table 1–7).

The bibliometric indicators given below were used to capture the quantity, quality and the internationalization of academic research:

Quantity

The following indicators are used to measure the research quantity:

  • Total Pubs: Total number of publications during a 4-year time window. If a publication has more than one author from different universities, then we count one for each university.

  • Total Cites: Total number of citations within a 4-year time window of papers published in the same time window. All citation counts in our calculation exclude self-citations.

Quality

The following indicators are used to measure the quantitative aspect of research; here the percentage indicators help to highlight universities which are small but producing high quality research:

  • %Pubs in Top 10 % SNIP: Percentage of Total Pubs published in source titles (journals, conference proceedings and book series) that are within top 10 % of that subject area, based on the Source Normalized Impact per Paper (SNIP) of the last year in the time window. Source titles in top 10 % based on SNIP values in a given subject category are tier-1, in top 11–25 % are tier-2, others are tier-3. For the window 2007–2010, the SNIP values of 2010 are used. The details and the definition of SNIP can be found at (Moed 2010).

  • %Pubs in Top 25 % SNIP: Percentage of Total Pubs published in source titles that are within top 25 % of that subject area, based on the SNIP value of the last year of the time window.

  • %Cites from Top 10 % SNIP: Percentage of Total Cites received from publications in source titles that are within top 10 % based on SNIP value.

  • %Cites from Top 25 % SNIP: Percentage of Total Cites received from publications in source titles that are within top 25 % based on SNIP value.

  • 4-year h index: A university having a 4 year h index of X means that at least X of their publications (during that 4-year window) have no less than X publications citing them (during that window). A 4-year h index of a university is computed for a particular subject area.

Internationalization of research

The following indicators are used to measure the internationalization of research:

  • %International Collaboration: Percentage of Total Pubs with international co-authorship, which means only the publication has at least 2 authors from two different countries will be counted.

  • %International Cites: Percentage of International Cites relative to Total Cites. Citations received from papers authored by researchers from outside the country in which a given university is located.

  • International Impact: This indicator is a measure of the impact a university’s research is having outside its country of location and measures the ability of a university to attract citations from outside of the country. It is defined as the ratio of International Cites to the total number of citations made by the papers in a given field, which are authored by researchers from outside of the country in which the university is located (Hassan & Haddawy, 2013).

Results and discussion

In this section, we discuss the research performance of Chinese universities. We first compare the quantity and quality of the publications produced by Chinese universities with American and European universities. We then analyze the internationalization of their research in terms of International collaboration and International citations. Finally a new score is proposed to evaluate the overall research performance with total points for each subject category for Chinese universities.

Quantity of publications and citations

We first normalized the number of publications and citations into more common scale [0, 1] using standard min–max normalization in order to make comparisons on normalized values (Dodge 2003). The original numbers can be found at Supplementary Figure 1 and 2. Figure 1 shows the normalized number of publications along with the actual counts in all three regions from 2007 to 2010. As we can see, Chinese universities performed very well in the discipline of Engineering, wherein they published nearly more than three times the number of papers published by universities in America and Europe. In the disciplines of Chemistry, Computer Science, Physics and Astronomy, Chinese universities also do better than American and European Universities regards to the number of publications. However, in the disciplines of Agricultural & Biological Sciences, Biochemistry, Genetics and Molecular Biology, and Environmental Sciences, America universities take the lead. Particularly in the disciplines of Biochemistry, Genetics and Molecular Biology, the number of publications from Chinese universities is only half of the publications produced by American universities. We also notice that European universities are behind American universities in all disciplines with the exception of Chemistry in terms of the number of publications.

Fig. 1
figure 1

Normalized number of publications by top 20 Chinese, US and European universities from 2007 to 2010

Full size image

Now we shall proceed to propose some basic intuitive explanations to this phenomenon. Compared to the other two regions, China has more researchers and PhD students due to its high population.Footnote 5 In addition, according to the Fig. 1, Chinese universities focus on research areas within the discipline of engineering, such as Computer Science or Engineering, and relatively theoretical areas, such as Chemistry and Physics.

Although number of publications is an important indicator in measuring a university’s research performance, it is also critical to focus on the impact of publications so as to gain a clearer picture. Figure 2 shows the normalized number of received citations based on publications by the top twenty Chinese, American and European universities during the timeframe of 2007–2010. Generally speaking, the number of citations should be in proportion to the number of publications. However, even in the discipline of Engineering, in which Chinese universities have the most number of publications, American universities have received much more citations than the Chinese universities. A similar situation occurs in other disciplines as well, e.g. Chemistry and Computer Science. In the rest of the disciplines, e.g. Biochemistry, Genetics and Molecular Biology, American and European universities already have an advantage in terms of number of publications, and have thus received nearly seven and four times more citations than Chinese universities, respectively.

Fig. 2
figure 2

Normalized number of citations received based on the publications by top 20 Chinese, US and European universities from 2007 to 2010

Full size image

Obviously, the difference between the number of publications and the number of citations emphasizes a fundamental gap in the research performance of Chinese universities. To further analyze the results, we apply the City-Block distanceFootnote 6 to calculate the difference for each subject category as denoted in Equation 1.

$$ d(x,y) = x - y,\quad x \ge y, $$
(1)

where x is the normalized number of citations and y is the normalized number of publications. We are only interested in those subject categories in which universities have a greater or equal number of normalized citations than the normalized number of publications as this means that there is a positive impact on research performance. While Chinese universities only have one category that can meet this condition, American universities have all seven categories with positive impact on research performance. The results are shown in Fig. 3.

Fig. 3
figure 3

Number of subject categories that have greater or equal number of normalized number of citations than normalized number of publications

Full size image

Table 1 shows the region that has the highest positive impact in all subject categories, calculated using Equation (1). A similar situation has happened for Chinese universities as they are not even listed in any of the categories. A very interesting finding is that US universities have 0.62 positive impacts in the discipline of Engineering, which proves that their research performance is not as bad as we originally thought with respect to the number of publications. European universities show that the positive impact means they receive more citations regardless of their lesser number of publications relative to China.

Table 1 Region that has the highest positive impact in all subject categories
Full size table

Quality of publications and citations

In this section, we are going to evaluate the quality of research publications and their citations following our assessment of the percentage of publications that have been published in top 10 % source titles of each subject area based on SNIP value.

The results are shown in Fig. 4. Chinese universities are behind American and European universities in all subject areas. For example, only 7.3 % publications published by the Chinese universities are in top 10 % SNIP in the discipline of Computer Science, which is nearly three times and two times lower than the US and the European universities, respectively. In addition, in none of the disciplines can Chinese universities have more than 35 % publications in top 10 % SNIP. The results of the same indicator for American universities are 58.9 and 63.3 % for European universities.

Fig. 4
figure 4

Percentage of publications in top 10 % SNIP by top 20 Chinese, US and European universities from 2007 to 2010

Full size image

Compared with the number of publications, we can observe that although Chinese universities have a high number of publications in certain areas, the proportion of publications in top journals/conferences is relatively low, a figure which can also be linked to the low number of citations. The top universities in America or Europe can have high proportion of publications in top journals/conferences, which reflect that the quality is the publication focus of top universities. In contrast, Chinese universities have less than 30 % publications in top journals/conferences.

A similar situation occurs in the percentage of total cites received from publications that are within top 10 % SNIP of the subject areas, based on the SNIP values from 2007 to 2010, which are shown in Fig. 5. This indicator denotes the extent of publication recognition. Chinese universities still lag behind American and European universities in terms of the quality of citations. However, the gap is not as big as that in the quality of publications. Using the example of the discipline of Computer Science again, we can see that Chinese universities have only a 7 % difference from American universities and a 4.1 % difference from European universities in terms of total cites received from the publications that are within top 10 % SNIP.

Fig. 5
figure 5

Percentage of total cites received from publications that are within top 10 % SNIP from 2007 to 2010

Full size image

As a complementary part of this research, we have also measured the percentage of publications and total cites in the case of top 25 % SNIP (Details can be found at Supplementary Figs. 3 and 4). A similar situation occurs as top 10 % SNIP, but the gap becomes smaller. In Computer Science, for example, in the case that total cites received from publications that are within top 25 % SNIP, Chinese universities have the same figure as European universities and only a 2.7 % difference from American universities. The details can be found at our online data snapshots as stated in the section of “Data and methodology”.

Such difference illustrates that publications from Chinese universities mainly fall into low impact venues. We can thus see that the performance of Chinese universities is developing significantly, but further betterment still requires greater publication in top venues.

We further assessed the research performance of Chinese universities using 4-year h index as shown in Fig. 6, which is an important figure to measure both the productivity and impact of the published work of a scholar. This indicator was first suggested by Hirsch (2005) and has been widely used in many places including Google scholar.Footnote 7 The h index is defined as:

Fig. 6
figure 6

Sum of 4-year h-index based on the publications by top 20 Chinese, US and European universities

Full size image

Definition 1

Assume a scientist has total Np papers, if h papers have at least h citation each, and the other (Np–h) papers have no more than h citations each, then his/her h index is h.

As we can see from Fig. 6, American universities have the highest h index in all subject areas. For the Chinese universities, Chemistry is the area closest to the US and the European universities, only around 10 % difference to the US universities and 2.5 % difference to the European universities.

Internationalization of publications and citations

For international collaboration in research, we measure the percentage of total publications with an international co-authorship. We only count the publications that have at least two authors from two different countries. As for international citations, we only count citations received from papers authored by researchers from outside of the country in which the given university is located.

As shown in Figs. 7 and 8, there is no doubt that European universities lead in these two indicators, due to their close collaboration with universities in other European countries, while Chinese and American universities are more likely to collaborate with universities in their own country. Generally speaking, International Collaboration and International Cites are important indicators reflecting the extent of research activity and they also denote the significance of academic communication.

Fig. 7
figure 7

Mean percentage of international collaboration of publications by top 20 Chinese, US and European universities

Full size image
Fig. 8
figure 8

Mean percentage of international citations received from publications by top 20 Chinese, US and European universities

Full size image

International impact is an indicator which is more precise in unveiling the worldwide reputation and the influence of the research. It is defined as the ratio of international cites to the total number of citations made of papers which are authored by researchers from outside of the country in which the university is located in a given field. From the Fig. 9, we can conclude that American universities are leading in almost every subject, while Chinese universities are lack in the worldwide impact. However, in the discipline of Chemistry, wherein Chinese universities have traditionally displayed good performance, they have an international impact similar to American and European universities, with gap of only from 0.004 to 0.005.

Fig. 9
figure 9

Sum of international impact from publications by top 20 Chinese, US and European universities

Full size image

Next, we shall summarize the results above and pick the top three disciplines for each region from Fig. 7, Fig. 8 and Fig. 9 as shown in Table 2. We find that Chemistry is one of top three subject categories for Chinese universities with respect to international cites and international impact but not in the top three subject categories with respect to international collaboration, which means Chinese universities have a very strong local research ability in Chemistry and the research outcomes have been recognized internationally. Engineering and Computer Science are the other two disciplines in which Chinese universities are faring well. Further, the disciplines of Physics and Astronomy is the subject category that takes our notice as for this area, Chinese universities are not in the top three subject categories of any international indicator.

Table 2 Region that has the highest positive impact in all subject categories
Full size table

Although there are various indicators available to evaluate the research performance of different universities, we can observe that these indicators will rank the universities in different ways since each indicator evaluates only a limited aspect of the research performance. In order to evaluate and rank the overall performance of academic research for different universities, we propose a new indicator; Research Performance Point (RPP) for each subject area to measure university research performance, based on all above outcomes from various indicators. Similar to the method of many universities ranking, including Times,Footnote 8 RPP is cumulative according to the figures of each indicator, but we assign equal weight to each indicator. The calculation of RPP is shown as below:

$$ RPP = \sum\nolimits_{i}^{n} {\alpha_{i} } ,i = 1,2,3,..,n , $$
(2)

where α i represents the normalized score of each indicator. For example, for the number of publications in Computer Science, Chinese universities published the most papers i.e. 55076 while the numbers of publications of American and European universities are 31072 and 27511 respectively. Thus, we convert the number of publications to normalized score 1 to Chinese universities while 0.56 and 0.49 to the US and the European universities accordingly. Since we have used 10 indicators in total, the maximum RPP that can be achieved for each subject area is 10. The results given by RPP are demonstrated in Fig. 10.

Fig. 10
figure 10

Sum of RPP for top 20 Chinese, US and European universities

Full size image

From the Fig. 10, we can conclude that similar to other outcomes, American universities dominate nearly all disciplines except for some particular disciplines, such as Chemistry. Chinese universities are doing well in the area of Chemistry. However, they still have a long way to go for other disciplines, particularly in the disciplines of Agricultural & Biological Sciences, Biochemistry, Genetics and Molecular Biology, where they do not even achieve half of total RPP.

Through RPP, we cumulatively consider all kinds of factors including the publication quantity, quality and international reputation of each university, so that the overall performance of academic research can be quantified and visualized. Additionally, researchers can more clearly recognize the position of Chinese universities’ research in the international context and this may help in the development of the direction of academic research in Chinese universities.

Concluding remarks

In this paper, we focused on measuring the research performance of Chinese universities using bibliometric methods. We first used several established indicators to measure the quantity and quality of publications and citations. Compared to American and European universities, Chinese universities have already reached world class level in terms of number of publications and citations, particularly on the area of Engineering and Chemistry. However, we find that the number of citations received from Chinese universities’ publications is not in proportion to the number of their publications compared to American and European universities, which implies that the quality of Chinese universities’ publications might be not as good as the quantity.

We then investigated the quality of publications and citations using SNIP value and h index. We found that in the comparisons of either SNIP or h index, Chinese universities still have quite a large deficit in terms of quality of research as compared to American and European universities, barring some disciplines such as Chemistry, Biochemistry, Genetics and Molecular Biology are the disciplines in which Chinese universities are weakest in terms of research, with less than 20 % publications being published in top journals/conferences. Such a gap reflects the fundamental reason for the different publication performance, so other than producing high quantity of publications Chinese universities should focus more on publishing high level research results at top venues.

We further measured the international collaboration, citations and impact for Chinese universities and found that even in the discipline in which they are the strongest; Chemistry, the research of Chinese universities is not internationalized enough compare to American and European universities. Such results indicate that Chinese universities have a large improvement space in international communication, and they can seek for collaboration among other countries in the region to better the quality of their research.

Last but not least, we introduced and applied the RPP, which sums the results from all the different indicators to normalize the overall research performance of Chinese universities. We also presented a visual comparison results for other researchers to recognize the position and overall level of Chinese universities in terms of academic research in the international context. This can be useful in helping Chinese universities develop their research direction and focus in the future. We further found that Chinese universities’ research level is much closer to American and European universities in the disciplines of Chemistry and Engineering. For the other disciplines in which Chinese universities are weak, including Agricultural and Biological Sciences and Biochemistry, Genetics and Molecular Biology, Chinese universities really need to learn more from their American and European counterparts e.g. European universities have done research in Environmental Sciences and Agricultural and Biological Sciences which is better than even American research in these areas. .

In future we plan to examine the research performance of universities in some of the disciplines wherein much research is being carried out currently, such as Sustainable Development and Renewable Energy. By examining the research being done in these areas, we expect to find out which universities are doing the best work so that universities or organizations in the developing countries can seek help from them.