Introduction

The research evaluation for different countries (e.g., Spain, Finland, Australia) is based on quantitative performance indicators which have the added advantage of being more cost efficient (OECD 2010; Butler 2004).

At the sectoral (university, hospital, government), institutional, and even lower levels of aggregation, such as faculties, departments, and researchers, the allocation of funds earmarked for research is based on a formula encapsulating a number of performance measures such as aggregate productivity counts and publication output (Butler 2004).

For example, since 1989 a research incentive system has existed in Spain, administered by the National Commission for the Evaluation of Research Activity (CNEAI 2011). Researchers were rewarded with salary bonuses for publishing in prestigious journals, principally articles appearing in a relatively high position (approximately the top one third) in ISIs Journal Citation Report lists by subject category (Jiménez-Contreras et al. 2003). In Finland part of the funding for universities rests on publication points, weighted according to the impact factor of the journals carrying the work (Adam 2002). While in the British Research Assessment Exercise the link between research rankings and performance measures, and hence funding, is less direct, they nevertheless play an important role in the deliberations of the review panels (Butler 2004).

The idea is that with no attempt made to differentiate between the quality, visibility or impact of the different journals when funding is allocated, there is little incentive to strive for publication in a prestigious journal. In journal ranking models, the ranking score is a numerical value assigned to a journal representing an indicator of its scientific prestige and influence. An analysis of a country’s presence in the given journal ranking model is the first step taken to investigate whether it is possible to demonstrate the apparent effect of the introduction of any funding formulas. For this analysis, journals can be allocated to quartiles based on a prestige indicator (e.g., the average citation per publication rates of the publications they carried). Journal quartiles are the value of the boundary at the 25th, 50th, or 75th percentiles of an ordered distribution of journal ranking scores divided into four parts, each containing a quarter of the journal ranking scores. And the share of total publications in each of the four quartiles is then tracked and studied over the full period of time under analysis.

To this aim journal quartile rankings are derived for journals in each of their subject categories according to which quartile of the score distribution the journal occupies for that subject category. They can play an important role in performance-based funding of public research.

For instance, using the ISI impact factor to represent the journal ranking score (Garfield 2006), quartile rankings are therefore derived for journals in each of their subject categories according to which quartile of the impact factor distribution the journal occupies for that subject category, where Q1 denotes top 25% of the ISI impact factor distribution, Q2 a middle-high position (between top 50% and top 25%), Q3 a middle-low position (top 75% to top 50%), and Q4 bottom position (bottom 25% of the ISI impact factor distribution). For example, the 2010 ISI impact factor for Scientometrics is 1.905. With this score the journal ranks 22nd (out of 97 journals, Q1 quartile) in subject category computer science interdisciplinary applications.

Giving the impact of performance-based funding schemes in countries like Spain and others, it follows that quartile ranking validation can be a very important issue because it needs to be established the soundness of journal quartile rankings for research evaluation systems. That is, assuming that journal impact relates to the recognition of the originality of research and its impact on the development of the same or related discipline areas from the multivariate viewpoint of several journal ranking models (e.g., ISI impact factor, SJR (González-Pereira et al. 2010), etc.), what is the link between quartile rankings and journal impact? In particular, are there first quartile journals in a given subject category which are not of highest impact? And, which are the impact rankings of journals in the four quartiles for a subject category?

In this paper we study the relationship between journal quartile rankings and unsupervised statistical classification in four impact classes: Highest impact, medium highest impact, medium lowest impact, and lowest impact journals. To this aim an unsupervised classification algorithm is used to identify groups of journals sharing similar characteristics in a multivariate indicator space, that is, a number of different indicators shall be used in the analysis, e.g., SJR, H-index, ISI impact factor, Eigenfactor Score, Article Influence Score, etc.

The objective of this study is to apply fuzzy clustering algorithms to identify impact classes (in a multivariate indicator space) and to study their relationship with journal quartile rankings. The fuzzy clustering algorithms allow impact classes to overlap, thereby accommodating for uncertainty related to impact classes transition zones (i.e., the confusion about the impact class attribution for a journal) and vagueness in impact classes definition (e.g., what is a medium-lowest impact journal?). The resulting impact classes will be optimal in the sense that the multivariate within impact class variance is minimal.

As a result, we will be able to study the occurrence of first quartile journals which are not of highest impact as well as the occurrence of medium-highest impact journals which are not in first quartile of ISI impact factor.

For example, the 2010 ISI impact factor for Pattern Recognition is 2.607. With this impact factor the journal ranks 16th (out of 108 journals, Q1 quartile) in subject category computer science artificial intelligence. And Pattern Recognition is found to be a medium highest impact journal using fuzzy clustering to identify impact rankings (see “Results” section).

The 2010 ISI impact factor for Pattern Recognition Letters is 1.213. With this score the journal ranks 62nd (out of 108 journals, Q3 quartile) in subject category computer science artificial intelligence. But Pattern Recognition Letters is also found to be a medium-highest impact journal using unsupervised classification (i.e., fuzzy clustering algorithms).

That is, two journals of different quartiles (Q1 and Q3) of ISI impact factor are both allocated to the medium-highest impact class when using fuzzy clustering in a multivariate indicator space (see “Results” section).

The setup of the paper is organized as follows: “Methods” section introduces study subject category, multivariate indicator space, and unsupervised statistical classification. “Results” section reports the results of our analysis. “Conclusion” section concludes.

Methods

Study subject category and multivariate indicator space

Several metrics based on citation counts have been developed to evaluate the impact of scholarly journals (van Raan 2004), one of which, the impact factor published by Thomson Scientific (also called ISI impact factor) (Garfield 2006), has been the dominant measure for ranking a journal’s impact, which is used by research institutions, policy makers, and journal editors alike.

The impact factor, often abbreviated IF, is a measure reflecting the average number of citations to articles published in science and social science journals. It is frequently used as a proxy for the relative importance of a journal within its field, with journals with higher impact factors deemed to be more important than those with lower ones. The impact factor was devised by Eugene Garfield (Garfield 2006), the founder of the Institute for Scientific Information (ISI), now part of Thomson Reuters. Impact factors are calculated yearly for those journals that are indexed in Thomson Reuter’s Journal Citation Reports.

Even though most evaluators stick to some form of the traditional impact factor, one of the earliest proposals was a weighted measure for journals developed by Pinski and Narin (1976). There are other exceptions like analyses carried out by Liebowitz and Palmer (1984), Palacios-Huerta and Volij (2004), Kalaitzidakis et al. (2003) and Kodrzycki and Pingkang (2006), who rank economic journals using an iterative procedure.

A recent trend is aimed to develop metrics which represent scientific impact as a function not of just the quantity of citations received but of a combination of the quantity and the quality (Palacios-Huerta and Volij 2004; Bollen et al. 2006; Ma et al. 2008; Bergstrom 2007). In particular, Rousseau et al. (2009) applies an alternative approach to the measurement of scholarly quality which summarizes the incidence, intensity, and inequality of these journals’ highly cited articles.

The SCImago Journal Rank (SJR) (González-Pereira et al. 2010), presents an indicator of “journal prestige” (Bollen et al. 2006), that belongs to a new family of indicators based on eigenvector centrality. The SJR indicator is a size-independent metric aimed at measuring the current “average prestige per paper” of journals for use in research evaluation processes. It has already been studied as a tool for evaluating the journals in the Scopus database, compared with the Thomson Scientific Impact Factor and shown to constitute a good alternative for journal evaluation (Leydesdorff et al., 2010).

The Eigenfactor Score calculation is based on the number of times articles from the journal published in the past 5 years have been cited in the JCR year, but it also considers which journals have contributed these citations so that highly cited journals will influence the network more than lesser cited journals (Journal Citation Reports 2011). References from one article in a journal to another article from the same journal are removed, so that Eigenfactor Scores are not influenced by journal self-citation.

The Article Influence determines the average influence of a journal’s articles over the first 5 years after publication (Journal Citation Reports 2011). It is calculated by dividing a journal’s Eigenfactor Score by the number of articles in the journal, normalized as a fraction of all articles in all publications. This measure is roughly analogous to the 5-Year Journal Impact Factor in that it is a ratio of a journal’s citation influence to the size of the journal’s article contribution over a period of 5 years.

The H-index is an index that attempts to measure both the productivity and impact of the published work of a scientist or scholar. The index is based on the set of the scientist’s most cited papers and the number of citations that they have received in other people’s publications. The index can also be applied to the productivity and impact of a department or university or country or journal. The index was suggested by Jorge E. Hirsch, a physicist at UCSD, as a tool for determining theoretical physicists’ relative quality (Hirsch 2005), and is sometimes called the Hirsch index or Hirsch number. In our analysis, the H-index expresses the journal’s number of articles (h) that have received at least h citations over the whole periode. To compute the H-index for each journal in a subject category we simply follow the values given in the SCImago Journal and Country Rank portal (available at: http://www.scimagojr.com).

An immediacy index is a measure of how topical and urgent work published in a scientific journal is. Along with the better known impact factor measure, it is calculated each year by the Institute for Scientific Information for those journals which it indexes; both impact factors and immediacy indices are published annually in the Journal Citation Reports (2011).

In this study we analyse the relationship between journal quartile rankings of ISI impact factor (at the 2010) in subject category computer science artifical intelligence and journal classification in four impact rankings (i.e., highest impact, medium highest impact, medium lowest impact, and lowest impact). To this aim, we use unsupervised statistical classification in order to identify groups of journals sharing similar characteristics in a multivariate indicator space. The seven variables used in this analysis are: (1) Scimago Journal Ranking (SJR); (2) H-Index (H); (3) ISI impact factor (IF); (4) 5-Year Impact Factor (5IF); (5) Immediacy Index (II); (6) Eigenfactor Score (ES); (7) Article Influence Score (AIS).

The last 5 variables are taken from the SCI JCR produced by Thompson Reuters ISI and derived off its Science Citation Index. The first two are produced by SCImago and derived off Scopus produced by Elsevier. The indicator scores are standardised to [0, 1] by a linear stretch between the minimum and maximum score values. Standardisation ensures that each of the indicators is equally weighted as no prior information about the relationships between the indicators and the journals of subject category computer science artificial intelligence was here assumed.

Part of the datasets (SJR and H scores) was retrieved from the website SCImago Journal and Country Rank portal (SCImago portal 2011). The rest of the data was retrieved from the website Journal Citation Reports (2011) Thomson Reuters. The data were downloaded in June–July 2011.

Thompson Reuters ISI is a US company, Elsevier is a European company, and SCImago is a Spanish research group, which is influential in Spain and Latin America. Journal coverage of the SCI and Scopus is based on different principles, and this–possibly the different national perspectives of the producers—influences the variables being input into our model. The former coverage is based on sociometric, elitist principles, whereas the latter aims to be more comprehensive. Information on this coverage can be obtained from the Web sites of the companies, where they discuss their respective products. It can be of interest to point out that SCImago uses Google PageRank to construct its measures.

Fuzzy k-means classification

The fuzzy k-means clustering algorithm is an unsupervised classification algorithm designed to identify groups of samples sharing similar characteristics in a multivariate feature space (Dunn 1973). Even though it is an extension to the k-means algorithm (Duda et al. 2001), the fuzzy k-means allows class clusters to overlap, thereby accommodating for uncertainty related to journal impact ranking transition zones and vagueness in journal impact definition. The clusters are optimal in the sense that the multivariate within cluster variance is minimal.

Small variance implies that all journals have similar indicators, which means a high density and small distance between them in multivariate indicator space. Large variance is equivalent to low density and large distances in multivariate space. An optimal cluster procedure will identify these dense spots in multivariate indicator space as class centres, while the boundaries between classes in multivariate indicator space should be located in the lowest density regions.

Hence the main aim of such a unsupervised clustering is to subdivide a complex continuous multivariate indicator space into a set of clusters. Unsupervised clustering algorithms do not attach meaningful class labels to these clusters (as opposed to supervised classifiers).

In our problem, journal impact relates to the recognition of the originality of research and its impact on the development of the same or related discipline areas from the multivariate viewpoint of seven indicators (i.e., IF, 5IF, II, ES, AIS, SJR, H). And we are interested in the link between journal quartile rankings and an unsupervised statistical classification in four impact classes: Highest impact, medium highest impact, medium lowest impact, and lowest impact journals. But although the central concepts of journal impact classes may be clearly defined, often for subject categories it is difficult to avoid overlap in both impact class definitions and in the values of key indicators.

There fuzzy classification may be successfully applied in a given subject category to overcome the problem of impact class overlap. The method serves as an unsupervised exploratory technique that suggests how best to divide a collection of journals (using a multivariate indicator space) into meaningful groups, both in terms of the number of impact classes and their definition.

The fuzzy k-means algorithm applies an iterative procedure that starts with an initial allocation of the journals to be classified into impact class clusters.

Let Q1 be top 25% of ISI impact factor distribution for subject category under analysis; Q2 be middle-high position (between top 50% and top 25%); Q3 be middle-low position (top 75% to top 50%); and Q4 bottom position (bottom 25% of ISI impact factor distribution. To obtain an initial allocation we proceed as follows: Q1 journals are allocated to highest impact class; Q2 journals to medium-highest impact class; Q3 to medium-lowest impact class; and finally, Q4 journals to lowest impact class.

Given a cluster allocation, the centre of each cluster (in terms of indicator values) is calculated as the weighted mean of the journal indicator values, also known as the centroid. Reallocation of the centroids proceeds by iteration until a stable solution is reached where the algorithm has found the optimal locations of the cluster centres and where the locations of the centroids do not change.

This process is also referred to as convergence of the fuzzy k-means and can be measured by the changes in impact class allocation to each journal. When the algorithm has converged, each journal in subject category can be classified according to their distances to the impact class centroids (Dunn 1973; Duda et al., 2001).

The similarity of a journal to each one of the four impact classes is expressed by a membership value. Let u k i be the membership probability of the ith journal of ISI impact factor distribution to the kth class cluster (where k = 1, 2, 3, 4 is for highest impact, medium highest impact, medium lowest impact, and lowest impact class cluster, respectively), which is defined as:

$$ u_{k i} = \frac{ \left(d_{k i}^2\right)^{\frac{1}{q-1}}}{\sum_{p=1}^{4} \left(d_{p i}^2 \right)^{\frac{1}{q-1}}}$$
(1)

where d pi is the distance between the indicator values of journal ith and cluster centre p in multivariate indicator space and q is the fuzziness exponent representing the degree of impact class overlap. Generally the Euclidean distance from a sample vector to a cluster mean is taken to be the similarity criterion. It induces hyperspherical clusters. Hence it can only detect clusters with the same shape and orientation.

The degree to which a journal in the original ISI impact factor distribution belongs to an impact class is expressed not in terms of a binary yes or no but by a continuous membership value that ranges between 0 and 1, where 1.0 indicates perfect similarity with the impact class centroid.

The parameter q > 0 is the fuzzy exponent determining the amount of overlap in the impact class model. A value close to 1.0 results in a classification with discrete impact class boundaries (which is similar to the k-means output).

A very large value of q, q → ∞, results in fully overlapping impact class clusters, i.e., u ki  = 1/4 for \(k = 1,\ldots,4.\) Values between 1.5 and 3.0 are commonly found in literature, but a value of 2.0 is most widely used (McBratney and Odeh 1997; McBratney et al. 1992; Burrough and McDonnell 1998; Fisher and Wood 1998).

The fuzzy k-means algorithm computes with the standard Euclidean distance norm, which induces hyperspherical clusters. Hence it can only detect clusters with the same shape and orientation since it assumes that clusters are hyperspherical with similar radii. But impact class clusters can be more irregularly shaped and have different sizes.

Fuzzy maximum likelihood estimation clustering

To circumvent the limitations of the fuzzy k-means, we have chosen the method of Gath and Geva (1989), sometimes referred to as fuzzy maximum likelihood estimation (FMLE). Unlike many other clustering algorithms, FMLE can accommodate elongated clusters and clusters of widely varying memberships, both of which may be encountered for journals in the multivariate indicator space, since it allows for ellipsoidal clusters of arbitrary extent.

The fuzzy cluster memberships u k i calculated using the FMLE algorithm are posterior probabilities which can be used for post classification processing as follows.

Let N be the number of journals in a given subject category under analysis (e.g., computer science artificial intelligence). Membership values are assigned to each ith journal so that all values sum to 1.0:

$$ \sum_{k=1}^{4} u_{ki} =1, \quad i = 1,\,\cdots,\,N.$$
(2)

In Eq. 1 the impact class membership u k i is replaced by the posterior probability P(k|g i ), following Bayes’ Theorem. The posterior probability is dependent on the conditional probability P(g i |k) of observing the value g i if it belongs to impact class k. The conditional probability is taken to be a multivariate normal density function. The FMLE clustering algorithm now replaces the Euclidean distance (of the fuzzy k-means) by the directionally-sensitive Mahalanobis distance (Bezdek and Dunn 1975):

$$d = \sqrt{ (g_i - m_k)^T s_k^{-1} (g_i - m_k)}$$
(3)

where m k is the cluster mean and s k is the fuzzy covariance matrix defined as:

$$s_k = \frac{ \sum_{i=1}^{N} u_{ki} (g_i - m_k) (g_i - m_k)^T} { \sum_{i=1}^{N} u_{ki}}$$
(4)

The membership value for a journal i to impact class cluster k is now defined as:

$$u_{ki} = \frac{1}{\sqrt{ | s_k |}} \exp \left[ \frac{-1}{2} (g_i -m_k)^T s_k^{-1} (g_i - m_k) \right] \frac{N_k}{N}$$
(5)

where N k is the number of journals for impact class k and N is the total number of journals in subject category; this ratio is calculated as \( \frac{1}{N} \sum_{i=1}^{N} u_{ki}. \)

For the FMLE algorithm, the covariance matrix s k and the mean m k of impact class k define its location and ellipsoidal extent in the multivariate indicator space given by the seven variables used in this analysis (i.e., Scimago Journal Ranking; H-Index; ISI impact factor; 5-Year Impact Factor; Immediacy Index; Eigenfactor Score; Article Influence Score).

In this study, we apply the FMLE algorithm as implemented in Abonyi et al. (2011). Because of the exponential distance dependence of the fuzzy cluster membership, the algorithm is very sensitive to initialization conditions, and can even become unstable. To avoid this problem we follow the suggestion of Gath and Geva (1989), and first obtain initial values for the membership values u ki by preceding the calculation with the fuzzy k-means algorithm (see “Fuzzy k-means classification” section), for which the class memberships can be chosen to follow an inverse square law. That is, it is recommended to use the resulting partition matrix of fuzzy k-means to initialize the FMLE algorithm.

As the membership values are a new continuous attribute, the distribution of these values for each journal impact class can be displayed by conventional methods of mapping providing useful information on classification uncertainty (e.g., see Figs. 2, 3, 4, 5, 6).

Results

Here we show the analysis performed upon subject category computer science artificial intelligence. Figure 1 shows the journal quartile ranking of ISI impact factor for artificial intelligence. Table 1 (second column) describe the numbers used in Fig. 1 to represent each journal of artificial intelligence (i.e., ranking by ISI impact factor).

Fig. 1
figure 1

Journal quartile ranking of ISI impact factor in computer science artificial intelligence

Full size image
Table 1 Subject category computer science artificial intelligence
Full size table

Our analysis is based on the multivariate indicator space described in “Study subject category and multivariate indicator space” section, and thus, the seven variables used in this analysis are: (1) Scimago Journal Ranking (SJR); (2) H-Index (H); (3) ISI Impact Factor (IF); (4) 5-Year Impact Factor (5IF); (5) Immediacy Index (II); (6) Eigenfactor Score (ES); (7) Article Influence Score (AIS).

One problem encountered is the absence of some indicator score in a particular dimension for a journal. To overcome this obstacle, the absent indicator value for jth journal was calculated by averaging the indicator scores of the immediate \(\{j-n,\ldots,\,j-1, \, j+1, \ldots, \, j+n\}\) neighborhood journals (sorted by ISI impact factor) at the dimension of interest; where n was set to 3 for this research.

In order to analyse the relationship between journal quartile rankings of ISI impact factor (at the 2010) in subject category computer science artifical intelligence and journal classification in four impact classes (i.e., highest impact, medium highest impact, medium lowest impact, and lowest impact), we firstly apply the FMLE algorithm to identify groups of journals sharing similar characteristics in the multivariate indicator space.

Figure 2 shows the four impact classes (i.e., highest, medium-highest, medium-lowest, and lowest impact) which were obtained using the FMLE classifier in the multivariate indicator space, as given in “Fuzzy maximum likelihood estimation clustering” section. As said above, Table 1 (second column) describe the numbers used in Figs. 2, 4, 5, and 6 to represent each journal of artificial intelligence (ranking by ISI impact factor).

Fig. 2
figure 2

Journal impact classes for artificial intelligence

Full size image

The problem is that even though the fuzzy clustering always tries to find the best fit for a fixed number of clusters (four clusters in our problem) and the parameterized cluster shapes (the FMLE clustering allows for hyperellipsoidal forms of the clusters), however this does not mean that even the best fit is meaningful at all. To evaluate whether the fuzzy partition fits to the journal data (i.e., cluster validity), we used the Partition Coefficient (PC) (Bezdek and Dunn 1975), that measures the amount of overlapping between clusters ∑ k (∑ i (u ki )2)/N [as implemented in Abonyi et al. (2011)].

The PC index values range in [1/nc,  1], where nc is the number of clusters. The closer to unity the index the crisper the clustering is, and thus, the closer this value is to one the better the journal data are classified (Bezdek and Dunn 1975). In case that all membership values to a fuzzy partition are equal, that is, 1/nc, the PC obtains its lower value. Thus, the closer the value of PC is to 1/nc, the fuzzier the clustering is. Furthermore, a value close to 1/nc indicates that there is no clustering tendency in the considered dataset or the clustering algorithm failed to reveal it.

To the fuzzy partition for subject category computer science artificial intelligence (following “Fuzzy maximum likelihood estimation clustering” section), we have that PC = 0.9797, and consequently, it indicates that journals of artificial intelligence (at 2010) were well classified in four impact clases using the FMLE algorithm.

For each one of the four quartiles of ISI impact factor (2010), Figs. 3, 4, 5, and 6 illustrate the fuzzy classification results in a map that shows the continuous spatial variation of the impact class membership values. Table 1 (five column) present the membership probabilities for each journal. The coordinates (x i , y i ) used to locate the ith journal of ISI impact factor (at the 2010) on impact classes maps (showed in Figs. 2, 4, 5, 6) are calculated as given by:

$$x_i = \sum_{k=1}^{4} u_{ki} \times \cos ((k-1) \times 45^{\circ})$$
(6)

and

$$y_i= \sum_{k=1}^{4} u_{ki} \times \sin ((k-1) \times 45^{\circ})$$
(7)

with u k i being the membership probability of the ith journal of ISI impact factor distribution to the kth class cluster (where k = 1,  2,  3,  4 is for highest impact, medium highest impact, medium lowest impact, and lowest impact class cluster, respectively).

Fig. 3
figure 3

Journal impact classes for Q1 journals of artificial intelligence

Full size image
Fig. 4
figure 4

Journal impact classes for Q2 journals of Artificial Intelligence

Full size image
Fig. 5
figure 5

Journal impact classes for Q3 journals of Artificial Intelligence

Full size image
Fig. 6
figure 6

Journal impact classes for Q4 journals of Artificial Intelligence

Full size image

From the membership values we can derive a hard classification by assigning the impact class label with the maximum membership value to each journal. This step is known as defuzzification. The resulting impact class label [i.e., Highest Impact (HI), Medium Highest Impact (MHI), Medium Lowest Impact (MLI), and Lowest Impact (LI)] are also illustrated in Table 1 (four column), for each journal in subject category computer science artificial intelligence (at 2010).

Now, from the four maps (for Q1,  Q2,  Q3 and Q4) illustrated in Figs. 3, 4, 5, and6, we can give a response to the initial points of interest: What is the link between quartile rankings of ISI impact factor and journal impact classes? Are there Q1 journals in a given subject category which are not of highest impact? and so on.

Only six Q1 journals of ISI impact factor were allocated to the highest impact class: Journals #1,  #2,  #4,  #5,  #9, and #21. It is interesting to note that Neural Computation ranks 21-st (out of 108 journals, Q1 quartile) in subject category computer science artificial intelligence.

Regarding the occurrence of Q1 journals in artificial intelligence which are not of higher impact, we have that five Q1 journals of ISI impact factor were allocated to the medium lowest impact class: Journals #7,  #22,  #25,  #26, and #27.

For example, Int. J. Inf. Tech. Decis. ranks 7th (Q1 quartile of ISI impact factor); while it ranks 90th (Q4 quartile of SJR), 88th (Q4 quartile of Article Influence), and 84th (Q4 quartile of H-index).

From Table 1, it also follows the occurrence of medium-highest impact journals which are not in first quartile of ISI impact factor. In particular Data Min. Knowl. Dis. ranks 60th (Q3 quartile of ISI impact factor) and Pattern Recogn. Lett. ranks 62nd (Q3 quartile of ISI impact factor). And Data Min. Knowl. Dis. and Pattern Recogn. Lett. are both allocated to the medium-highest impact class when using fuzzy clustering in a multivariate indicator space.

It is interesting to point out that Data Min. Knowl. Dis. ranks 18th (Q1 quartile of 5-years impact factor), 19th (Q1 quartile of Immediacy Index), 16th (Q1 quartile of Article Influence), and 17th (Q1 quartile of SJR).

Also, Pattern Recogn. Lett. ranks 8th (Q1 quartile of Eigenfactor Score), and 17th (Q1 quartile of H-index).

There exists another interesting result regarding Q2 journals of ISI impact factor. From Table 1 we have that a Q2 journal of ISI (J. Amb. Intel. Smart En.) is allocated to the lowest impact class in the multivariate space. In fact, J. Amb. Intel. Smart En. ranks 93-rd (Q4 quartile of Immediacy Index), 100th (Q4 quartile of Eigenfactor Score), and 69th (Q3 quartile of Article Influence).

The fuzzy clustering algorithms allow impact classes to overlap, thereby accommodating for uncertainty related to impact classes transition zones (i.e., the confusion about the impact class attribution for a journal) and vagueness in impact classes definition (e.g., what is a medium-lowest impact journal?). The resulting impact classes will be optimal in the sense that the multivariate within impact class variance is minimal.

Multidimensional clustering provides an elegant method for flexible, continuous, and automatic clustering of journals in a subject category along multiple dimensions (using several prestige indicators). This can result in significant improvements in the performance of research evaluation. That is, multidimensional clustering enables a subject category to be physically clustered on more than one indicator, or dimension, simultaneously.

Journal prestige indicators play an important role in the deliberations of the review panels and have the added advantage of being more cost efficient. With multidimensional clustering, these benefits are extended to more than one dimension, or prestige indicator. In the case of journal performance analysis involving any, or any combination of, specified dimensions (indicators) will benefit from multidimensional clustering.

Regarding the number of dimensions (prestige indicators) used to produce the fuzzy classification, here we consider indicators with different degrees of correlation among them, but which should be used for a distinct analysis of structural changes at the score distribution of journals in each subject area. In fact the seven prestige indicators were proposed in the Literature to this aim, and all of them are actually used to perform the journal ranking (Journal Citation Reports 2011; SCImago portal 2011). In any case, multidimensional fuzzy clustering provides a panel of experts with an automatic tool which can be used to objectively rank the journals in Artificial Intelligence or any other subject category in multidimensional space involving any combination of prestige indicators.

For instance, Table 2 shows impact class differences with respect to results in Table 1, when using only five indicators: IF, 5IF, II, ES, AIS. And, even though we delete Scimago indicator (SJR) from the original seven indicators, the resulting classification was not quite different from what showed in Table 1. By the contrary, when using only one prestige indicator (impact factor) to perform the fuzzy classification, the resulting classification would be quite different as illustrated in Table 3.

Table 2 Impact class differences with respect to results in Table 1, when using only five indicators: IF, 5IF, II, ES, AIS
Full size table
Table 3 Impact class differences with respect to results in Table 1, when using only one indicator: IF
Full size table

Conclusions

From the results in Table 1 it follows the occurrence of both first quartile journals (of ISI impact factor) which are not of highest impact class as well as of medium-highest impact journals which are not in first quartile of ISI impact factor.

As can be seen from Table 1 and Fig. 6, all the Q4 journals of ISI impact factor were allocated to the lowest impact class. Hence we can conclude the link between Q4 of ISI impact factor and lowest impact class using the fuzzy classifier in the multivariate indicator space, at least in subject category computer science artificial intelligence.

Also, from Table 1 and Fig. 5, it follows the link between Q3 of ISI impact factor and two different impact classes: Medium-lowest impact and lowest impact class.

Table 1 and Fig. 4 show the link between Q2 of ISI impact factor and two different impact classes, that is, medium lowest and medium highest impact class.

And Table 1 and Fig. 3 illustrate the complex relationship between Q1 of ISI impact factor and journal impact classes in the multivariate indicator space: Q1 journals are allocated to three different impact classes (highest impact, medium highest impact and medium lowest impact class).

To sum up, this analysis shows that the quartile ranking of journals is a complex field. The results of cross journals comparisons depend on the chosen indicator (i.e., ISI impact factor, SJR, H-Index, Eigenfactor Score, etc). Therefore several indicators should be used for a distinct analysis of structural changes at the score distribution of journals in a subject category. Here we have proposed that it can be performed in a multivariate indicator space using a fuzzy classifier.

We are developing a publicly available suite of Web-based tools designed to facilitate analysis of subject categories using the proposed approach. It will be freely available to the scientific community at: http://cvg.ugr.es/scientometrics.