Introduction

The monitoring of scientific activity started receiving considerable attention after the Second World War when it became indispensable that science brought effective results for defense and competition among nations (Gingras 2014). In this context, the countries organized their research incentive agencies, creating evaluation criteria for the allocation of resources. These criteria, over time, were also incorporated for the granting of scholarships to researchers, evaluation of stricto sensu graduate programs, hiring and evolution of the teaching career.

The first Brazilian agency of this nature was created in 1951 under the name of the National Research Council (CNPq—Conselho Nacional de Pesquisas). One of its objectives was the promotion of scientific research and the training of human resources for research, through scholarships and policies for the improvement of graduate courses.

CAPES—Coordenação de Aperfeiçoamento de Pessoal de Nível Superior was established in 1964. Since its inception, this agency focuses on the improvement of higher education personnel and of the teaching and research standards. Such purposes already indicated the responsibility of the new entity in the monitoring of scientific production. In 1981, CAPES became the manager of postgraduate policies in Brazil. CAPES then adopted instruments for the evaluation of courses that include the scientific production of teachers and students as one of the quality requirements.

CAPES invited foreign specialists to analyze such production. They worked from 1996 to 1997, and as a result, the basis “Qualis” was created in 1998.

Qualis is a categorized list of journals, both foreign and Brazilian. The categories, or strata, are defined on a regular basis by each area of knowledge, taking into account the venues where the production of graduate courses is published. It is noteworthy that such scientific production, in the light of Qualis, weights 35% of the grade every graduate course receives. This ranking has, thus, a strong influence on the perceived quality of the Brazilian scientific production.

Although Qualis has been formulated exclusively for the stratification of the intellectual production of graduate programs, and explicitly not of individuals, it induces the choice of the vehicles in which researchers wish to publish their works. As such, Qualis has a substantial impact on the venues where people submit their work.

Although Brazilian scientific production has expanded in the last five decades, representing 2.59% of the world total (Ramos 2018), the publication of this production in indexed journals is still modest. In the 2013 and 2014 catalogs, of the 313 national journals indexed in SCImago and 141 in Web of Science (WoS), only 32% were indexed in both databases (Packer 2014). Indexing is one of the criteria for the evaluation of journals by CAPES.

CAPES establishes that journals must be serialized, indexed in databases, and have an ISSN to belong to Qualis. International journals must also be indexed in SCImago, and in the Social Sciences Citation Index from WoS. It is recommended, although not compulsory, that Brazilian journals be indexed in SCImago; nothing is established for Brazilian journals regarding WoS.

As an academic product, publications promote the scientific exchange necessary for internationalization in Brazilian higher education, especially in stricto sensu graduate courses. In this context, the “National Plan for Graduate Studies 2011–2010” of CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior 2010) indicated the relevance of developing strategies for international insertion, not only concerning the training of human resources, but also the generation of knowledge and innovation in the country. The report “Special Monitoring Commission” (Comissão Especial de Acompanhamento do PNPG 2011–2020 2017) indicated, among its recommended actions for internationalization, the need to compare the quality of national scientific and technological products with other countries.

While most of the World uses WoS and SCImago, Brazilian researchers are implicitly compelled to adhere to Qualis. At the same time, CAPES pushes towards internationalization as one of the most critical metrics for the assessment of graduate courses. This work verifies if these two forces point in the same (or very close) directions, as they should be to promote a sensible policy.

The focus of this work is the set of indexed journals in the area of Education. The objective of this work is to elucidate if Qualis Education aligns with international quality metrics and, thus, if it promotes internationalization.

Theoretical grounds

Qualis base is not focused on the indexing of journals. It consists of a classification system of scientific journals, arising from the need to measure the production of Brazilian stricto sensu courses. Each area has its own Qualis so the same journal may have different qualifications in different areas.

The classification has seven strata since 2007 (in decreasing order of quality): A1, A2, B1, B2, B3, B4, B5, and C. The latter comprises publications that do not meet the minimum criteria of the area (Comissão Especial de Acompanhamento do PNPG 2011–2020 2017).

Each area defines its criteria for assigning venues to Qualis strata. HIndex, Pagerank, Journal Impact Factor (JIF), Article Influence Score (AIS), and Eigenfactor Score (EIS) are among the quantitative metrics of quality employed for such purpose. Except for Pagerank, which is computed by Google, these metrics are provided by the leading international scientific bases: Scopus/SCImago, and WoS, which publishes the Journal Citation Reports (JCR).

The first level of measuring scientific production is productivity, i.e., publishing. The second level is popularity, which consists in being cited by other researchers. The third level is prestige, which measures the influence of the scientific production. These levels are increasingly difficult to assess. The first requires just counting papers, the second needs a cross-referenced data base, and the third depends on a global view of the citation network.

JIF and HIndex are “popularity” metrics and they situate at the second level, as they only take into account the number of citations. Pagerank, Article Influence, and Eigenfactor are “prestige” measures, at the third level of assessment, since they consider both the number of citations and the originating source. The reader is referred to the work by Franceschet (2010) for details.

Considering this context and focusing on Education, the objective of this paper is to analyze whether the Qualis base aligns with international quality metrics or not.

For the area of Education, Qualis-indexed vehicles must have the serial publication, ISSN and indexing in at least one database. For the A1 classification, CAPES requires the venue to be on six bases. In the case of international journals, to reach strata A1 or A2, indexation is required in the SCImago/Scopus or Social Sciences Citation Index from WoS databases (Comissão Especial de Acompanhamento do PNPG 2011–2020 2017). These bases are among the significant classification systems at the global level (Rocha-e-Silva 2010; Colepicolo 2015).

Related works

In search of previous investigations on the theme explored here, we considered works that addressed the Brazilian scenario concerning the indexing of journals and the impact of research and researchers. In this context, without the intention of a systematic review, we identified fourteen studies published between 1992 and 2020.

de Oliveira et al. (1992) conducted a study focused on Nutrition using a sample of journals indexed by the Science Citation Index (SCI) from 1985 to 1990. The authors showed that 72.5% of articles did not receive citations, and 27.5% were cited at least once. The analysis indicates that these works did not reach the international community, and that of the 305 articles analyzed, only 84 were cited at least once, and that 13 were self-cited. However, there was an increase in the number of citations during the period investigated.

The work by Wainer et al. (2009) presents a comparative study of Brazilian scientific production in Computer Science with other countries, using an analysis based on international impact factors. The authors point out that Brazilian production in this field is the largest in Latin America. Concerning the world scope, the quantity of this production is similar to the majority of countries.

Gracio et al. (2013) analyze the impact of the Brazilian scientific production in the field of Dentistry from 2000 to 2009, using SCImago. The paper compares Brazil to the thirteen most affluent countries in the area, with at least 2% of the World scientific production. In conclusion, the authors point out that Brazil is the only country in Latin America among the most productive in the world in Dentistry, showing growth in both the visibility and impact of their publications.

Reverter-Masía et al. (2014) present a comparison of Brazilian and Spanish journals on Physical Education in the WoS database. The authors perform descriptive, comparative, and correlational analyses of the documents, sampling 373 publications: 108 Brazilian, and 265 Spanish. The results indicated that, on the one hand, the Spanish researchers obtained HIndex of 2.6 in WoS, and 3.1 in Scopus. On the other hand, Brazilians obtained HIndex of 6.05 in WoS and 7.5 in Scopus. Most researchers in both countries have published in Sports and Exercise scientific journals.

Strehl (2016) analyze works of Brazilian researchers from 19 fields of knowledge, between 2002 and 2011: a total of 85,082 articles in the Web of Science. The authors present a methodology that allows the analysis of productions and impacts of areas in emerging communities, using aspects of origin and collaboration. Among the results, it was pointed out that the published works in international journals have greater impact than the national venues.

Soares (2014) study the Qualis database in the areas of Administration, Accounting, and Tourism, regarding the impact of 15 scientific journals in those fields. The authors measure the impact by the number of citations. The results indicated that the citation indicators are not aligned with classification of the journals in five strata of Qualis.

schmidt (2019) study Qualis for Psychology regarding internal coherence (if subareas follow similar criteria) and external adherence to international measures of quality. They conclude, owing to inconsistencies in these two regards, that the advisory committee of the Psychology area, responsible for producing the Qualis list, does not use internationally accepted indexers. The authors criticize such lack of adherence, in particular for its negative consequences on interdisciplinarity and internationalization.

de Souza et al. (2018) analyze publications by their impact in the field of Economics in the 2013–2016 Qualis. The authors point out the increase of journals in the higher Qualis strata that have no research agenda bias: some do not even have peer review. This is because, although there are criteria presented by the areas, there is subjectivity in the allocation of journals within the strata.

Pires et al. (2020) analyze the Qualis periodicals between 2007 and 2016 from eight areas (Agricultural Sciences, Biological Sciences, Medicine, Computer Sciences, Engineering, Education, Administration, Literature and Linguistics) with respect to the SJR indicator. As a result, the authors report that Qualis directed the publication to journals with a low impact factor. According to the authors, although there has been an increase in Brazilian production, this has not been accompanied by their impact.

Perlin et al. (2018) study predatory journals and their inclusion in Qualis, covering the period from 2000 to 2015. The examination also consider the profile of their authors. The conclusions indicate an exponential growth of these journals in the last 5 years, receiving more publications than the non-predatory ones in Qualis.

Ferreira et al. (2013) analyze journal citation data from the Web of Science of six journals in the Biological area. The authors conclude that, although they belong to different Qualis strata, there are no significant differences among their indicators of performance.

This research stands out for analyzing the Qualis Education base in its entirety, relating the impact of journals as measured by JIF and SJR, and their categorization in Qualis strata.

We conclude this section referring the reader to the work by Kulczycki and Rozkosz (2017) which comments national (French, Australian, Serbian, Norwegian, Taiwanese, Colombian, Dutch and Brazilian) solutions. The authors focus on the Polish Comprehensive Evaluation of Scientific Units, which is based upon the Polish Journal Ranking.

Methodology

This study aimed to analyze Qualis for the area of Education and its relationship with international impact factors. We compared and correlated the journals’ bibliometric indicators from Qualis, Scopus/SJR, and JCR/WoS databases. The guiding questions were:

  1. Q1

    Is the Qualis basis for education aligned with international impact factors?

  2. Q2

    Does this base promote the internationalization of Brazilian research?

  3. Q3

    Does this basis reflect the impact of journals in the field of Education?

We listed all the Qualis journals from Education for the period 2013–2016. This is the latest available Qualis for the area. Such a table has 4203 entries with ISSN, journal title, and stratum. Figure 1 shows the percentage of journals in Qualis Education per stratum.

Fig. 1
figure 1

Percentage of journals in Qualis Education per stratum

Full size image

We eliminated in the subsequent study the journals in stratum C because they did not meet the minimum criteria listed by CAPES (Comissão Especial de Acompanhamento do PNPG 2011–2020 2017), resulting in 2914 venues.

We then performed individual queries in the JCR/Wos and SJR/Scopus databases. We extracted JIF, Eigenfactor, and Article influence from WoS, and HIndex from SJR. The JCR/WoS data base was populated with information from the categories “Education and Educational Research”, “Education, Scientific Disciplines” and “Special Education” to form the group “WoS-Education” with 318 indexed journals. We also extracted the field “Education” from SJR/Scopus, consolidating a group called “SJR-Education” with 1262 journals.

We then contrasted these two data sets with the Qualis base.

Results

The following analysis observes two main aspects: (1) the proportion of journals from each data base in each stratum (“Proportions” section), and (2) statistics of the measures of quality per stratum (“Measures of quality” section).

Proportions

Qualis has 131,274 classified journals, of which 4203 belong to Education, corresponding to 3.20% of the total. Table 1 shows the number of journals per stratum, along with the number of these that belong to either JCR/WoS or SJR/Scopus.

Table 1 Number and proportion of journals in the Qualis (Q) and Qualis Education (QE) databases, along with the values of JCR and SJR journals indexed in QE; (*) does not apply to the scope of this research
Full size table

For short, in the following we denote “Q” the Qualis data base; “QE” the set of all Qualis Education journals; “QE.JCR” the set of all journals that are both in QE and in JCR/WoS; “QE.SJR” the set of all journals that are both in QE and in SJR/Scopus. Notice that the number of journals indexed in the QE.SJR is almost double than those in QE.JCR.

Figure 2 shows the proportion of venues in each stratum (from C to A1) in each base (Q, QE, QE.JCR, QE.SJR). The proportions are base-wise; for instance, there are 361 QE-JCR journals (100%, cerulean bars), of which 121 (33.52%) are A1, 159 (44.04%) are A2, 58 (16.07%) are B1, 6 (1.66%) are B2, 7 (1.94%) are B3, 4 (1.11%) are B4, 6 (1.66%) are B5. The differences between bases per stratum appear significant and will be further assessed.

Fig. 2
figure 2

Proportion of venues per stratum in each base

Full size image

Figure 3 shows the distribution of QE.JCR and QE.SJR journals per Qualis stratum. We notice that they are very different, and that neither follows the Qualis proportions presented in Fig. 2.

Fig. 3
figure 3

Distribution of journals indexed by JCR/WoS and by JCR/WoS per QE stratum; NA = Journals not indexed in any of the Qualis-Education strata

Full size image

Table 2 informs the relative deviations of each stratum, in absolute value and percentage, between (a) Qualis and Qualis Education, (b) Qualis Education and JCR, and (c) Qualis Education and SJR.

The only close match between these two bases occurs in stratum A2. The most significant differences between QE.SJR and QE.JCR are in strata B2 and B5.

Table 2 Deviations (absolute value Of the percentual difference) of QE from Q, of QE.JCR from QE, and of QE.SJR from QE
Full size table

The closest match is between Q and QE at stratum A2 (9.80%), but the other strata exhibit deviations that range, roughly, between 20 and 130%. A \(\chi ^2_7\) (chi-squared with seven degrees of freedom, owed to the eight categories) test for the null hypothesis that these values arise from the same distribution returned a negligible p-value. We have, thus, significant evidence that QE does not obey the proportion of venues per stratum that Qualis follows. QE.JCR and QE.SJR are unrelated to QE, with deviations up to more than 750% (stratum B5 for QE.JCR), and about 380% (stratum A1 for QE.SJR).

These results provide strong evidence that the distribution of journals in Qualis-Education (QE) does not follow the proportion per stratum of the full Qualis (Q) base. The distribution per stratum of those journals in JCR and in SJR that belong to QE is still further.

Measures of quality

In the following, we will analyze only QE journals belonging to the A1–B5 strata, i.e., we will not consider those classified as “inadequate” (stratum C). This subset amounts to 2914 journals. QE journals will be contrasted with those in WoS-Education (318 journals), and SJR-Education (1262 journals).

Of those 2914 journals, 277 are indexed by JCR, representing 9.50%, and 498 belong to SJR, representing 17.08%. These data are presented in Table 1 and in Fig. 4.

Fig. 4
figure 4

Distribution of journals in the Qualis-Education base, and the JCR and SJR indexes

Full size image

The Journal of American Medical Association (JAMA) is the venue with highest impact factor in both WoS-Education and SJR-Education; it is classified as A2 in QE. By extracting only the 20 journals with the highest JIF (Table 3), we observe that only two achieved the Qualis A1 classification: Scientific Reports and Journal of Research in Science Teaching.

Table 3 Top 20 QE journals, according to their JCR JIF
Full size table

None of the twenty journals with the highest Hindex/JCR, is classified in the A1 stratum of QE, as shown in Table 4.

Table 4 Top 20 journals of QE, according to their HIndex/SJR
Full size table

There are no Brazilian journals in the top-20 list of either JCR/WoS or SJR/Scopus; cf. Tables 3 and 4. Another important aspect is that, of these top-20 QE journals, only Research in Teaching of Science and Informatics and Education are in the field of Education, since both are in WoS-Education and the last one is in SJR-Education.

Although there are several Brazilian journals scored in JCR and SJR, the Brazilian journal directed to Education and with the most significant impact factor is the Revista Brasileira de Ensino de Física (Brazilian Journal of Physics Teaching), classified in stratum B1; its JIF is 0.099. Another journal scored in the field of Education, but related to Health, is Revista CEFAC (Speech, Language, Hearing Sciences, and Education Journal). With a scope focused on speech therapy, this journal has stratum B1 in Education, with an SJR value of 30. When analyzing the vehicles focused on Education with SJR, the journal Educação e Sociedade, under stratum A1, has the highest score.

Thus, there is an imbalance between the Qualis strata and international impact factors. Analyzing the average JIF value by Qualis stratum, we identified that the group with the highest mean stratum is B3. This is due to six journals in the group, among them, Emerging Infectious Diseases, which has a JIF of 8.222. The following strata by the mean JIF are A2, B4, B1, and finally A1. Figure 5a shows the notched boxplots of the JIF values of each stratum. The notches represent approximate confidence intervals of the median at the 95% confidence level. The discrepant point in stratum A2 corresponds to the journal JAMA, which has a much higher impact than the other journals. Figure 5b shows the same analysis, but with the SJR values. The highest mean impact is in the A2 stratum, followed by B3 and B4, with the A1 stratum in the fourth position.

Fig. 5
figure 5

Notched boxplots of JIF and SJR of QE journals per stratum

Full size image

The information presented in Fig. 5 provides quantitative evidence that there is no significant difference in the median value per stratum, neither in JIF nor in SJR. The confidence intervals overlap, and there is no visible trend that one could associate between JIF or SRJ and stratum.

Comparing Fig. 5a and b, we notice that the former has less variability (as measured by the interquartile range) than the latter. Such an effect may be due by both the allocation procedure and the larger number of journals in the SJR-Education base.

Figure 6 shows the dispersion diagram between Article Influence and Journal Impact Factor of QE journals. This plot does not show the points corresponding to JAMA and Journal of Ethnobiology and Ethnomedicine, because they behave as outliers and make the visualization of the other points difficult. It is noteworthy that none of these journals deal with Educational content. This plot shows no evidence of clusters, as one should expect if there was a significant association between quality and QE strata.

Fig. 6
figure 6

Article influence versus JIF of QE journals

Full size image

We have thus shown evidence that the distribution of journals with either JCR/WoS or JCR/WoS in QE strata is not correlated with their measured quality.

Discussion

The impact of a journal implies its internationalization, which is a standard adopted worldwide. Currently, this theme has been gaining prominence in higher education and research institutions, which have begun to consider the production of their researchers in indexed journals. In the case of the field of Education, although the area document (Comissão Especial de Acompanhamento do PNPG 2011–2020 2017) requires journals to be indexed for classification in the Qualis database, the most relevant international databases are not being included in the requirements.

Thus, as the results show, there is a relatively low number of journals with a real impact on scientific dissemination in the field of education. Only 9.50% of the journals are indexed in JCR and 17.08% in SJR. Even considering that SJR indexes more periodicals, the quantity is still small.

The general picture, when analyzed by stratum, is that A1 journals have little or no prominence in the indexers. However, A2 journals stand out, as they are among the venues with impact on both SJR and JCR. But most of these journals are not in the field of Education and are not, therefore, indexed in WoS-Education or SJR-Education.

Another problem identified is that there are several journals under the Qualis-Education classification that are not related to Education. This fact introduces distortions on the quality measures of journals in this field.

In view of the guiding questions of our study, we conclude that there is little alignment between the stratification of the Qualis-Education base and international quality metrics, given that the journals with the most significant impact are those classified as A2, when they should be A1 journals. Also, there is no increasing trend of JCR and SJR to the B5–A1 strata; as such, Qualis-Education does not promote the search for better venues. This already indicates the importance of the qualification of journals concerning their adherence to WoS-Education and SJR-Education.

Although it is possible to note an increase in the number of journals for the field of Qualis-Education, it is necessary to align this growth with a more integrated assessment of international standards. Brazilian journals need to promote internationalization and the impact of national production since these measures the worldwide acceptance of the research delivered by the Brazilian academic community. As a positive consequence, there will be higher interest from international researchers in the Brazilian production.

We have shown evidence that Qualis Education strata are weakly related to the standard measure of popularity, namely JIF. A further step should stimulate influential publications, for instance in journals with high AIS.

Therefore, it is up to the Qualis Education Committee to define integrated policies that adjust the national and international classification, aiming at having a respectable position in the world scientific scenario.

Conclusion

We presented some aspects related to the impact of scientific journals classified in the Qualis database for the area of Education. We identified that, although the quantitative expansion of classified journals has occurred, the impact of these vehicles did not follow this evolution. The Qualis-Education stratification has no international alignment. There are journals with high impact and low Qualis classification, and periodicals that are in the upper strata are often not even indexed.

Thus, as CAPES evaluates graduate programs using the Qualis database, the lack of alignment between impact, internationalization and classification will have repercussions not only on the production of researchers but also on the graduate courses to which they belong.

In the current scenario, researchers in the area of Education will continue to publish their work in journals with little or no international visibility.

Regarding our leading questions:

  • Is the Qualis basis for education aligned with international impact factors? The answer is no, as the proportions are disparate and the international measures of quality do not correlate with the strata.

  • Does this base promote the internationalization of Brazilian research? The answer is no, since the number of impact journals on this database is relatively low.

  • Does this basis reflect the impact of journals in the field of Education? The answer is no, since journals in lower strata often have better indicators than those in the top Qualis.

As for the three stages of development of the quality of scientific outcomes, namely productivity, popularity, and influence, currently, Qualis Education is only concerned with productivity (number of papers).

To ensure that the Qualis stratification does not distance itself from the international context, it is necessary to adopt criteria that privilege factors such as adherence to WoS-Education and SJR-Education, especially for the A1 and A2 journals, considered of highest quality. One initiative is worth mentioning: some Universities subsidize the cost of publications provided they are in journals with good JCR indicators (the Universidade Federal de Alagoas, for instance, required in 2019 any of JIF, AIS, 5-Year JIF, or Eigenfactor to be above the median of the category). The result of such initiatives will be greater visibility and influence of the Brazilian scientific outcomes.

We thus conclude our study with a call to the Brazilian Education community to aim at publishing in internationally-recognized venues, to use internationally-recognized parameters to assess the quality of journals, and to push policies aligned with such objectives.

The data and code used in this analysis are available at https://github.com/gomesrocha/ANALYSIS_IMPACT_OF_THE_BRAZILIAN_BASE_QUALIS_EDUCATION.