Introduction

More and more countries introduce performance indicators in systems for allocation of research funding (Jonkers and Zacharewicz 2016; Pajić 2015). Often such performance-based research funding systems (PRFSs) employ bibliometric indicators thus equating research performance with quantitative measures based on publication and/or citation counts, in other words, with measures of research output and impact respectively (Moed et al. 1985). In these systems, it is implied that funding flows to those who perform best (Hicks 2012). Indeed, the Swedish Government introduced in 2009 a funding system, whose explicit purpose was to encourage higher education institutions to find research-profiles that give a competitive advantage in relation to others (Ett lyft för forskning och innovation 2008). In this system, one of the indicators which structure the competition between these institutions is the number of publications and citations (Aldberg and Jacobsson 2014).

This assumption of funding-flow privileging high performance forms the core problem with bibliometric indicators of research performance. It is suggested that the use of PRFSs may have potential implications for knowledge produced in such performance-oriented contexts (e.g., de Rijcke et al. 2015). Studies by, among others, Butler (2003, 2004), Evaristo Jiménez-Contreras et al. (2003), and Henk Moed (2008) on Australian, Spanish and UK contexts respectively, indicate that PRFSs lead to changes in publication practices. If there is a system that favours publications of type X, then it is reasonable to assume that scholars will adapt their behaviour by writing more publications of type X. On the other hand, there are studies suggesting that claims concerning effects of PRFSs are based on studies with methodological or conceptual limitations (see Osuna et al. 2011 as well as the recent PRFS debate in the Journal of Informetrics: e.g., Gläser 2017). Thus causal claims linking publication patterns and PRFSs appear unjustified, while knowledge of the consequences that the use of PRFSs may have on knowledge systems remains limited (Hicks et al. 2015; de Rijcke et al. 2015).

The way PRFSs are used suggests that a broad consensus on research performance is nowadays routinely achieved (e.g., Rabovsky 2014; Hillman et al. 2015; Dougherty et al. 2016). However, we believe that the ongoing debate on PRFSs and publication practices may be furthered by research that looks in more detail into the ways research performance is classified and enumerated. Therefore, we here pursue a bibliometric analysis of changes in publication patterns which is complemented with an analysis of the indicators themselves. Drawing on a social–epistemological approach, we discuss how the observed changes in publication patterns depend on existing performance measurements and how the assumptions implicit in the PRFSs are imposed on the targeted research groups. This approach also helps us to foreground that, contrary to the key assumption in PRFSs, there is more than one way to understand performance across different knowledge domains. To illustrate this, we explore assessments of educational research within the University of Gothenburg (GU) in Sweden (2005–2014). We look, more particularly, at the two different PRFSs that were introduced in 2009 both at the national and the institutional level. We look at the changes that take place in publication patterns within educational research at GU in Sweden (2005–2014) and the ways these changes relate to implicit assumptions about research goals—both within the two PRFSs that operate in the context and the field of educational research in Sweden. In addressing these issues we highlight problematic aspects of PRFS as an incentivising mechanism.

The Swedish context in relation to PRFS is not extensively explored. A bibliometric analysis of publication patterns (2006–2013) in the Faculty of Humanities at the University of Uppsala showed that the distribution of publication types (monographs, book chapters, and peer-reviewed journal articles) remains stable over the whole period of time, while the share of the use of Swedish as opposed to English in scholarly communication is on a declining slope (Hammarfelt and de Rijcke 2015). Noteworthy, it is pointed out that it is not possible to attribute the observed change to the use of metrics in funding allocation models due to the range of other potential factors of influence. Bibliometric analyses for other faculties, universities or other knowledge domains, to the best of our knowledge, do not exist.

The structure of this paper is as follows: we begin with theoretical considerations guiding this study and continue with an overview of the literature on publication patterns in the field of educational research and an introduction to the targeted groups. We highlight some main features of educational research at GU in Sweden and provide an outline of the two PRFSs: the system referred to as the Swedish model that is implemented at the national level and the system derived from the national PRFS in Norway (Norwegian Publication Indicator, NPI) that is used in GU. We then describe the research method and data sources used in this study. In the findings section, we present a detailed overview of changes in the publication patterns. In the final section, we discuss the findings contrasting research goals implicit in bibliometric indicators and goals one may encounter in the field of educational research.

Framing publication patterns

Building upon insights from a social–epistemological framework, we look at the context in which knowledge practices are embedded (e.g., Fuller 2002). This framework, more particularly, emphasises the dialectical aspect of knowledge, meaning, that knowledge is knowledge not due to some abstract considerations residing in a social vacuum. Instead, knowledge is a relational concept, which acquires its meaning in being recognised as meaningful in a particular community. The relevant community thus also constrains what may or may not be considered as ‘relevant’ or ‘legitimate’ research and ‘relevant’ or ‘legitimate’ knowledge.

Seen in this light, the introduction of bibliometric practices may have considerable impact upon research, while their underlying principles give way to a particular view of research—and its output. Almost self-evidently, bibliometric indicators reify a particular goal-orientation within research. This is most evident in publication type classifications and further hierarchies and differentiations that are made in PRFSs. It is implicitly assumed that results of research are visible, first of all, in particular data sets. But the rewards often also vary across specific publication types (and their specific characteristics), producing a hierarchy of publication types. The search for optimal performance seems to prioritise the kind of research that is most likely to lead to the greatest possible number of publication types associated with the highest possible rewards. For example, a system that rewards indicators derived from data on the number of articles in peer-reviewed journals reifies an assumption that research ought to result in journal articles. Moreover, an emphasis is set on quantity (regardless of other more fine-grained differentiations).

For sure, researchers working in such a context may, or may not, act in correspondence with the goals implied. But most researchers have become acquainted with the measurement and classification principles that are used in the context they are active in. They know what counts; they also know, for example, how to make a distinction between scientific, peer-reviewed, and ‘other’ articles. This is especially the case in institutions where authors have to report data on their research output or when indicators in PRFSs are linked up with other rewarding practices (on the latter see Aagaard 2015). In this sense, the introduction and diffusion of PRFSs strengthen the link between different knowledge practices and particular expectations regarding output.

Not surprisingly, the introduction and rapid diffusion of PRFSs based on bibliometric indicators has especially been contested within the Social Sciences and Humanities (SSH). Publication practices in SSH tend to be more diverse than those in natural sciences (Hicks 2004). SSH scholars communicate not only through international journals (indexed in international citation databases), but also make use of national journals and media addressed to wider audiences. In the same way, a prominent role in SSH is typically attributed to monographs and other book publications. Such characteristics have been identified in several national and disciplinary contexts (e.g., Engels et al. 2012; Sivertsen and Larsen 2012; van Leeuwen et al. 2016). For SSH scholars, the number of peer-reviewed journal articles as an indicator of research performance in fact often seems to imply a singular view whereby one, first, is required to assume that it is meaningful to equate research performance with this particular indicator and, second, that it is meaningful in the pursuit of research to strive for an ever greater number of such articles (why else allocate more funding to those who score higher?). Despite the existence of different opinions within their disciplinary communities, other forms of output do no longer seem to count.

Within this framework, we look at the changes that have taken place in publication patterns in educational research in a 10-year period within which PRFSs (with a hierarchy of rewards linked to a formally identified set of publication types) was introduced. Consequently, we discuss how publication pattern changes in the period after the introduction of the PRFSs can be interpreted and how these changes are dependent on the adopted indicators.

Context

Characteristics of educational research

In a historical sense, educational research has long been characterized by a focus on understanding the various phenomena surrounding schooling and education. In the view of its own practitioners, its primary goal was to provide a scientific base for teacher education (Vanderstraeten 2011; Vanderstraeten et al. 2016). Even in more recent years, educational research is often characterized by a strong orientation to its (potential) professional users: education practitioners, policy-makers, students, parents and any other social groups with an interest in education. But this orientation might also be at odds with the expectations of other academics, of peers (whether working in educational research itself or in other disciplines or specialisations).

The diversity of the audiences to which educational research caters is the main reason why we choose this field as a case study. One may expect that the strong ties with varied social groups will result in different publication practices and hence render the employment of bibliometric indicators of research performance problematic.

Indeed, studies exploring publishing patterns of educational research suggest broad cross-national variations, which might be reflective of different orientations towards relevant audiences. For example, a study from Norway (2005–2009) shows that book chapters are the most popular category of publication output, accounting for 49%, while books in this field account for 5%. Peer-reviewed journal articles in educational research constitute about half (46%) of the total volume, but only 9% of these publications are indexed in WoS (the averages for SSH being 20 and 11% respectively). In terms of language, 33% of the publications within educational research were published in English, while the rest was published in the national language (Sivertsen and Larsen 2012). This pattern is not characteristic for all SSH. In these fields in general, the average share of publications in English is about half of the total volume.

In Germany (2004–2006), as a study from Dees (2008) indicates, journal articles accounted for about one-third (33.4%) of the total number of publications. The German emphasis on book publications is particularly strong: while about half of the total number of publications within German educational research (2004–2006) conducted at 15 institutions were book chapters (46.7%), the share of books was 14.8%. Moreover, in that dataset, 88.1% of all publications were written in German, although the share of English-language publications varied among institutions. In one institution, about half of all publications were in English (Dees 2008). In contrast, in Flanders (Belgium; 2000–2009), peer-reviewed journal articles constitute 92% of all publications in their dataset (Engels et al. 2012) thus suggesting that a quite different publication practice operates in Flanders, relative to both Germany and Norway.

In Sweden itself, educational research has over time also evolved into a rather diverse field (Englund 2006). This diversity is mirrored in the GU Faculty of Education where, for example, large-scale quantitative studies informed by psychological theories can be found next door to qualitative inquiries guided by sociological concerns (e.g., Angervall and Gustafsson 2014; Yang Hansen and Gustafsson 2016) and research closer to educational practice (e.g., Svensson 2013). A study exploring the practice of educational research in Sweden has, moreover, distinguished between three main types of institutions: practice-oriented, research-oriented and a combination of the two (Öhrn and Lundahl 2013). The practice-oriented type typically has closer ties with teacher education, educational practice and policy making, while the research-oriented type is more oriented towards the national and international scholarly community. It can be easily imagined that these differences are also reflected in publication practices. Within more practice-oriented institutions, often priority is for those means of communication that facilitate more direct engagement with schools and society in general. In contrast, in the research-oriented institutions, peer-reviewed international journals tend to be the channels with the highest priority. Thus, in this context, within the field of educational research one can identify variations in more general ways of understanding and doing research, and consequently of appropriate means for communication. For example, in the study by Eva Silfver (2013), one of the respondents pointed out that the (short) format of a peer-reviewed journal article is not suitable for providing an argument of quality (pp. 91–92). Such considerations may point to mismatch between agendas within research practice and implicit priorities set within PRFSs that could be apparent also in publication patterns.

An earlier bibliometric analysis of Swedish educational research (2004–2008) shows publication patterns comparable to the patterns identified in Norway and Germany (Hansen and Lindblad 2010). While 23% of these publications were peer-reviewed journal articles, book chapters accounted for 25%, books for 5%, edited books for 2%, conference contributions for about 20%, reports for 8%, and doctoral theses for 2%. A relatively large number (10%) were publications categorised as ‘Other article’. In relation to language, the study suggested that about half of the publications were in English, while Swedish was used for 44% of the publications. Peer-reviewed journal articles were mostly (88%) published in English. The Swedish language was most often used for edited books, reports and other articles. That study, though, refers to a period before the different PRFSs were introduced in Sweden. Since then, publication patterns may have changed. The changes over time in these publication patterns also deserve closer examination.

All the studies we refer to draw on different sources of data; each source has its own specific characteristics and comparisons thus cannot easily be made. However, some of the differences, such as the share of journal articles, are so pronounced that it is unlikely that they are a consequence of differences in data collection alone. Hence, publication patterns within educational research indicate diversity that is linked to context-specific research and communication practices within this knowledge domain.

Performance-based research funding systems

Educational research in GU is pursued in a context of several PRFSs: the national system that allocates a part of state funding to universities, the institutional system that distributes a share of GU research funding across faculties, and the GU Faculty of Education system that further allocates resources to departments within the faculty. Here, we focus on the two models that are used at the national and the institutional level.

Swedish model

The PRFS was introduced at the national level in 2009 (Ett lyft för forskning och innovation 2008) with an emphasis on increased research quality, competitiveness, and the autonomy of institutions. Between 2010 and 2014, the performance-based share of state funding has increased gradually: from around 10 to 20% (Nelhans and Eklund 2015). The bibliometric part accounts for half of this; the other half is based on the amount of acquired external funding (Carlsson 2009). In the bibliometric part of this PRFS, funding is allocated to each university and university college on the basis of the number of publications, the number of field-normalised citations, the average field-normalised citation rate, and a bibliometric index for each institution (Aldberg and Jacobsson 2014). The number of publications and citations is fractionalised by the number of addresses and the number of research areas. The bibliometric index is a composite citations-based measure, calculated as the sum of field-normalised citation rates divided by ‘field factors’ for each research field. Field factor is an estimate of the average number of publications a researcher produces over a period of 4 years within one of 34 macro classes, which are more general research domains based on WoS categories (Aldberg and Jacobsson 2014).

For the calculation of this indicator, a database of the Swedish Research Council (SRC) is used, which is itself based on data from the Science Citation Index Expanded, the Social Sciences Citation Index and the Arts and Humanities Citation Index of WoS. In terms of publication types, only articles and reviews are taken into account (for further details on the SRC database, see Kronman et al. 2010; Vetenskapsrådet 2017).

Given these data sources, the design of this PRFS implicitly prioritises publishing of articles in journals that are indexed in WoS. For this reason, a compliance with goals of this PRFS may manifest in an increase in the number of articles published in journals indexed in WoS.

The Norwegian Publication Indicator in the University of Gothenburg

The PRFS in GU was introduced in 2009, the same year as the model at the national level. The GU PRFS employs two performance indicators: bibliometric numbers and the number of external grants (University of Gothenburg and Universitetslednings kansli 2008, sec. 10). The proportion of the funding that is allocated on the basis of bibliometric indicators has been increasing over years, from 5 to 10% (Anslagsfördelning och kostnadsdebitering för budgetåret 2015 samt planeringsramar för 2016–2017, Anslagsfördelning och kostnadsdebitering för budgetåret 2015; Inriktningsbeslut för budgetarbetet 2012. Sammanfattning 2011). Within GU each of the faculties could choose the bibliometric indicator to be employed. The bibliometric part of the GU model for the Faculty of Education (as well as for the other faculties for SSH) is derived from the general design of the Norwegian Publication Indicator (NPI) as it was used in Norway until 2016.

Overall, NPI is thought to be a more context-sensitive approach to bibliometric performance measurement than indicators reliant on a rather restricted set of data (such as the Swedish model) due to its emphasis on the use of data on a broad variety of scholarly publications (Sivertsen 2016b, p. 81). In contrast to models primarily focused on data on journal publications, the NPI-model takes into account monographs, book chapters and articles in journals. Furthermore, in NPI, bibliometric indicators are calculated employing a differentiated approach to publications in terms of the prestige different publishing channels (journals and publishers) have in the research community: a distinction is made between the publications that meet minimum criteria of inclusion (Level 1) and ‘the more prestigious’ ones (Level 2) (Schneider 2009, p. 371). In addition, GU allows administrators and researchers to add publication channels to Level 1 when they meet the inclusion criteria used in NPI (Gothenburg University Library: Allocation 2015). The bibliometric indicators are thus calculated by assigning weights to each publication, as indicated in Table 1. Due to these characteristics, NPI is a more differentiated form of PRFS, which takes into account specifics of publication practices in fields such as SSH.

Table 1 Publication points in the Norwegian Publication Indicator
Full size table

Finally, when it comes to co-authored publications, the publications are fractionalised by the number of authors and institutions. This is also the way the NPI is used at GU. It should, however, be noted that following an evaluation of NPI conducted in 2013, the design of NPI has been changed and a new version has been implemented in 2016 (Sivertsen 2016a). Due to the choice of timeframe for this study, these recent adjustments of NPI were not further explored.

While funding is allocated on the basis of output or performance, one may expect an increase in the share of publications included in NPI. Considering the points assigned to each publication type, one may identify an implicit hierarchy among different media (Table 1). Thus, it is possible that publication patterns show a convergence with the relative worth of publication types. Similarly, a possible change may be observed in the choice of language of publications. More specifically, such a change may be expected in the language of journal articles, since in the Norwegian Register of Journals (version 2015), used in NPI, there were no journals assigned to Level 2 in educational research that publish in other languages than English. But, given the contested nature of the new funding policy, especially within a diverse, partly practice- and partly research-oriented discipline such as educational research, and the different funding models in use at the national and the institutional level, we might also predict that measurements of the changes (if any) in the publication patterns will be contested.

Method

Operationalising educational research

Educational research within the Faculty of Education in GU takes place within three institutions: the Department of Education and Special Education, the Department of Education, Communication and Learning, and the Department of Pedagogical, Curricular and Professional studies. The average number of research staff in these institutions over the period 2005–2014 was 178.9 individuals (SD 24.2) (Source: data provided by GU Faculty of Education administration). In full time equivalents (FTE), the average for this period was 160.3 (SD 21.54). These figures are aggregated counts of individual and FTE for the following categories: professors, senior lecturers, post-doctoral researchers, and doctoral students. According to the Swedish Higher Education Authority, GU has the largest number of researchers associated with the field of educational research in Sweden (15% of total in 2015, Swedish Higher Education Authority).

Data

The source for data on publications was the GU publications database ‘Gothenburg University Publications’ (henceforth GUP, http://gup.ub.gu.se/). The data were acquired in October 2015. In this database, the data are self-reported, thus potentially containing some inaccuracies. The classification within GUP (‘Gothenburg University Library: Publication types in GUP’ 2015) was the starting point to operationalise the concept ‘publication’ and to set up inclusion/exclusion criteria. The classification of types within GUP is quite extensive: of the 25 publication types, 21 were identified in the original dataset. The analysis is limited to the following publication type categories: ‘peer-reviewed journal article’, ‘scientific journal article—non peer reviewed’, scientific journal article—review article, book review, other article, monograph, edited volume, text book, report, and book chapter. It should be noted that in this study no distinction is made between refereed and non-refereed book publications, since such a distinction was not in place in the GUP classification throughout the whole period explored. Publication types that were excluded from this study are theses (n = 138), conference papers and other conference contributions (n = 1274), and publications assigned to categories ‘Artistic research and development project’ (n = 1) and ‘Other’ (n = 58). The resulting dataset contains 2678 unique publications.

Each publication within the dataset has the following variables: identifier, publication year, type, language, level, and indexation in WoS. The data source for levels was the list of journals and publishers approved for the year 2015 (Available here: https://dbh.nsd.uib.no/publiseringskanaler/AlltidFerskListe). The variable for the indexation in WoS was used only for peer-reviewed journal articles (n = 753). The indexation in WoS (SCIE, SSCI, and AHCI) was identified manually using journals’ ISSN in combination with the publication year and (parts of) the title of an article. Change over time in the number of peer-reviewed journal articles indexed in WoS was analysed using two data sets. The first data set (A) includes articles (n = 235) in journals indexed in WoS without taking into account at which year the indexation has begun or stopped. The second data set (B) includes only articles (n = 129) in those journals that had been continuously indexed throughout 2005–2014.

Analysis

Our analysis is carried out in two steps. First, we pursue a descriptive empirical analysis of educational research publication patterns. Second, we pursue a second-order analysis of publication patterns, this time employing a social–epistemological approach. Using this approach, we contrast implicit assumptions within PRFS (specifically, the prioritisation of certain publishing behaviour highlighted in the previous section) with those within educational research (specifically, the co-existence of multiple research goals and the consequent diversity in publication patterns).

To identify changes in publication patterns, we explore the distribution of the number and the share of publications across years in relation to the following publication characteristics: publication type, language, level (NPI), and indexation in WoS. In addition, for the two most common publication types—journal articles and book chapters—we explore also the distribution across years and language, years and level, and years, language, and level.

The analysis of publication patterns covers a period of 10 years (2005–2014). This timeframe is chosen to cover a period of 5 years before and after the introduction of PRFS at the institutional level (in GU) and at the national level. We expect that changes in the publication output at the GU Faculty of Education during this period of time can be related to the introduction of the two performance-based funding systems. However, we do not assume a direct causal relation between PRFSs and publication practices. Instead, we use the PRFS designs as lenses that produce hierarchies among different communication media and hence can be used to discuss to what extent these hierarchies are in alignment with specifics of research groups subjected to PRFSs.

In the analysis we employ full counts of publications. Full counting is assumed more appropriate due to the interest in changes in publication patterns as such as opposed to latent phenomena such as research performance and the like. Furthermore, for a subset of data for which we have information on co-authorship (n = 1767; 66% of all publications), about half of the publications are single-authored, while about one-third are attributed to two authors. Thus effects on numbers resulting from the choice of the counting technique are likely to be minor.

Limitation

A limitation of this study is the focus on one specific field of research in a single university. Due to this, the generalisability of our findings is limited. However, we wish to foreground that, first of all, the empirical case presented here is illustrative of more general theoretical considerations that are transferable to other similar domains and to the employment of PRFS more generally. Secondly, educational research deserves interest as a field that is rarely explored by bibliometric means.

Findings

The analysis shows that over a period of 10 years, the number of publications per year has doubled (170 publications in 2005 and 351 in 2014; Table 2). The number fluctuates from year to year with no indication of an accelerated increase after the introduction of the two PRFSs in 2009. After a discussion of the major trends, we look at the relation between these trends and the introduction of the PRFSs used at GU.

Table 2 Distribution of publications across different publication types (2005-2014)
Full size table

Publication types and language

The general distribution of publications across different publication types suggests that peer-reviewed journal articles account for less than one-third, while book chapters constitute slightly more than one-third of the total number of publications (Table 2). These publication shares are higher than those reported by Hansen and Lindblad (2010) in their bibliometric analysis of Swedish educational research for the period 2004–2008.

Taking a closer look at dynamics in the distribution of publications across different publication types during the period 2005–2014, the change in the dominant communication channels is moreover remarkable. In 2005, the three most often used media were book chapters (33.5%, n = 57), peer-reviewed journal articles (23.5%, n = 40) and reports (23.5%, n = 40). In 2014, by contrast, the share of peer-reviewed journal articles has increased by 15.5 percentage points (39%; n = 137). The share of reports has decreased and now represents only 2.85% (n = 10). Book chapters in this data set appear as the second most often used medium, and its share has remained stable (33.3%, n = 117).

Another noteworthy change is in the share of popular scientific articles directed towards the general public (category ‘Other articles’). In 2005, this category of publications accounted for 5.9% (n = 10), but in 2014 it has increased by 5.2 percentage points (11.1%, n = 39). In 2014, this category is the third most often used medium of communication. Exploring publishing channels in which these articles are published, one can see various Swedish outlets either devoted to some aspect of education (e.g., ‘Förskoletidningen’, ‘Pedagogiska magasinet’) or to more general discussions about Swedish culture and language (e.g., ‘Sverige kontakt’). Thus the increase in the number of peer-reviewed journal articles does not imply that communication with a non-academic audience is no longer pursued. The new performance-based evaluation systems rather seem to stimulate researchers to gain some visibility beyond academia.

Concerns have been expressed that the emphasis on peer-reviewed journal articles can lead to the obsolescence of monographs, the category often seen as the most prestigious traditional medium in SSH. This data set shows that the share of monographs has remained stable, with a minor increase over time. In 2005, monographs accounted for 3.5% (n = 6). In 2014, the share has risen to 3.7% (n = 13).

Also, a slight increase can be observed for the number of book reviews, edited volumes, review articles, and text books, though the numbers for these types are relatively small and fluctuate from year to year.

In relation to the language of publications (see Supplementary Table 1), it can be added that slightly over half of all publications are written in Swedish (51%, n = 1360), while 45% (n = 1213) are in English. Analysis of changes over time shows a decrease in the share of publications in Swedish by 15.5 percentage points (from 56.5%, n = 96, in 2005 to 41.3%, n = 145, in 2014). In contrast, the share of publications in English has increased considerably (from 35.3%, n = 60, in 2005 to 56.4%, n = 198, in 2014) indicating a reorientation in the choice of language for scholarly communication within educational research in GU and in Sweden.

Altogether, these findings show that characteristics of educational research publication patterns in GU are similar to those identified in earlier studies on the Swedish and Norwegian context in terms of publication types and language (Hansen and Lindblad 2010; Sivertsen and Larsen 2012) and are fairly typical for SSH in general as shown elsewhere (Engels et al. 2012; Hicks 2004; van Leeuwen et al. 2016).

Distribution of publications in relation to the two performance-based research funding systems

The earlier identified increase in the number of journal articles can be directly linked to priorities implicit in the two PRFSs operating in this context. The Swedish model (the national PRFS) only takes into account peer-reviewed journal articles and reviews. Similarly, the hierarchy between different means of communication implicit in NPI (Table 1) implies a higher priority for peer-reviewed journal articles relative to book chapters. However, the increase in the number of articles in this category began in 2007, which is before the introduction of PRFSs.

The share of peer-reviewed journal articles indexed in WoS (those taken into account in the Swedish model) is rather small: only 30% are indexed in WoS (n = 235). Figure 1 and Supplementary Table 1 show an increase by a factor of 6.7 in the number of articles indexed in WoS in the total period under study. Before the introduction of PRFS, the number of articles increased by a factor of 3.3 (from 7 in 2005 to 23 in 2009). After the introduction of PRFS, the increase is smaller and more fluctuating (from 21 in 2010 or 37 in 2011 to 47 in 2014). Our interpretation needs to be cautious, however. A typical problem with the analysis of indexation in WoS is the expansion of the number of indexed journals. If we limit our analysis to the subset of journals indexed throughout the entire period under study (2005–2014), only 17.1% of GU educational research journal articles (n = 129) was identified in WoS. The changes over time in this data set, labelled B in Supplementary Table 1, show a fluctuating pattern, while the peak reached in year 2011 does not have an obvious explanation in the data analysed here. However, the discrepancies between set A and set B suggest that the observed increase is as much a consequence of the broader coverage of WoS as of the focused publication efforts of GU educational researchers: 60% of the total set of WoS-indexed publications in year 2014 appeared in journals that were added to WoS in recent years (i.e. after 2006).

Fig. 1
figure 1

The number (left) and the share (right) of peer-reviewed journal articles and their indexation in Web of Science (2005-2014)

Full size image

In relation to NPI, the analysis shows that over half of the publications are assigned to one of the two NPI levels (51.6%, n = 1384): 9.3% (n = 251) is assigned to Level 2 and 42.0% (n = 1133) to Level 1. Figure 2 shows the changes in the absolute and the relative number of publications in relation to the NPI levels (see also Supplementary Table 1). Between 2005 and 2014, the number of publications assigned to Level 1 has increased by a factor of 2.7 (from 60 publications in 2005 to 163 in 2014). This relative change is smaller than the one identified for the higher level publications, i.e. Level 2 publications, where the increase was by a factor of 2.9 (from 16 to 46). However, as Fig. 2 shows, the evolution of the share of Level 2 publications also displays some fluctuations.

Fig. 2
figure 2

The number (left) and the share (right) of publications by NPI level (2005-2014)

Full size image

It should also be noted that the number of publications that do not meet the minimum criteria to be included in NPI has been relatively stable from 2005 to 2011. After this period, the number fluctuates with a tendency towards a minor increase. In relative terms, the share of these publications has been decreasing between 2005 and 2014.

Figure 3 provides an overview of changes for the two main publication types, peer-reviewed journal articles and book chapters, in relation to NPI levels and language (see also Supplementary Table 1). For the Level 2 publications, both journal articles and book chapters, the time path shows clear fluctuations, although the overall trend is slightly increasing. For Level 1 journal articles, a distinctive increase can be observed after 2009. This number has risen by a factor of 2.3 (from 45 in 2010 to 105 in 2014). For the number of book chapters, remarkable peaks occur in years 2009 and 2011. Apart from these 2 years, the number of book chapters has been increasing but without a clear trend after 2009. For the publications that are not taken into account in NPI (marked as ‘Not NPI’), a slight increase in the number of journal articles can be noted from 2012 onwards. However, given the small numbers, this hardly can be regarded as a trend. For book chapters, the pattern is somewhat different. From 2006 to 2009, the number of book chapters not included in NPI has been decreasing. From 2010 onwards, the number has been slightly increasing with a rather steep increase in years 2013 and 2014. We will return to this shortly.

Fig. 3
figure 3

The number of peer-reviewed journal articles and book chapters by year, language and NPI level (2005-2014)

Full size image

Considering the language of publications, there has been a greater increase in the number of publications written in English as opposed to Swedish after 2010. A particularly substantial increase can be noted in the number of Level 1 journal articles in English (from 33 in 2010 to 98 in 2014). In contrast, for book chapters, the aforementioned increase in 2013 can be observed only for publications in Swedish (from 13 book chapters in 2012 to 39 in 2014).

A possible reason for the increase in the number of book chapters in Swedish not included in NPI may be changes within the register of publishing channels, which is used to assign publishers and journals to Level 1 and Level 2 in NPI. As noted earlier, the inclusion of publishing channels—journals and publishers—is a matter decided upon by the Norwegian scholarly community. GU also allowed to add publication channels to NPI, provided they meet the inclusion criteria used in NPI. In 2013 and 2014, the 2 years with the rapid increase in book chapters in Swedish (Fig. 3), the total number of book chapters in Swedish not in NPI was 81. 46% of these book chapters were in books published by the Swedish publisher ‘Studentlitteratur’ (http://www.studentlitteratur.se) and 2013 and 2014 are exactly the 2 years in which this publisher was not included in the register. Further, 30% of the book chapters in these 2 years were published in books issued by the University of Gothenburg, but this publisher likely did not meet a basic requirement for approval as a NPI Level 1 publication, viz. having not more than two-thirds of all authors belonging to the same institution (NSD—Norsk senter for forskningsdata 2017). Again, we see that the publication output measurements are strongly determined by the coverage of the data sets.

Summary of findings

This study explored educational research at GU in Sweden (2005–2014) in a context of national and institutional PRFSs. As in many other contexts, the total number of publications is increasing. This increase even predates the introduction of the PRFSs. In terms of publication types, a rather steep increase occurred for the number of peer-reviewed journal articles. This was especially so for the number of such articles in English. Another noteworthy pattern was the increase in the number of articles addressing a non-academic audience (category ‘Other article’). The number of monographs, the medium often perceived as endangered by the use of bibliometric indicators in PRFS, has slightly increased and the share of publications in this category has remained stable.

The analysis of changes in the language of publications showed a consistent change towards communication in English. While about one-third of the publications were in English in 2005, by 2014 more than half of the publications were published in this language.

The timeframe for this analysis was chosen including 5 years before and 5 years after the introduction of the PRFSs. It was assumed that publication patterns in the period 2005–2009 are not linked with PRFSs, but that, if any changes observed between 2010 and 2014 indicate a convergence with priorities within the two PRFSs, it may be an indication of the influence of PRFS. Surely, it could also be that trends in publication patterns related to the use of PRFS are either delayed or precede the introduction of PRFS in 2009. First of all, the time between writing and publishing plays a role. Second, it may take time before incentives from the university level ‘trickle down’ to faculty, departmental and individual level. In the same way, it may be the case that discussions about the national PRFS, or the earlier introduction of PRFS in neighbouring Norway (or elsewhere), played a role and help explain the changes in publishing behaviour we can observe already before 2009. Either way, this study shows that the most evident changes (the change in language use as well as the increase in the number of peer-reviewed journal articles) were present already before the introduction of the PRFSs. At the same time, the number of articles indexed in WoS shows an increase, especially after 2010. While this holds for the subset that included only those journals that have been indexed throughout the whole period analysed here, it is also, and most of all, the result of the expanded coverage of WoS in 2005–2010 (Thomson Reuters 2010).

In relation to NPI levels, this study showed that the increase in Level 1 and Level 2 publications is complemented by a decline in the share of publications that do not meet the minimum criteria to be included in NPI. At the same time, these trends started prior to the period when the two PRFSs were introduced.

A slightly different trend can be identified exploring changes in the number of publications in relation to the NPI Levels and publication types. While there has been a distinctive increase in the number of Level 1 peer-reviewed journal articles after 2010, such an evolution can neither be identified for Level 2 articles nor for those not included in NPI. If one assumes that NPI may be the cause of these changes, then this insight contrasts with earlier findings on changes in publication patterns in the context of NPI (Aagaard et al. 2015; Schneider et al. 2016). Despite the differentiated counting in NPI, there is a move towards publishing in those channels that lead to ‘easier’ success (here: Level 1).

Finally, a change can be noted for the number of book chapters in Swedish that are not taken into account in NPI. As explained earlier, this can largely be explained by changes in the list of approved publishers that underlies NPI. This illustrates how not just the Swedish model, which relies on WoS, but also the Norwegian model, which allows to include a broader variety of publications, is dependent on its inclusion criteria.

Discussion and conclusion

Adopting the aims of PRFSs

If one assumes that the designs of PRFS do justice to educational research and that the selection of certain kinds of publications for inclusion into performance indicators does provide valid insight into research performance in this field of research, then one can draw the following conclusions. Any PRFS that employs bibliometric indicators implicitly prioritises quantity (even if coupled with citation counts and/or differentiated counting). To this end, the observed changes seem desirable, because the number of publications is continuously increasing. Further, both PRFSs operating in the explored context prioritise communication in peer-reviewed journal articles (NPI also includes monographs and book chapters). The emphasis on peer-review implies that communication ought to be directed towards a research audience. Often, the focus on peer-review is seen as a condition for the validity of bibliometric indicators (else there are no other means to judge whether the contents of publications counted are indeed a knowledge contribution). In the Swedish model, only journals indexed in WoS are taken into account.

A possible interpretation of the observed overall increase in the number of WoS-indexed articles is enhanced performance of educational research in GU. Yet, the discrepancies between the two WoS subsets make such a judgement arbitrary, since it conflates a property of WoS (expansion) with properties of indexed publications. NPI, although including more publications in its performance measures, similarly emphasises the peer community as the desirable audience. Hence the observed increase in publications, itself fulfilling the requirements of NPI, can be seen as change towards better research performance. This is also exemplified in relation to publication types. In NPI, priority is given to articles in journals over book chapters, while reports fall beyond the scope of NPI. Here, the goals of research are tied with specific means of communication.

In GU educational research, chapters in edited books were the most common publication type in year 2005, while the number of peer-reviewed journal articles was equal to the number of reports (n = 40) (Table 2). The changes over the period of ten years (2005–2014) show a shift whereby journal articles have become the main channel for communication (n = 137 in 2014). Book chapters are still common, yet their number is now smaller than the number of articles in journals (n = 117 in 2014). Further, the increasing number of articles in journals in English assigned to the lower Level 1 may reasonably be considered an indication of the fact that researchers are seeking out ‘easier’ publication types. Alternatively, this may equally be an indication of a learning process. The number of reports (not considered in any of the PRFS) has decreased considerably. Typically, reports are not subjected to formal peer-review, unlike journal articles. Overall, one may assume that an increasing level of challenge for authors operates among the different PRFS categories used in this study (Not NPI, Not WoS, Level 1, Level 2). The changes we observed seem to point to a convergence between publication practices in educational research and the goals of research implied by the two PRFSs. If convergence between publication practices and the research goals as encoded in PRFS is assumed to be valid, then the application of PRFS in practice can appear desirable to academic authors.

Contrasting the aims of PRFSs with those in educational research

Whether the implied goals are valid for the field of educational research is an aspect that can be illuminated by drawing on insights from social epistemology. It is evident that indicators based on publications (and citations) are treated as generally valid representations of research performance, while at the same time neglecting that there are many different ways to understand and judge research performance. Hence the validity of bibliometric indicators of performance hinges on consensus within a particular research community about the relationship between bibliometric indicators and the performance of researchers.

As noted earlier, a characteristic of educational research is its sheer diversity in conceptual sources (such as its bewildering array of research traditions) as well as in research agendas. This great diversity in individual researchers’ beliefs, commitments and actions spans national and even institutional contexts, which makes a general consensus on the relationship between bibliometric indicators and research performance hardly achievable. How likely would it be that scholars who represent different paradigms would reach consensus on which indicators would best represent research performance? Would they even agree on what counts as research?

A similar point may be raised with respect to the increase in the number of NPI Level 1 journal articles. Following NPI assumptions, one is invited to see book chapters as less desirable than journal articles. A very different view can however easily be found in educational research, where placing priority on short, standardized journal articles may be seen as an undesirable move away from the sorts of extended theoretical reflections that are considered key to guiding the discipline, the scope of which is simply beyond the length of a journal article (Silfver 2013). In that case, an increase in the number of journal articles is regarded as undesirable by some researchers, since it is judged indicative of a decline in research performance.

The case of educational research is just one example of a field of research where multiple paradigms coexist, thus highlighting that even within a single field there may be researchers with conflicting views on the usefulness of bibliometric indicators within PRFS. In such context, PRFS becomes a structural constraint that not only prioritises a sense of research performance that is viewed by many as external—and perhaps even as an external threat—to the discipline itself. It might also produce a divisive hierarchy between those who prefer to write journal articles and those who assign priority to books and book chapters. The same argument applies to those who are better versed in writing in national languages (e.g., Swedish) and those who only ever write in English or, perhaps, invest most in communicating research through lectures and oral presentations. One may conclude that this is a matter of professionalisation. At the same time, however, the clear ambiguity in the relation between bibliometric measures in PRFS and certain fields of research requires that it is made clear precisely whose views prevail in designs of PRFSs—since a consensus cannot reasonably be assumed even for a single discipline, as argued here. These last considerations, clearly, also apply beyond the field of educational research.

Implications

We conclude by discussing some implications from our analysis, both for bibliometric analyses of publication patterns in a context of PRFS and for research policy that advocates the use of PRFS. Pursuing an analysis of publication patterns and contrasting it with the implicit assumptions of PRFSs that operate in the context, we have presented a way to discuss the desirability of PRFS as an incentivising mechanism. Our strategy has allowed us to avoid the common pitfall to imply causation from correlation. We cannot and do not claim that, for example, the increase in the number of peer-reviewed articles is an effect of PRFS. As has been argued by, for example, Gläser (2017), such a claim would require an identification of a mechanism that links PRFS and researchers’ behaviour empirically, or at least a discussion of alternative causal factors that may have contributed to the observed changes in publication patterns (e.g., Butler 2003). Here, employing a social–epistemological approach, we have treated PRFSs as conceptual devices that (implicitly) set priorities for publishing behaviour thus allowing to focus on the desirability of the changed patterns regardless of their cause. In doing so, we are able to higlight that it is not self-evident that the goals reified in the PRFSs are aligned with goals one can identify in educational research.

This brings us to implications for research policy. PRFSs are typically seen either as unobtrusive measures of research performance or as incentive mechanisms. Assumptions implicit or explicit in the choice of specific indicators and data sets often result in the prioritisation of certain publishing behaviour. In this study, we showed that only a few of the identified changes are in line with the priorities set within the PRFSs in use. This can be interpreted as an indication of the fact that the incentives produced by PRFSs are relatively weak. There is sufficient space for researchers to follow what they themselves regard important. At the same time, however, the assumptions underlying the PRFSs do not entirely correspond with the goals within the field of educational research in Gothenburg and Sweden. This might imply that the PRFS designs need to be adapted to and aligned with the research fields—both in terms of the choice of indicators and of data used to measure publication output. For educational research, this strategy could involve the use of a broader set of indicators that takes into account the multiple relevant audiences. Furthermore, given the co-existence of different research paradigms in a single context, it could also involve giving credit to different communication means. However, any PRFS design that incorporates particular priorities will set constraints for researchers who pursue novel, creative work that is beyond the sensibilities of the indicators employed. One way to anticipate this is to plan continuous revisions of the choice of indicators. Another way, given the unlikely feasibility of such an ongoing design process, is to reconsider whether a relatively diffuse, open-ended incentivising mechanism is an example of a desirable research policy.

Conclusive answers to such concerns are beyond the scope of this article, yet the contrasting interpretations to which one can arrive when analysing publication patterns point to a need for, among else, bibliometric analysis complemented with qualitative and/or conceptual research that helps to bring to the surface the more subtle ways in which the use of PRFS structures the pursuit of knowledge.