Abstract
Citations to published work are gaining increasing prominence in evaluations of the research performance of scientists. Considering the importance accorded to gender issues in South African science, it is surprising that (to our knowledge) no research has as yet ascertained the extent of sex differences in citations to the published work of scientists in this country. Our literature study shows that studies that have been conducted elsewhere tend to neglect in their analyses important gender-related and other factors, such as the sex composition of multi-authored papers and the extent of foreign co-authorship. Against this background, we illustrate the difficulties inherent in measuring the quality aspect of sex-specific research performance by means of an analysis of a dataset of articles (n = 229) that were published between 1990 and 2002 in the field of invasion ecology and in journals included in the Thomson Reuters Web of Science. Each article has at least one South African author address. The results indicate that foreign co-authorship is a better correlate of high citations than the sex of South African authors, and this is true irrespective of whether the annual citation rate or window period is used, whether or not self-citations are excluded, and whether or not the number of authors is controlled for by calculating fractional counts. The paper highlights these and other considerations that are relevant for future gender-focused bibliometric research, both in South Africa and beyond.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Increasing women’s participation in science and collecting sex-disaggregated data on the issue is considered a pressing need internationally (United Nations Educational, Scientific and Cultural Organization (UNESCO) 2007). Considering the central role that research performance and its measurement plays in assessing and rewarding the role performance of scientists, it may be argued that one cannot address the issue of women’s participation in science without taking into account gender variations in research performance. This construct is commonly defined in terms of two dimensions: the number of papers individuals publish and the number of times they are cited. As noted by Ward and Grant (1996) and by Mcelhinny et al. (2003), the majority of studies that compare the research performance of women and men have focused on the former, without measuring gender difference in citations. Their almost universal finding, that women publish less than men, has been found to apply to South Africa as well (Prozesky 2006).
This focus on the quantitative dimension of research performance ignores the fact that output volume does not necessarily imply impact or recognition of the output by others (Cole and Zuckerman 1984; Helmreich et al. 1980), particularly when one considers the exponential growth in the volume of scientific publications globally. With regard to gender in particular, one cannot merely argue that, because women are known to publish less than men, they are cited less than men. There are indeed some indications that women as a group may tend to exhibit different publication behaviour or habits than men do—publishing less in terms of quantity but more in terms of quality, for example (Sonnert 1995; Sonnert and Holton 1995).
Against this background, it is important to note that citation-based measures have become increasingly important in determining rewards such as funding, hiring, promotion and tenure, and despite strong criticism against the use of citations to scientists’ work to assess their research performance, this practice is “unlikely to be abandoned in the near future” (Leimu and Koricheva 2005, p. 31). From a gender perspective, the criterion of quantity may count more heavily for women than for men. For example, Persell (1983, p. 45) reports that, in the field of education, “[w]hen women hold their own in the quality of their research, they derive no benefit from it, while the quantity of publication (where men outperform women) counts more heavily for women than for men”. Citations also represent the most common form of recognition within science, which encourages future publication (Ward and Grant 1996). From the perspective of the sociology of knowledge, sex differences in citations may indicate the extent to which a gender’s ideas dominate and therefore shape a field. However, patterns in citations that reflect or even reproduce gender inequalities can be “subtle, not easily detectable without tally sheets and statistics; they can operate below the level of awareness” (Hutson 2002, p. 331). Quantifying them by means of citation analysis should therefore constitute an important dimension of any research aimed at assessing women’s status, performance and contribution to science.
In South Africa, the National Advisory Council on Innovation (NACI) has recently advised the Minister of Science and Technology that, “public funds must be utilized to increase the volume and enhance the quality of scientific output performed by women” (National Advisory Council on Innovation (NACI) 2008, p. 11, italics added). What is implied is that South African women scientists fall short, relative to their male counterparts, in terms of both the quantitative and qualitative dimensions of research performance. However, this seems to be an untested assumption: to the author’s knowledge, not a single study has been conducted in South Africa on sex differences in terms of the quality of scientific output.
This paper is aimed at investigating to what extent this may be attributed to challenges involved in applying bibliometrics as a tool for the valid measurement of sex-specific research performance. It reports on the findings of a dual research strategy, involving both a literature study and the analysis of primary data on citations to South African authored scientific journal contributions to the field of invasion ecology. The contribution of the paper is situated primarily in the methodological realm, by taking into account various potentially confounding variables, some of which, as the literature study shows, have not been taken into account in any previous research on sex differences in citations. These variables concern time since publication, the collaborative nature of both publications and citations (i.e., between sexes and between authors from different countries) and the tendency to self-cite. In order to determine whether and to what extent they have a distorting effect on bibliometric measures of gender-specific research performance, we compare results of two sets of analyses: one that does not control for these variables, and one that does. The paper concludes with a summary of the methodological considerations that should (or would not) be taken into account in future gender-focused bibliometric research, both in South Africa and beyond.
Literature study
In order to determine the extent to which potentially confounding variables have been taken into account in previous research on sex differences in citations (as generated by the Institute for Scientific Information [ISI]), a total of 49 such studies, with publication dates ranging from 1973 to 2011, were identified and reviewed (Table 1).
The following section reports on the data processing methods utilised in these studies, and highlights methodological issues we will be investigating in more detail through our own analysis of primary citation data.
Time-related controls
In the majority of studies reviewed, citations were collected over a continuous period that consists of a number of years, with only 5 studiesFootnote 1 that recorded article and/or journal citation rates for only one year. A number of studies (7) do not fall into either category, as they involved recording citations received in a number of discrete (i.e., non-continuous) years.Footnote 2 Most of the researchers who collected citations for more than one year control for the fact that citations may increase over time since the publication of the cited article, and in the majority of cases this involves computing an annual citation rate by dividing the total number of citations that a cited article has received at a specific point in time (usually at the time of data collection) by the number of years that have lapsed since that article’s year of publication. Three of the more recent studies,Footnote 3 that use regression models, included time as a variable in the equation in order to control for the fact that more recent articles have had less time to be cited. Another relatively rare approach among researchers who compare the sexes in terms of citations was followed by Stack (1994), Gonzalez-Brambila and Veloso (2007), Borsuk et al. (2009), and Powell et al. (2009), who used a fixed citation window, which involves limiting for each cited article the time period for which citations are counted to the same number of years after publication. This is in spite of the fact that the fixed citation window approach is considered the “standard method in bibliometric analysis, in order to give equal time spans for citation to articles published in different years” (Craig et al. 2007, p. 243, as cited in Bornmann 2010).
Controlling for multiple authorship
Counting citations without taking into account multiple authorship has been termed “one of the most serious errors in empirical judgement made in the sociology of science” (Lindsey 1980, p. 145). Already in 1980, Lindsey noted that, “[s]ince scientific publication is now characterized by extensive multiple authorship, this error has become critical” (p. 146). It has been found that multi-authored papers are more likely to be cited and to be more highly cited than single-authored papers (Beaver 1986, cited in Fox 1991; Rousseau 1992, cited in Cronin and Overfelt 1994; also see Lindsey 1980 for a review). More to the point of this review is evidence that women are more heavily represented as co-authors and men as sole authors of articles in (1) major sociology journals, which “might reflect young women scholars’ avoidance of submission of solo-authored works to top-ranking journals” (Ward and Grant 1991, p. 213), and (2) education journals (Demetrulias 1986). However, neither Cole and Zuckerman (1984), nor Long (1992) found any evidence that women collaborate more or less often than men, while Over and Moore (1980) report that women academic psychologists published slightly more often than men as sole authors.
In 1981, Garfield proclaimed that “[t]here is as yet no agreed-upon method for assigning credit to each co-author” (p. 277). A decade later, Cronin and Overfelt (1994) summarised three different approaches to allocating credit in the case of multi-authored works: (1) straight counts (first author only receives credit); (2) whole countsFootnote 4 (each author receives full credit); and (3) fractional counts (fractions are awarded to each co-author, usually by dividing citation count by number of co-authors).Footnote 5 Of the 32 studies reviewed, more than a third (14) made no reference as to how credit was assigned in the case of multi-authored works. Of the remaining 18 studies, by far the majority (12) applied straight counts, 2 applied whole counts and only 4 applied a fractional count.
According to MacRoberts and MacRoberts (1987, p. 346, as cited in Cronin and Overfelt 1994), applying these three very different methods of credit allocation will “in all likelihood give widely different results”. In response, Cronin and Overfelt (1994) compared results that were obtained by using all three of the above-mentioned approaches to allocating citation credit in the case of multi-authored works. Although they used only straight counts to test the relationship between citation level and gender, it is important to note that they found little difference in the overall rankings produced by the three approaches. They therefore conclude that “all three approaches could be used interchangeably without introducing significant distortions”, but do acknowledge that this may not apply to other populations with, for example, different authorship patterns (p. 65).
Extending this point to sex differences in citations, possible sex differences not only in the tendency to co-author, but also in terms of authorship position, require taking into account the sex composition of an author set. Findings are not conclusive, however. On the one hand, there is a lack of evidence among biochemists (Long 1992, p. 170) that women are “relegated to the back of the author list”, Cole and Zuckerman (1984) found that women scientists in various fields are first authors as often as men. Ferber (1986), cited in Davenport and Snyder 1995) reports that citation counts for mixed gender collaborative authors fall somewhere in between citation scores for each gender of single authors, and in the field of ecology Borsuk et al. (2009) found no significant differences in citation rates by first-author gender. On the other hand, the weight of evidence seems to indicate that women are more likely to be subsidiary or non-lead co-authors in major (prestigious and widely read) sociology journals (Ward and Grant 1991), education journals (Demetrulias 1986), as well as in the fields of business (Walters et al. 1990), academic psychology (Over and Moore 1980 Footnote 6; Teghtsoonian 1974), ecology (Borsuk et al. 2009), and construction (Powell et al. 2009).
In 1984, Cole and Zuckerman noted that the sex composition of sets of authors, particularly combinations of men and women, has not been taken into account in most research on sex differences in citation rates. In fact, at the time of writing, they knew of no study that had taken into account papers authored by men and women jointly, and it was “still not clear how best to count citations when the sex composition of collaborative groups varies” (p. 241). Our review, conducted more than two decades later, shows that only approximately 1 in 5 studies use whole or fractional counts, and could therefore (at least potentially) take into account papers authored by men and women jointly. One study (Montpetit et al. 2008) took the gender of such papers to be that of the majority of authors and, when there was a tie, reverted to a straight count (the gender of the lead author).
The tendency to use straight counts, and as a consequence defining female(male)-authored articles as those with female(male) solo or first authors, and neglecting the sex composition of author sets may be attributed to two main factors. The first of these is the convention followed during approximately the first two decades after the inception of Science Citation Index (SCI)Footnote 7 in 1961 to attribute all citations to a multi-authored paper to the first or solo author (Garfield 1981; Sonnert 1995). The lack of all-author data meant that citation counts were underestimated for the subset of authors who collaborate more than usual and whose names tend not to be the first in the author sets. Thus, sex differences in authorship practices mentioned above, “may produce artifactual differences in citation counts” (Cole and Zuckerman 1984, p. 220). Even though all-author data became available in the ISI indexes of the 1980s, our literature study shows that researchers who compare the sexes in terms of citations still tend to collect and link citations to first or solo authors only, rather than use whole or fractional counts.
The second possible reason why researchers have neglected the sex composition of author sets in their citation comparisons between the sexes, is that obtaining data on the sex of authors—and especially of all the authors in author sets that are ever-increasing in size—is very time-consuming, since most periodicals (and therefore the Thomson Reuters Web of Science [WoS] indexes based on them) only record the initials of authors.Footnote 8 Even if a study is limited to papers published in journals that generally publish authors’ first names (e.g., Tregenza 2002), many papers may have authors with androgynous names that are not evidently male or female.
The last issue concerning multi-authored papers relates specifically to internationally co-authored articles. Leta and Lewison (2003) and Lewison (2001) cite several studies that have demonstrated that internationally co-authored papers tend to have higher impact than domestic ones. More importantly, the tendency to co-author papers internationally may relate to gender, as it may depend on the availability of travel grants to go to conferences abroad (Lewison 2001). Leta and Lewison (2003) and Lewison (2001) are the only researchers whose studies we reviewed, who determined whether females are as likely as males to co-author papers internationally. Leta and Lewison (2003) investigated the frequency of domestic and foreign publications according to the sex of a sample of Brazilian scientists, but found that female scientists tend to collaborate internationally as much as men do—a factor they suggest might have contributed to the similarity they found between men and women in terms of the potential impact of their publications. In his study of Icelandic scientists, Lewison (2001) divided his sample of papers between purely Icelandic papers and ones with a foreign co-author, and found a “relative paucity of female authorships” (p. 42) in the latter category of papers.
The tendency to self-cite
The final methodological issue considered in this review concerns the way in which self-citations are dealt with in studies that compare men and women in terms of citations. According to Creamer (1998, p. 35), the “rate of self-citation is high enough that it is fairly common practice to remove self-citations from total citation counts”. Of the 43 studies that utilised WoS indexes, more than half (24) did not indicate explicitly whether self-citations were included or not. Of the remaining 19 studies, approximately half (10) excluded self-citations, while 5 did so for only a sub-sample, or part of the analysis, primarily because of the time-consuming process involved in identifying and excluding self-citations. In order to overcome this problem, two sets of researchers (Kelly and Jennions 2006; Symonds et al. 2006) analysed only a randomly selected sub-sample of scientists. The exclusion (or not) of self-citations is of particular relevance for this paper, as the likelihood to cite oneself seems to be related to gender, with women having been shown to be less likely to cite themselves than men (Lutz 1990; Helmreich et al. 1980; also see Creamer 1998 for a review). However, Hutson (2006), Borrego et al. (2010), and Symonds et al. (2006) found no evidence of gender differences in the rates of self-citation, and the latter conclude that their citation analysis is unlikely to be systematically biased by any gender differences in the rates of self-citation.
Methodology
The subfield of invasion ecology in South Africa
Webster (2001) notes that bibliometric studies, particularly those focusing on gender differences in publication productivity, are usually concentrated on small samples, partly because sexing a relatively large sample of scientists is very time-consuming (as mentioned above), but also because citation characteristics are field-dependent, and therefore indirectly related to gender, as the sexes differ in terms of their participation in various fields.Footnote 9 The present study follows suit, by including only South African authors of articles on a subfield of ecology—invasion ecology—in the study population.
Richardson and Van Wilgen (2004, p. 47) define invasion ecology as “the study of the human-mediated introduction of organisms, especially introductions to areas outside the potential range of given organisms, as defined by their natural dispersal mechanisms and bio-geographical barriers”. According to these South African invasion ecologists, the field (sometimes also referred to as invasion biology) addresses all aspects relating to the introduction of organisms, their ability to establish, naturalize and invade in the target region, their interactions with resident organisms in their new location, and the consideration of costs and benefits of their presence and abundance with reference to human value systems. Fundamental research questions in invasion ecology include: (1) Why are some species more successful invaders than others? (2) Why are some systems more susceptible to invasions than others? and (3) How can the harmful impacts of invasions be prevented, reduced or mitigated? (Drake et al. 1989, and Williamson 1996, as cited in Richardson and Van Wilgen 2004).
From a global perspective, South Africa has been identified as one of only three regions outside the USA where researchers who produce highly cited articles in invasion ecology are well-represented (Pyšek et al. 2006), and the country alone accounts for two-thirds of research effort on the African continent (Pyšek et al. 2008). A recent bibliometric study of all biological invasions-related publications in the SCI from 1991 to 2007 ranked a South AfricanFootnote 10 as the most productive author in the field over this period, ranked South Africa in the top 15 publication position, and found that South Africa had higher citation per publication (CPP) values compared to other world regions, including the US and Western Europe (Qiu and Chen 2009). South African researchers in this field therefore enjoy a high level of productivity and representation and effective international collaboration in the international invasions literature. It is therefore not surprising that one of only six highly prestigious DST-NRFFootnote 11 Centres of Excellence in South Africa, the Centre for Invasion Biology (CIB), focuses on invasion ecology. The first author’s involvement as a core team member of this Centre provided her with background knowledge of the field of invasion ecology, which was especially useful to define the field of invasion ecology according to its outputs, and to identify its authors.
Data source
In compiling a dataset of relevant publications, all articles published in the field of invasion ecology between 1990 and 2002, with at least one South African author address and published in journals listed in the WoS, were extracted from SA Knowledgebase. The latter is a dynamic database of public science in South Africa, developed by the Centre for Research on Evaluation, Science and Technology (CREST) at Stellenbosch University. Only full-length articles were extracted, meaning that notes, letters, reviews, proceeding papers and editorial material were excluded. Articles were selected in terms of the presence of invasion ecology-related terms and their combinations or derivatives in article titles and abstracts, keywords, journal and institutional affiliation.Footnote 12 The help of scientists at the CIB was enlisted to assess the exhaustiveness of the key terms and the selected articles. A total of 246 articles were identified.
Determining the sex and national affiliation of authors
Although SA Knowledgebase provides some author-specific data on demographic variables, data on sex were available for less than 30% of the authors in the invasion ecology dataset. The next step was to fill in the missing data by examining, among others, the original articles and/or other publications by the same author, and web pages providing staff profiles of academic departments. This significantly reduced the number of authors with missing sexes. The procedure also helped in the identification of the national affiliation of authors (i.e. South Africa or foreign country address), in cases where SA Knowledgebase did not capture the national affiliation of authors in the invasion ecology dataset. The sex and national affiliation of authors could be fully identified for 229 of the 246 articles.
Calculating citations and applying time-related controls
As SA Knowledgebase does not contain any data on citations, the total number of citations for each article was obtained from the online WoS citation database. Specifically, the total number of citations per article since the year of publication until December 2008 was calculated. Only citing articles (as opposed to reviews, proceeding papers, etc.) were considered. The names of authors of the citing articles were also recorded and checked against those of the cited articles to identify self-citations. The latter enabled the calculation of a total citation count that excludes self-citing. For both sets of citation counts (i.e. self-citations respectively included and excluded) a time-related control was applied by dividing the total number of citations (at the end of 2008) by the number of years since publication of the cited article.
Apart from this annual citation rate, a second time-related control was introduced by limiting, for each cited article, the time window for which citations are counted to three years after the year of publication (the minimum duration, according to Bornmann 2010). For instance, if a cited article was published in 2001, citations up to 2004 were counted; similarly, citations until 2005 were included for an article that was published in 2002. This fixed citation window measure also had two variants: self-citations respectively included and excluded. Thus, in this study the annual citation rate and the fixed citation window are treated as two separate approaches that provide different comparable perspectives in the measurement of citations.
Controlling for multiple authorship
Fractional counts were calculated to control for the number of authors per cited article. Practically this means that, for each cited article, the four sets of citation counts (two time-related controls, each with and without self-citations) were divided by the number of co-authors. Thus, an additional four sets of fractional counts were calculated. Together with the other four sets of (whole) citation counts, it implies a total of eight citation measures.
Results
The first set of results for the eight citation measures is based on authorships, as opposed to articles, as the unit of analysis. In an authorship dataset an author is counted for every article that he or she produces, which means that the total number of authorships will always exceed the total number of (non-duplicate) authors. The 229 articles are associated with 386 South African male authorships, 109 South African female authorships, 21 foreign male authorships and 6 foreign female authorships. The authorship analysis in Table 2 shows that the average number of citations received by foreign male authors is always higher than that received by either South African authors (male or female) or foreign female authors. This is the case irrespective of the measure of citation used. South African male authors, in turn, received higher citations than South African women on all eight measures.
The next step was to control for the effect of foreign male authors by splitting the authorship dataset in two: a subset of authorships where every underlying article is co-authored by at least one foreign male, and a subset where none of the underlying articles is co-authored by a foreign male (Table 3). The results of this procedure confirm the previous finding: the highest citations are associated with foreign male authorships and, when the influence of foreign men is systematically removed, the larger number of citations is associated with South African male authorships.
It is important to note that an analysis of authorships ignores the sex composition of sets of collaborating authors. For instance, it is not clear in Table 3, in subset 2, how many of the 107 South African female authorships are co-occurring with South African male authorships. A better solution is to consider the article as the unit of analysis and to categorise each article in terms of author sex and national affiliation. This was done in Table 4, which shows that the highest citation overall is associated with the 18 articles that are co-produced by South African men and foreign authors. This is true for seven of the eight citation measures. The exception is the window measure that uses fractional counts and includes self-citations. For this measure, articles that are authored by a single South African male solicited the highest citations. However, the calculation of fractional counts assigns a larger citation weight to single-authored articles, as these articles are divided by 1 and the others by a factor greater than 1, meaning that, for single-authored articles, the fractional count will always equal the whole count. Thus, where fractional counts are reported, the citation rate of single-authored articles will always exceed that of multi-authored papers in cases where both attract the same number of citations. Since the calculation of fractional citation counts is considered a fair control for papers with multiple authors, it could be argued that the use of whole citation counts for multi-authored papers will produce results that are inflated.
Table 4 reveals another instance where the results seem to differ according to citation measure: the comparison of articles produced by a single South African man versus those produced by a single South African woman. In terms of the annual citation rate, articles by a single South African woman attract, on average, more citations than articles by a single South African man. However, these differences are not statistically significant, based on a series of independent t-tests (self-citations included: t = 0.194, df = 52, p = 0.847; self-citations excluded: t = 0.328, df = 52, p = 0.744). Furthermore, when a fixed citation window is applied, the average citation counts appear highest for articles produced by a single South African man. These differences are also not statistically significant (self-citations included: t = −1.754, df = 49.4, p = 0.086; self-citations excluded: t = −0.709, df = 52, p = 0.481).
Overall the results indicate that foreign co-authorship is a better correlate of high citations than the sex of South African authors, and this is true irrespective of whether the annual citation rate or window period is used, whether or not self-citations are excluded, and whether or not the number of authors is controlled for by calculating fractional counts.
Conclusion
Since the publication in 1973 of the first study that compared men and women in terms of citations by analyzing data generated by the ISI, approximately 50 similar studies have been conducted in the past 3–4 decades. This is a small research effort, compared to the much larger body of empirical literature comparing men and women in terms of publication productivity, which most probably points towards the fact that comparing the sexes in terms of citation counts is much more complex and time-consuming than comparing them in terms of article counts.
On the basis of an analysis of this body of literature, we have highlighted some of the difficulties that are involved in the drawing of reliable and valid comparisons between men and women in terms of the citations their work attracts. At the core of the methodological complexities involved, lies the time-consuming process involved in sexing articles, especially those with a large number of joint authors of different sexes. Citation studies with a gender focus have as yet not managed to address these challenges well: sample sizes have been relatively small, which limits generalisability; credit tends to be allocated via straight counts, which does not take into account possible sex differences in authorship practices and as such may disadvantage women more than men; and the exclusion of self-citations tends to be neglected, which again may bias findings towards men. In addition, among researchers who compare the sexes in terms of citations, average annual rate of citations has been the preferred method for controlling for the fact that articles have different “ages”, while the standard bibliometric method involving the use of fixed citation windows has been all but ignored.
These methodological considerations have all been taken into account in the analysis presented here. A foreign (male) author effect is evident, as the highest citations to South African (co-) authored articles in invasion ecology appear to be associated with this subgroup. If one controls for foreign male authors, a clear sex effect emerges, as the highest citations are then associated with South African men (compared to South African women). However, internationally co-authored articles dominate the citation profiles, even when controlling for both self-citations and number of co-authors (through the calculation of fractional counts). This supports Van Raan’s (1998) conclusion that the higher citations achieved by internationally co-authored papers is not because of a greater probability of self-citing (which, in turn, is possible because of a larger number of authors generally associated with international co-authorship), but because of a genuine “impact-strengthening” effect of international collaboration.
Lastly, within the South African science policy arena, calls are frequently made to address the status of women in science. The trend towards corporatism in the South African higher education sector (which produces by far the majority of the country’s scientific output) means that this status increasingly depends on what are argued to be objective measures of research performance. Objective measures, for instance, must take into consideration that more than half of South Africa’s total article output is produced by foreign authors (Sooryamoorthy 2009). The research presented here provides evidence that the use of citation counts to compare men and women in terms of the qualitative dimension of their research performance requires meeting a number of methodological challenges, which have thus far been neglected in the majority of studies that draw such a comparison. Descriptions of gender differences in citations should therefore be approached cautiously, and with particular sensitivity to the effect that gender differences in the tendency to co-author internationally may have on citation counts.
Notes
Also referred to as “normal counts” (Trifunac 2006, p. 1070).
In the latter case, however, the sex differences were small enough to lead Over and Moore (1980, p. 415) to conclude that “[t]he men and women were […] equally likely to employ authorship patterns that enhance individual visibility”.
Already in the early 1980s, ISI recognised the need “to treat all authors in every article as though they were listed first” (Garfield 1981, p. 269) and started developing a system that would allow this. In 1978, Garfield (1981) was the first to use these so-called “all-author data”. He does mention that “[d]eveloping all-author lists is considerably more complex than conducting first-author studies” (p. 269). However, his findings (the average author received 1,178 citations to papers on which he or she appeared as first author and 2,633 as co-author) clearly demonstrate “the importance of all-author data” (p. 275).
In some countries, it is possible to differentiate between the sexes based on the surname alone. For instance, in Poland more than sixty percent of all surnames can be determined as belonging to a man (those which end with “-ski”, “-cki” or “-owy”) or a woman (those ending with “-ska”, “-cka” or “-owa”) (Webster 2001). In Iceland, women’s surnames typically end in “dottir”, whereas the names of men end in “son”, which allowed Lewison (2001) to conduct one of the most extensive and detailed analyses of gender differences in citations thus far. Lewison (2001, p. 42) rightly comments that, “It would be very difficult to extend such a study to other countries unless there were a complete list of the country’s researchers available with their sexes so that comparisons could be made”.
See Boshoff (2005) for South African statistics in this regard.
Prof David Richardson, currently Deputy Director of the CIB and a Professor in the Department of Botany and Zoology at Stellenbosch University.
The South African Department of Science and Technology (DST) and National Research Foundation (NRF)—the two bodies responsible for the funding and administration of the Centres of Excellence.
Examples include invasive/invasion (management); alien/plant invasions; naturalized/non-indigenous; indigenous; native; exotic; biological invasions, alien species, invasive alien species and invasion biology/ecology. In subsequent searches, the net was thrown wider, as broader terms and their derivatives (e.g., biological diversity and biological control) were used to identify papers that may have been overlooked by the more obvious key terms. In case of doubt, the contents of papers were scanned in order to decide whether they should be included or not.
References
Aksnes, D. W., Rorstad, K., Piro, F., & Sivertsen, G. (2011). Are female researchers less cited? A large-scale study of Norwegian scientists. Journal of the American Society for Information Science and Technology, 62(4), 628–636.
Beaver, D. B. (1986). Collaboration and teamwork in physics. Czechoslovak Journal of Physics, 1, 14–18.
Bordons, M., Morillo, F., Fernández, M. T., & Gómez, I. (2003). One step further in the production of bibliometric indicators at the micro level: Differences by gender and professional category of scientists. Scientometrics, 57(2), 159–173.
Bornmann, L. (2010). Towards an ideal method of measuring research performance: Some comments to the Opthof and Leydesdorff (2010) paper. Journal of Informetrics, 4, 441–443.
Borrego, Á., Barrios, M., Villarroya, A., & Ollé, C. (2010). Scientific output and impact of postdoctoral scientists: A gender perspective. Scientometrics, 83, 93–101.
Borsuk, R. M., Budden, A. E., Leimu, R., Aarssen, L. W., & Lortie, C. J. (2009). The influence of author gender, national language and number of authors on citation rate in ecology. The Open Ecology Journal, 2, 25–28.
Boshoff, N. (2005). The representation of women academics in higher education in South Africa: Progress in the pipeline? South African Journal of Higher Education, 19(2), 359–377.
Cole, J. R. (1979). Fair science: Women in the scientific community. New York: Free Press.
Cole, J. R., & Cole, S. (1973). Social stratification in science. Chicago: University of Chicago Press.
Cole, J. R., & Zuckerman, H. (1984). The productivity puzzle: Persistence and change in patterns of publication of men and women scientists. Advances in Motivation and Achievement, 2, 217–258.
Corley, E. A., & Sabharwal, M. (2010). Scholarly collaboration and productivity patterns in public administration: Analysing recent trends. Public Administration, 88(3), 627–648.
Craig, I. D., Plume, A. M., McVeigh, M. E., Pringle, J., & Amin, M. (2007). Do open access articles have greater citation impact? A critical review of the literature. Journal of Informetrics, 1(3), 239–248.
Creamer, E. G. (1998). Assessing faculty publication productivity: Issues of equity. ASHE-ERIC Higher Education Report (Vol. 26, No. 2). Washington, DC: Association for the Study of Higher Education.
Cronin, B., & Overfelt, K. (1994). Citation-based auditing of academic performance. Journal of the American Society for Information Science, 45(2), 61–72.
Davenport, E., & Snyder, H. (1995). Who cites women? Whom do women cite? An exploration of gender and scholarly citation in sociology. Journal of Documentation, 51(4), 404–410.
Davis, D. E., & Astin, H. S. (1987). Reputational standing in academe. The Journal of Higher Education, 58, 261–275.
Demetrulias, D. M. (1986). Gender differences in co-authorship. Journal of Educational Equity and Leadership, 6(2), 119–127.
Drake, J., Mooney, H. A., Di Castri, F., Groves, R., Kruger, F. J., Rejmánek, M., et al. (Eds.). (1989). Biological invasions: A global perspective. Chichester: Wiley.
Ferber, M. A. (1986). Citations: Are they an objective measure of scholarly merit? Signs: Journal of Women in Culture and Society, 11(2), 381–389.
Flores, L. Y., Rooney, S. C., Heppner, P. P., Browne, L. D., & Wei, M.-F. (1999). Trend analyses of major contributions in The Counseling Psychologist cited from 1986 to 1996: Impact and implications. The Counseling Psychologist, 27(1), 73–95.
Fox, M. F. (1991). Gender, environmental milieu, and productivity in science. In H. Zuckerman, J. R. Cole, & J. T. Bruer (Eds.), The outer circle: Women in the scientific community (pp. 188–204). New York: Norton.
García-Pérez, M. A. (2009). The Hirsch h index in a non-mainstream area: Methodology of the behavioral sciences in Spain. The Spanish Journal of Psychology, 12(2), 833–849.
Garfield, E. (1981). The 1,000 contemporary scientists most-cited 1965–1978 (Part I: The basic list and introduction). Current Contents, 41, 5–14. [Reprinted in: Garfield, E. (1981–1982). Essays of an information scientist, 5, 269–278].
Gonzalez-Brambila, C., & Veloso, F. M. (2007). The determinants of research output and impact: A study of Mexican researchers. Research Policy, 36, 1035–1051.
Hammel, E. A., Mason, C., Prater, A., & Lundy, R. (1995). Gender and the academic career in North American Anthropology: Differentiating intramarket from extramarket bias. Current Anthropology, 36(2), 366–380.
Haslam, N., Ban, L., Kaufmann, L., Loughnan, S., Peters, K., Whelan, J., et al. (2008). What makes an article influential? Predicting impact in social and personality psychology. Scientometrics, 76(1), 169–185.
Helmreich, R. L., Spence, J. T., Beane, W. E., Lucker, G. W., & Matthews, K. A. (1980). Making it in academic psychology: Personality correlates of attainment. Journal of Personality and Social Psychology, 39(5), 896–908.
Housri, N., Cheung, M. C., Koniaris, L. G., & Zimmers, T. A. (2008). Scientific impact of women in academic surgery. Journal of Surgical Research, 148, 13–16.
Hunter, L. A., & Leahey, E. (2010). Parenting and research productivity: New evidence and methods. Social Studies of Science, 40(3), 433–451.
Hutson, S. R. (2002). Gendered citation practices in American Antiquity and other archaeology journals. American Antiquity, 67(2), 331–342.
Hutson, S. R. (2006). Self-citation in archaeology: Age, gender, prestige, and the self. Journal of Archaeological Method and Theory, 13(1), 1–19.
Irvine, J., & Martin, B. R. (1986). Women in radio astronomy—Shooting stars? In J. Harding (Ed.), Perspectives on gender and science (pp. 80–103). London: The Falmer Press.
Kelly, C. D., & Jennions, M. D. (2006). The h index and career assessment by numbers. Trends in Ecology & Evolution, 21(4), 167–170.
Kolpin, V. W., & Singell, L. D., Jr. (1996). The gender composition and scholarly performance of economics departments: A test for employment discrimination. Industrial and Labor Relations Review, 49(3), 408–423.
Krampen, G. (2008). The evaluation of university departments and their scientists: Some general considerations with reference to exemplary bibliometric publication and citation analyses for a Department of Psychology. Scientometrics, 76(1), 3–21.
Leahey, E. (2007). Not by productivity alone: How visibility and specialization contribute to academic earnings. American Sociological Review, 72(4), 533–561.
Leimu, R., & Koricheva, J. (2005). What determines the citation frequency of ecological papers? Trends in Ecology & Evolution, 20(1), 28–32.
Leta, J., & Lewison, G. (2003). The contribution of women in Brazilian science: A case study in astronomy, immunology and oceanography. Scientometrics, 57(3), 339–353.
Lewison, G. (2001). The quantity and quality of female researchers: A bibliometric study of Iceland. Scientometrics, 52(1), 29–43.
Lindsey, D. (1980). Production and citation measures in the sociology of science: The problem of multiple authorship. Social Studies of Science, 10(2), 145–162.
Long, J. S. (1992). Measures of sex differences in scientific productivity. Social Forces, 71(1), 159–178.
Lutz, C. (1990). The erasure of women’s writing in sociocultural anthropology. American Ethnologist, 17, 611–627.
MacRoberts, M. H., & MacRoberts, B. R. (1987). Problems of citation analysis: a critical review. Journal of the American Society for Information Science, 40, 342–349.
Mauleón, E., & Bordons, M. (2006). Productivity, impact and publication habits by gender in the area of materials science. Scientometrics, 66(1), 199–218.
Mauleón, E., Bordons, M., & Oppenheim, C. (2008). The effect of gender on research staff success in life sciences in the Spanish National Research Council. Research Evaluation, 17(3), 213–225.
Mcelhinny, B., Hols, M., Holtzkener, J., Unger, S., & Hicks, C. (2003). Gender, publication and citation in sociolinguistics and linguistic anthropology: The construction of a scholarly canon. Language in Society, 32, 299–328.
Montpetit, E., Blais, A., & Foucault, M. (2008). What does it take for a Canadian political scientist to be cited? Social Science Quarterly, 89(3), 802–816.
National Advisory Council on Innovation (NACI). (2008). Creating the future South African national system of innovation: Gender, race and SET sector issues. Report by the Science, Engineering and Technology for Women (SET4W), NACI Permanent Sub-Committee. http://www.nacinnovation.biz/about-naci/naci-sub-committees/set4women/set4w-downloads. Accessed 9 March 2011.
Over, R., & Moore, D. (1980). Research productivity and impact of men and women in psychology departments of Australian universities, 1975–1977. Australian Psychologist, 15(3), 413–418.
Peñas, C. S., & Willet, P. (2006). Gender differences in publication and citation counts in librarianship and information science research. Journal of Information Science, 32(5), 480–485.
Persell, C. H. (1983). Gender, rewards and research in education. Psychology of Women Quarterly, 8(1), 33–47.
Powell, A., Hassan, T. M., Dainty, A. R. J., & Carter, C. (2009). Exploring gender differences in construction research: A European perspective. Construction Management and Economics, 27, 803–807.
Prozesky, H. E. (2006). Gender differences in the journal publication productivity of South African academic authors. South African Review of Sociology, 37(2), 87–112.
Pyšek, P., Richardson, D. M., & Jarošik, V. (2006). Who cites who in the invasion zoo: Insights from an analysis of the most highly cited papers in invasion ecology. Preslia, 78, 437–468.
Pyšek, P., Richardson, D. M., Pergl, J., Jarošík, V., Sixtová, Z., & Weber, E. (2008). Geographical and taxonomical biases in invasion ecology. Trends in Ecology & Evolution, 23, 237–244.
Qiu, H., & Chen, Y.-F. (2009). Bibliometric analysis of biological invasions research during the period of 1991 to 2007. Scientometrics, 81(3), 601–610.
Reese-Evans, L. (2010). Gender and citation in two LIS e-journals: A bibliometric analysis of LIBRES and Information Research. Library and Information Science Research Electronic Journal, 20(1), 1–19.
Reskin, B. F. (1978). Scientific productivity, sex and location in the institution of science. American Journal of Sociology, 83(5), 1235–1243.
Richardson, D. M., & Van Wilgen, B. W. (2004). Invasive alien plants in South Africa: How well do we understand the ecological impacts? South African Journal of Science, 100(Jan/Feb), 45–52.
Rousseau, R. (1992). Letter to the editor: Why I am not cited or why are multi-authored papers more cited than others? Journal of Documentation, 48, 79–80.
Sandström, U. (2009). Combining curriculum vitae and bibliometric analysis: Mobility, gender and research performance. Research Evaluation, 18(2), 135–142.
Sonnert, G. (1995). What makes a good scientist? Determinants of peer evaluation among biologists. Social Studies of Science, 25(1), 35–55.
Sonnert, G., & Holton, G. J. (1995). Gender differences in science careers: The Project Access Study. New Brunswick, NJ: Rutgers University Press.
Sooryamoorthy, R. (2009). Collaboration and publication: How collaborative are scientists in South Africa? Scientometrics, 80(2), 419–439.
Stack, S. (1994). An analysis of the impacts of books and journal articles. International Review of Modern Sociology, 24(2), 119–125.
Symonds, M. R. E., Gemmell, N. J., Braisher, T. L., Gorringe, K. L., & Elgar, M. A. (2006). Gender differences in publication output: Towards an unbiased metric of research performance. PLoS ONE, 1(2), e127.
Teghtsoonian, M. (1974). Distribution of sex by authors and editors of psychological journals, 1970–1972: Are there enough women editors? American Psychologist, 29, 262–269.
Toutkoushian, R. K. (1994). Using citations to measure sex discrimination in faculty scales. The Review of Higher Education, 18(1), 61–82.
Tregenza, T. (2002). Gender bias in the refereeing process? Trends in Ecology & Evolution, 17(8), 349–350.
Trifunac, M. D. (2006). A note on publication and citation rates of female academics in earthquake engineering. Soil Dynamics and Earthquake Engineering, 26, 1063–1075.
Trimble, V. (1985). Some notes on patterns in citations of papers by American astronomers. Quarterly Journal of the Royal Astronomical Society, 26, 40–50.
Trimble, V. (1993). Patterns in citations of papers by American astronomers. Quarterly Journal of the Royal Astronomical Society, 34, 235–250.
United Nations Educational, Scientific and Cultural Organization (UNESCO). (2007). Science, technology and gender: An international report (executive summary). Science and technology for development series. Paris: UNESCO Publishing. http://unesdoc.unesco.org/images/0015/001540/154027e.pdf. Accessed 9 March 2011.
Van Raan, A. F. J. (1998). The influence of international collaboration on the impact of research results. Scientometrics, 42(3), 423–428.
Walters, G. D. (2006). Predicting subsequent citations to articles published in twelve crime-psychology journals: Author impact versus journal impact. Scientometrics, 69(3), 499–510.
Walters, C. G., Fry, E. H., & Chaisson, B. D. (1990). Women scholars: Closing the publication gap. Research in Higher Education, 31(4), 355–367.
Ward, K. B., Gast, J., & Grant, L. (1992). Visibility and dissemination of women’s and men’s sociological scholarship. Social Problems, 39(3), 291–298.
Ward, K. B., & Grant, L. (1991). Gender and publishing in sociology. Gender & Society, 5(2), 207–223.
Ward, K. B., & Grant, L. (1996). Gender and academic publishing. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. xi, pp. 172–212). New York: Agathon Press.
Webster, B. M. (2001). Polish women in science: A bibliometric analysis of Polish science and its publications, 1980–1999. Research Evaluation, 10(3), 185–194.
Wennerås, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387, 341–343.
Williamson, M. (1996). Biological invasions. London: Chapman and Hall.
Acknowledgments
The authors thank CREST for the use of SA Knowledgebase, and the former database manager, Derick van Niekerk, for running the keyword searches. Acknowledgements are due to the Director (Steven Chown) and Deputy Director (David Richardson) of the CIB for their assistance in identifying the sex of some of the authors in the dataset, and in identifying relevant keywords. David Richardson is also thanked for highlighting the need to control for the time available for accumulation of citations, and his and Dan Simberloff’s insightful comments on an earlier draft of this paper, as well as the comments of two anonymous reviewers, are much appreciated. Research assistance for this project was funded from the first author’s CIB core team member grant. Finally, we are highly indebted to Charline Mouton, Marion van Dorssen and Christopher Mechnig for their assistance in collecting data for this project.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Prozesky, H., Boshoff, N. Bibliometrics as a tool for measuring gender-specific research performance: an example from South African invasion ecology. Scientometrics 90, 383–406 (2012). https://doi.org/10.1007/s11192-011-0478-7
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-011-0478-7