Introduction

Mendeley readership counts are promising indicators of scholarly impact (Gunn 2013; Haustein and Siebenlist 2011; Maflahi and Thelwall 2016) and appear much earlier than citations because they are as less affected by publication delays. For example, an article may be registered in Mendeley on the day that it is published. Mendeley readership counts can also reveal readers’ disciplines and nationalities, giving more specific impact evidence (Thelwall and Sud 2015). Investigations of Mendeley readership counts so far have focused on either journal articles or books, but conference papers are valuable in some engineering-related fields and so it is important to assess whether they could also be applied to conference papers.

Although there are many limitations with using citation counts in formal and informal research evaluation as scholarly impact indicators, they are more robust than indicators derived from the web because these can easily be manipulated, making them unsafe for most formal evaluations (Wouters and Costas 2012). Several years ago, Mendeley had about 2.4 million users who uploaded over 420 million documents across disciplines ranging from life science to maths to the arts and humanities (Gunn 2013). Although Mendeley can be spammed, its large user base and positive results from previous analyses with it (see below) suggest that it does not currently suffer from a substantial amount of spam.

Papers presented at conferences in many fields are seen as a stage towards the creation of journal articles (Drott 1995). Nevertheless, in some research fields conference papers are valued for being timelier, more cutting-edge and more cited than journal articles (Goodrum et al. 2001) and can be either regarded as the main outputs of research or broadly comparable to journal articles as research outputs.

The gap that this research tries to fill is to discover whether the impact of conference papers is reflected in their Mendeley readership counts in the engineering-related fields in which they are important. Although several studies have found correlations between Mendeley readership counts and citation counts (Li et al. 2011; Bar-llan 2012; Mohammadi and Thelwall 2014), the extent to which Mendeley readership counts capture the impact of conference papers is unknown. The current study compares Mendeley readership counts and citation counts for both journal articles and conference papers in four engineering-related Scopus subject categories: Computer Science Applications; Computer Software; Building and Construction Engineering; and Industrial and Manufacturing Engineering. The paper also investigates why articles can be highly cited in Scopus but have few Mendeley readers and vice versa.

Altmetrics

The term altmetrics (alternative metrics) refers to academic indicators derived from social web data. Altmetrics rely on real-time data and interactions that can be quantified and measured immediately (Galloway et al. 2013). Existing altmetrics have used a variety of data sources including article downloads (Bollen et al. 2008) views and saves, as well as tweets, blogs, bookmarking sites and wikis. These are all used by scholars to communicate different kinds of research impact (Cronin 2013).

The main, but not only, way to assess altmetrics is through correlation tests (Sud and Thelwall 2014). One altmetric, tweet counts, might not be suitable for correlation tests, based on its increasing uptake resulting in newer articles having higher tweet counts than the older articles (Thelwall et al. 2013). However, there is no evidence that this issue also applies to Mendeley readers.

The two major shortcomings of citation counts for assessing scholarly impact are that they are slow to accumulate and only reflect scholarly impact rather than applied impact. This has led to a need for new metrics to compliment traditional citation metrics (Priem et al. 2012). However, many scholars have argued that the new metrics should not be restricted to overcoming the limitations of the previous citation indicators, but can also be expected to provide new insights into research evaluation (Priem et al. 2010; Torres-Salinas et al. 2013).

Mendeley readership

Mendeley is an academic social web site for managing references, creating online profiles and sharing research documents with peers. It has an open Applications Programming Interface (API) that can be used for compiling usage indicators with a database of 2.6 million users as of October 2014 (Mohammadi et al. 2015a). Currently, Mendeley readership statistics seem to be the most closely related to citation counts, in comparison to other altmetrics. Many studies have used correlations to assess the relationship between Mendeley readership counts and citation counts for the same articles. A study of Nature and Science articles published in 2007 shows significant and moderate correlations between Mendeley readership counts and citation counts (Li et al. 2011). A study of five social science fields with 62,647 articles and five humanities fields with 14,640 articles found low to moderate significant positive correlations for each discipline (Mohammadi and Thelwall 2014). This study provides substantial evidence that Mendeley readership could be useful for assessing scholarly impact. Generally, most studies investigating the relationships between Mendeley readers and citation counts (Li and Thelwall 2011; Haustein et al. 2014; Costas et al. 2015) have reported either weak or moderate positive correlations between Mendeley readership counts and citation counts.

Mendeley has a higher proportion of articles with non-zero metric values than most other altmetrics (Zahedi et al. 2014). Out of 19,722 publications in this study, 62.6% had at least one reader. Mendeley is particularly used by undergraduates and postgraduates, whereas only academic authors can make citations. An analysis of the ‘career stages’ of the different Mendeley users found that Postdocs and PhD students register more in Mendeley than any other user category (Zahedi et al. 2013). A Mendeley survey found that out of 860 Mendeley users, 55% who had bookmarked articles in Mendeley had read them or intended to read them (Mohammadi et al. 2015b). However, not all readers record their articles in Mendeley, so the data does not represent all readers, but, most importantly, the survey shows that Mendeley bookmark counts are an indicator of readership.

Articles in Mendeley may be widely read but rarely cited in Scopus-indexed publications and vice versa for a number of reasons (Thelwall 2015): Authors from countries that do not publish in Scopus journals may receive more Mendeley readers than Scopus-indexed citations; short articles may support the process of research findings but may not be cited; users of Mendeley may register for an updated version of an article but the original version may be cited by others; some communities do not use Mendeley due to the nature of their professions; multidisciplinary articles may attract many citations but few readers based on multiple categorization norms; readers may prefer to read review articles but cite the reviewed articles rather than the review.

Research questions

The primary goal of this paper is to assess the value of Mendeley readership counts in conference-based fields and, as part of this, to find why articles in Mendeley may be widely read but rarely cited and vice versa in these fields. This study focused on four Scopus subject areas: Computer science applications; Computer Software; Building and Construction engineering; and Industrial and Manufacturing engineering. These represent different fields in which conference papers are important. The following research questions drive the study.

  • Do Mendeley readership counts and citation counts reflect the scholarly impact of conference papers in Computer Science Applications, Computer Software, Building and Construction Engineering and Industrial and Manufacturing Engineering?

  • Does the answer to the above research question differ between engineering fields in comparison to journal articles?

  • What are the causes of conference papers having many readers compared to citations or vice versa?

Methods

Correlations between Mendeley readers and citation counts were calculated to ascertain the relationship between readership counts and citation counts for both conference and journal articles. A significant positive correlation gives evidence of a common factor between readership and citation counts.

All bibliographic information and citation data for journal articles and conference papers in the four fields from 2011 was extracted from Scopus. The year 2011 was chosen to give a substantial period for citations to accrue, so that there is more chance of getting high correlations between citation and readership counts for both journal articles and conference papers. The first and last 5000 journal articles and conference papers for each subject category were downloaded from Scopus in March 2015. This has the limitation that conferences in the middle of a year for large categories may be omitted, and these may be particularly prestigious in some fields. Narrow subject categories were used to ensure more comprehensive coverage. From the Scopus computer science category, a field in which conferences are arguably more important than journals, the two categories of Computer Science Applications and Computer Software were chosen. Conferences are also known to be important in engineering in general, and so from the broad Scopus Engineering category, the two narrow categories of Building and Construction and Industrial and Manufacturing Engineering were chosen.

Spearman correlations were used to compare citations and Mendeley readers because the data are skewed. Mendeley reader data was obtained using Webometric Analyst, a free software package. The Mendeley API in Webometric Analyst was used to extract data for Mendeley readers. The Spearman rank correlation formula was used to calculate 95% confidence intervals for the correlation coefficients. The formula used was \( { \tanh }\left( {{\text{arctanh}}\left( r \right) \pm \frac{1.96}{{\sqrt {n - 3} }}} \right) \). Here, r is the sample correlation and n is the sample size. The sampling distribution of the estimate was approximately normal on the transformed scale; hence a 95% CI was found by taking the transformed estimate and adding and subtracting 1.96 times its standard error (Dowdy et al. 2011, pp. 245–246).

To determine outliers in order to find why articles in Mendeley are widely read but rarely cited in Scopus-indexed publications and vice versa, the logarithmic transformation ln(1 + x) was used on the data set (Thelwall and Wilson 2014) for both readers and citation counts to reduce the skewness of the data, before regressing the reader counts against the citation counts. The residuals from the linear regression were used to determine the main outliers. These were then manually investigated for likely causes. The purpose of the logarithmic transformation was to avoid focusing too much on papers with high values of one or other variable but to include outliers where the anomalies were within the low or moderate end of the scale. Thus, for example, a difference between 0 and 4 citations seems more significant than a difference between 40 and 44 citations, whereas the differences would be the same for untransformed data.

Results

The Mendeley readership counts correlate strongly (0.560–0.662) with citation counts in all subject categories for journal articles (Table 1). For conference papers, readership counts correlate moderately (0.437–0.439) with citation counts in Computer Science Applications and Software. Readership counts have low correlations (0.143–0.168) with citation counts in Building and Construction and Industrial and Manufacturing Engineering. The low correlations for conference papers in Building and Construction Engineering (0.143) and Industrial and Manufacturing Engineering (0.168) might be due to the low coverage of conference proceedings in engineering subject categories, reducing their Scopus citation counts.

Table 1 Spearman correlations between Mendeley reader counts and citation counts for articles and conference papers in Scopus from 2011 in the four subject categories analysed
Full size table

There are low proportions of cited papers in Scopus for Industrial and Manufacturing Engineering (17.5%) and Building and Construction Engineering (18.3%) conference papers (Table 2). This could be due to low coverage in Scopus of conference proceedings. Also, there are low Mendeley reader counts for Building and Construction Engineering (18.7%) conference papers, which could be due to few Mendeley users or the low value of conference papers in this field.

Table 2 Scopus citation counts and Mendeley readership counts, median, geometric mean and percentage coverage for both journal articles and conference papers
Full size table

The following were identified as likely causes of high Mendeley readership counts compared to Scopus citation counts for conference papers.

  • Papers that are written based on improving the performance of an existing system. Computer Science Applications; ‘‘Purlieus: Locality-aware resource allocation for Mapreduce in a cloud’’, has 74 Mendeley readers but no citations. ‘‘Reducing electricity cost through Virtual Machine placement in high performance computing clouds’’, (66 readers, 0 citations) demonstrates a system that can be used to reduce electricity cost and load migration at minimum low electricity consumption rates. ‘‘ On the duality of data-Intensive file system design: Reconciling HDFS and PVFS’’ (62 readers, 0 citations).

  • Papers that create public awareness, motivation and participation for new scientific discoveries. For example, three papers from Computer Software; ‘‘Attention please! Learning analytics for Visualization and recommendation’’, (202 readers, 1 citation) ‘‘Dusting for Science: Motivation and participation of digital citizen science volunteers’’, (149 readers, 6 citations) and ‘‘A survey of risks, threats and vulnerabilities in cloud computing’’, (115 readers, 0 citations).

  • Papers that are relevant to daily life or religious beliefs. For example, paper in Industrial and Manufacturing Engineering, ‘‘Halal supply chains in the food industry- A conceptual model’’, (50 readers, 6 citations).

  • Practical solutions to important real-world problems. For example, Building and Construction Engineering, ‘‘Overview of UFC 3-340-02, Structures to resist the effects of accidental explosions’’, has 29 Mendeley readers but no citation. ‘‘Sandnet: Network traffic analysis of malicious software’’ (36 readers, 10 citations) In the same Building and Construction Engineering Scopus subject category for conference paper, ‘‘Exploiting home automation protocols for load monitoring in smart buildings’’ (32 Mendeley readers 1 citation).

  • Social media articles that may be of general interest to users. For example, in Computer Science Applications article papers; ‘‘Serious social media: On the use of social media for improving student’s adjustment to college’’ (170 readers, 15 citations), and ‘‘Personal Learning Environments, social media, and self-regulated learning: A natural formula for connecting formal and informal learning’’ (404 readers, 74 citations).

  • Practical commercial advice. In Building and Construction Engineering, an article titled; ‘‘Characterizing entry mode for international construction markets: paving way to a selection model’’ (16 readers, 0 citations).

  • Articles of regional interest may attract more readers than citations. For example, Computer Software, ‘‘A citizen-oriented approach for evaluating the performance of e-government in Sri Lanka’’ (28 readers, 0 citations).

The following were identified as likely causes of high Scopus citation counts compared to Mendeley readership counts for conference papers.

  • Papers on software packages that may be cited if the software is used, without necessarily reading the paper. In Computer Software, ‘‘MICE: Multivariate Imputation by Chained Equation in R’’ (249 citations, 0 readers) and “ContextFJ: A minimal core calculus for context-oriented programming” (18 citations, 0 readers).

  • Papers with a set model for completing a task. For example, in Computer Science Applications, ‘‘Recommended steps for thematic synthesis in software engineering’’ (20 citations, 0 readers). “A framework for capturing distinguishing user behaviours in novel interfaces” (13 citations, 0 readers) and Building and Construction Engineering “A naming convention for the piano key weirs geometrical parameters” (20 citations, 0 readers).

Discussion

This study has several limitations. The citation data is from Scopus and is dependent on the coverage of Scopus for the magnitude of the citation counts. Similarly, Mendeley is not the only reference manager and the results may have been different for another reference manager if it is more widely used by engineers. The results may also vary by year and could be different for other types of engineering that have not been investigated. The causes of higher Mendeley readership counts than citations or vice versa are based on the small sample of papers analysed in the current paper and may not apply to other years or fields. Another limitation is that in some fields a few conferences may be very important whereas others are not and so the scope of the current study (analysing all Scopus-indexed conference papers) may hide important differences between conference types.

Mendeley readership counts and Scopus citation counts have strong and significant positive correlations for journal articles in all of the engineering fields analysed and for conference papers in the two computing fields but not in the other two engineering fields, Industrial and Manufacturing Engineering and Building and Construction Engineering, which have weak but positive correlations.

For journal articles, the strong and positive correlations between Mendeley readership and citation counts for all four of the studied Engineering subject categories corroborate past studies of other areas (Li et al. 2011; Bar-Ilan 2012; Mohammadi and Thelwall 2014).

For conference papers, 68.6% of the papers in the Computer Software subject category have at least one Mendeley reader and 54.7% of the papers have at least one Scopus citation. These findings show that the impact of conferences is high in Scopus and Mendeley for computing research. In Building and Construction Engineering, conference papers have a much lower percentage coverage; 18.3 and 18.7% of the papers have at least one Scopus citation and at least one Mendeley reader, respectively. This may be due to low coverage of conference proceedings in the field of engineering for Scopus but this cannot explain the results for Mendeley. It may be that a high percentage of engineering conference papers are not of interest to publishing academics, either because of their applied focus or due to disciplinary norms in citation practices.

The list of reasons why papers may attract many readers compared to their citations, or vice versa, shows that there are legitimate causes of outliers. It is therefore important to accept that Mendeley reader counts will not always be a good approximation to Scopus citation counts for individual papers.

Conclusions

Based upon high and positive correlations in the subject categories of Computer Science Applications and Computer Software, Mendeley readership counts for conference papers in computer science should be acceptable as scholarly impact indicators. Since Mendeley readership counts are particularly useful for early impact evidence (Thelwall and Sud 2016), this may be their greatest value in computing. In contrast, the weak correlations between Mendeley readership and citation counts in the subject categories of Building and Construction Engineering and Industrial and Manufacturing Engineering, coupled with low proportions of papers with at least one reader, suggest that conference papers in these types of engineering do not support a similar claim. This may be due to the low scholarly impact of conference papers in these fields and it may be that their value is not primarily within academia. If this is the case, then new indicators, perhaps including download counts, would be needed to reflect this impact.