Introduction

Since the publication of the Altmetrics Manifesto (Priem et al. 2010), a huge number of academic publications have been released around the world on alternative metrics (Holmberg 2015; Tattersall 2016; Roemer and Borchardt 2015). The main objective of these altmetric studies has been to understand the meaning of these metrics and their possible application to research evaluation (Erdt et al. 2016; Sugimoto et al. 2017). In this form, a first step has been to compare and put into context these metrics with regard to bibliometric indicators (Thelwall et al. 2013; Costas et al. 2015). The poor correlations found lead us to think, for now, that these metrics express a new dimension of the academic impact, which is closer to the popularization of science, societal impact and research dissemination (Ortega 2016).

Altmetric studies have been supported by data produced by a wide scope of services that track the use, mention, sharing and citation of research papers in social networks, publisher platforms, repositories, etc. Faced with this great amount of metrics and sources, altmetric providers have been a key element in the consolidation of this young research field, gathering alternative metrics on a single platform. These databases have made easier the comparison and study of different metrics, putting them in relation to each other and to the citation impact. Based on web syndication and open data, events on the Web about academic outputs are collected by these aggregators using several identifiers (DOIs, ISBN, URLs, etc.). The counting of these events introduces a new perspective about the social and online impact of these academic results. Today, these services are inserted in the scholarly communication system providing altmetric information to journals (PLOS, Nature), publisher platforms (ScienceDirect, SpringerLink) and digital libraries.

However, and in spite of the importance of these instruments for altmetric studies, there are a few studies that have investigated the consistency and reliability of altmetrics across data providers (Jobmann et al. 2014; Zahedi et al. 2015). This is an important gap because many of the current results might be limited to the data supplied by these aggregators. The aim of this study is to fill this gap and to analyze the coverage and counting differences between three representative data providers: Altmetric.com, PlumX and CED.

Related Research

Altmetric providers constitute an indispensable part in the altmetric research because they supply data for these studies. Some works have described and explored the functionalities of these services, as a way to explain their advantages and drawbacks. At this point, it is worth mentioning the studies of Champieux (2015) and Lindsay (2016) about PlumX, who make a descriptive analysis of its functionalities; the analyses performed by Adie and Roe (2013) and Trueger et al. (2015) about Altmetric.com and the serious criticisms of Gumpenberger et al. (2016) about the Altmetric Score. There are no studies to date about Crossref Event Data (CED) because this service is still in beta and it has not been officially released yet. However, other studies have analyzed the coverage of these services and have described the proportion of altmetric events in several samples. Robinson-García et al. (2014) analyzed the coverage of Altmetric.com and they found that 87.1% of articles had at least one tweet and 64.8% one Mendeley reader. In a similar way, Bornmann (2014) explored a set of articles from Altmetric.com and he observed that 71% of articles were tweeted and a moderated proportion of documents were mentioned in blogs (16%) and news (13%). Fraumann et al. (2015) also described the distribution of blogs and news covered by Altmetric.com and they found an important bias towards U. S. sites. According to PlumX, there are no studies that have treated its coverage. We can only mention the recent work of Torres-Salinas et al. (2017) about the collection of books and the assessment of these materials, and the study of Ortega (2016) about the tweeting of research articles using PlumX data.

However, much fewer papers have carried out comparative studies between data providers. Jobmann et al. (2014) were the first ones to compare the altmetric coverage and counts of ImpactStory, Altmetric Explorer, Plum Analytics and Webometric Analyst by research areas. Their results show important divergences between services. Plum Analytics is the platform that better covers Mendeley and Facebook data, while Altmetric.com stands out gathering blogs, news and CiteULike data. Zahedi et al. (2015) explored the consistency of data across three altmetric aggregators: Altmetric.com, Mendeley and Lagotto. They also detected significant differences, finding that Altmetric.com gathers more tweets, but it is less accurate collecting Mendeley readers. Baessa et al. (2015) evaluated several altmetric providers for their institutional repository and they recognized that Altmetric.com has a better coverage of blogs, news and government documents, while PlumX is most exhaustive covering different formats. Kraker et al. (2015) studied the collection of research data in three data sources: Figshare, PlumX and ImpactStory. They observed that PlumX detects considerably more items in social media and also finds higher altmetric scores than ImpactStory. Peters et al. (2016) extended their former study (Peters et al. 2015) with the inclusion of Altmetric.com. Their results confirmed that PlumX is the best provider for covering non published materials such as research data. The most recent comparative approach was performed by Meschede and Siebenlist (2018), who compared Altmetric.com and PlumX, finding that less than half of the publications analyzed are included in Altmetric.com, while PlumX covers almost the totality (99%). Zahedi and Costas (2018) performed the most exhaustive comparison between data providers, finding substantial differences in the metrics offered by these platforms.

Many of these studies have focused more on coverage of altmetrics than on counting events and, in the case that events were studied, they were analyzed at research area or source level. This study thus attempts to investigate these counting differences comparing the altmetrics of each article in three representative sources. We consider that this method would bring more detailed results.

Objectives

The main purpose of this study is to compare the results of three important altmetrics providers of the same set of publications. The objective is to determine the differences and similarities between services when they extract and gather these metrics, detecting inconsistencies between different results. These outputs would allow the scholarly community to value the quality and reliability of these services as data providers. Several research questions were formulated:

  • What are the differences between data providers according to the proportion of altmetric events captured?

  • What are the counting inconsistencies between the metrics provided by these altmetrics services? Is it possible to quantify these differences?

  • Could these altmetric providers be used for research evaluation equally?

Methods

Three of the most important altmetric providers were selected for this study: Altmetric.com, PlumX and Crossref Event Data (CED). Two main criteria were applied to select these platforms. First, altmetric providers should have a global coverage, not limited to specific publishers, disciplines or regions. In this way, altmetric services of publishers (i.e. PLOS, Nature Publishing Group, etc.) were excluded. Second, these services should offer possibilities to extract data from them, using a public API or scraping their web pages. Due to these restrictions, ImpactStory was not analyzed because it does not provide an API key (ImpactStory 2017). Both conditions are necessary to be fulfilled in order to make a fair comparison among platforms.

Altmetric providers

PlumX PlumX (plu.mx/plum/g/samples) is a provider of alternative metrics created in 2012 by Andrea Michalek and Michael Buschman from Plum Analytics. This product is addressed to the institutional market, offering altmetric counts of publications for particular institutions. PlumX is the aggregator that offers more metrics, including citation and usage metrics (i.e. Views and Downloads). It covers more than 52.6 million of artifacts, being the largest altmetric aggregator (Plum Analytics 2018). However, only a limited set of institutions make their profiles public, therefore, not all the publications tracked by PlumX are accessible. In 2017, Plum Analytics was acquired by Elsevier (www.elsevier.com) and its altmetric information was added to the Scopus database (Elsevier 2017). This change could influence data sources, because many of their sources are EBSCO services (www.ebsco.com), the former proprietary.

Altmetric.com It was the first altmetric provider and it was initiated in 2011 by Euan Adie, with the support of Digital Science (www.altmetric.com). Unlike PlumX, Altmetric.com is centered in the publishing world, signing agreements with publisher houses to monitor the altmetric impact of their publications. This information is accessible through a public API. Today, Altmetric.com tracks the social impact of close to 9 million of research papers (Altmetric.com 2018). However, this platform does not include metrics about citations and usage in its public API. In 2015, Altmetric.com launched Altmetric for Books, an exclusive service for books and book’s chapters (King 2015).

Crossref Event Data (CED) CED is the youngest service, created in 2016 and officially released in 2017. Due to this, the platform claims that the service is still in beta (www.crossref.org/services/event-data). Unlike the previous ones, CED is not a commercial site and it provides free access to their data through a public API. Another important difference is that it does not provide metrics, but it only displays information about each altmetric event linked to a DOI identifier. For instance, it shows the information about the mention of an article on Twitter (date, user, tweet, etc.), but it does not show a count of the number of tweets. For that reason, CED’s data would have to be processed to be comparable with the other services.

Data extraction

An initial sample of 67,147 research articles was obtained from the public pages of PlumX. This platform was used as the starting source because it does not permit to retrieve publications by searching by DOI. This drawback makes the search of publications from other altmetric sources in the PlumX database impossible. Another important reason is that this platform only shows altmetric information of publications from institutions that make their profiles public. Therefore, we may suppose that those publications could also be indexed in the other services because they have no problem to display their records. Another reason is that PlumX includes both publications that have and that do not have altmetric events, which also allows us to detect articles that may have altmetric events in other aggregators.

The first step was to identify the organizations that make their profiles public. A search in Google (site:plu.mx) retrieved the web pages of 17 research institutions. The sites that contribute most publications to the sample are: Mount Sinai Health System (19,827), Concytec (8141), University of Pittsburgh (8103), Seoul National University, College of Medicine (7871), King Abdullah University of Science and Technology (7281), University of Helsinki (5287), Xi’an Jiaotong University (2781), International Islamic University of Malaysia (2350), Universidad del Rosario (1648), Georgia Southern University (1083), and Centro Hospitalar e Universitário de Coimbra (1009). The remaining institutions contribute with fewer papers. Only article types published after 2013 and with a DOI were selected, being in total 67,147 research papers. This selection process could introduce a bias because the articles were not randomly gathered and the sample was selected from specific organizations. Later on, we will discuss this possible limitation with previous studies.

This initial sample of research papers was searched again in Altmetric.com using its public API (api.altmetric.com), and in CED using its API (query.eventdata.crossref.org) as well. In this way, three samples were obtained that were comparable amongst the providers. A routine in SQL was written to crawl and extract the altmetric information from the three providers. The samples were obtained in May 23rd, 2017.

Metrics

Because each service has different numbers and types of metrics, it is necessary to define the metrics analyzed in this study. These indicators have to be common to the three platforms and they have to be counted in a similar way. Five metrics were compared:

  • Blog posts Number of blog posts that mention a research article. Each service has its own list of blogs where the mentions are searched. Differences between services could be caused more by the number of indexed blogs than by the way in which the mentions are extracted. The only service that provides a number of blogs is Altmetric.com, which monitors 11,000 blogs (Williams 2017). The other platforms do not make public the list of blogs nor the number of blogs covered.

  • News Number of news feeds that mention a research article. In the manner of blog posts, each platform has their own list of news media. Therefore, differences between services could also be due to coverage criteria. Altmetric.com collects a list of 1300 news outlets, which could be expanded to 80,000 thanks to a partnership with Moreover.com (A LexisNexis Company) (Williams 2015). The number of news media covered by PlumX is unknown. CED defines this section as “web” and it includes links to other sites different from blogs, thus it is possible that many of these counts do not belong to news mentions.

  • Tweets Number of tweets that cite a research article. In this case, the source is the same for the three sites. Possible differences might be found in the way that they obtain their data (i.e. data providers) and how they search the article in the source (i.e. use of identifiers). It is important to mention that Altmetric.com shows only the number of individual accounts (tweeters) that tweet or retweet a publication, while PlumX and CED include all the tweets that mention a publication.

  • Wikipedia citations Number of Wikipedia entries that include a citation to a research article. This is a unique source as well, and the differences might be caused by the way in which the citations are extracted.

  • Mendeley readers Number of users in Mendeley that include a research article in their libraries. As the two previous metrics, differences in this count would be caused by technical reasons.

Results

Distribution of altmetrics by provider

Table 1 presents the number of documents in the three samples that have altmetric events. Overall, Altmetric.com includes 28,123 articles (41.84%) of the initial sample of 67,218 papers in PlumX. This proportion shows that not even one out of every two articles has statistics in both services, making evident a low overlapping rate. According to CED, the overlap is much higher because 63,637 articles (94.67%) of the PlumX’s sample are as well registered in CED. However, only 2635 (3.92%) of these documents have altmetric events. This absence of data could be due to the service is still in beta and many of the events are not counted yet.

Table 1 Number of articles with at least one altmetric event by provider
Full size table

Observing the distribution of altmetric events in the three services, one can detect that there are important differences according to the percentage of articles with an altmetric event. PlumX, the original sample, shows a high proportion of articles bookmarked in Mendeley (77.74%) and mentioned in Twitter (17.56%), and a lower proportion mentioned in blogs (2.66%), news (1.27%) and Wikipedia (1.74%). However, in the case of Altmetric.com, these percentages are much higher. Thus, the proportion of articles saved in Mendeley is 95.76% and mentioned in Twitter is 91.38%. These high percentages are due to Altmetric.com only recording articles that have at least one altmetric event, whereas PlumX indexes articles without distinction. If the articles without altmetrics (14,251) are subtracted, PlumX would then include more articles with readers (98.66%) than Altmetric.com (95.76%). However, this subtraction does not explain why the proportion of tweeted papers in PlumX (22.28%) is much lower than in Altmetric.com (91.38%). This significant small proportion of tweets was already noticed by Jobmann et al. (2014) and this could be due to the fact that PlumX did not use Gnip (support.gnip.com/sources/twitter/), the official Twitter data provider, until 2016.

Regarding CED, the number of articles with an altmetric event is very low, with only 2635 papers (3.92%) from the initial PlumX’s sample. This demonstrates that CED is still in beta and the coverage of altmetric events is for now incomplete. Taking only into account papers with at least one altmetric event, 34% articles have a Twitter mention and 32.03% a citation in Wikipedia. This last is the highest percentage of all the three providers, and it suggests that CED has a special coverage of Wikipedia articles.

Counting differences between providers

The main objective of this work is to analyse the counting differences between altmetric providers. To this end, a new distribution was created from the differences between two altmetric aggregators. For example, distribution Altmetric.com/PlumX shows the counting differences of one metric (blog posts, tweets, citations, etc.) between Altmetric.com and PlumX by each article. The mean of an altmetric distribution (a) is calculated by adding the counting differences of each document (d) in two providers (p), and then dividing them by the number of documents (D) with an altmetric event in both providers (p1 ∩ p2). This is an easier and clearer way to present and analyze the differences between providers as well as it allows to detect systematic or random errors. In this way, the dispersion (Standard Deviation, SD) informs us about the amount of differences between two services. The more pointed and slender a distribution, the less different the two altmetric providers are to each other.

$${\text{Mean}}_{a} = \frac{{\sum d_{p1} - d_{p2} }}{{D_{p1\; \cap \;p2} }}$$

Blogs

Figure 1 and Table 2 present the distribution of the counting differences between Altmetric.com, PlumX and CED regarding blog posts. Generally, Altmetric.com is the service that counts more blog posts. In average, 1.52 more posts than PlumX and 2.21 more than CED. These differences suggest that Altmetric.com has the largest list of blogs. In fact, it gathers more than 11,000 blogs. By contrast, the number of blogs that PlumX and CED track is unknown. However, the fact that 13.92% of the articles contain more blog posts in PlumX than in Altmetric.com and that only 10.66% of the articles show the same count in both services would mean that the PlumX’s list could be quite different to the Altmetric.com one. It is then possible that PlumX covers an important number of blogs that Altmetric.com does not manage. CED shows more significant differences with respect to the other providers. In almost all the cases, CED contains less blog mentions than Altmetric.com (2.21) and PlumX (1.54). These high differences suggest that the CED’s blog list is short and overlapped with the other providers.

Fig. 1
figure 1

Histograms of counting differences between providers according to blog posts

Full size image
Table 2 Some parameters of the three comparative distributions according to blog posts
Full size table

News

Figure 2 and Table 3 show the distribution of the counting differences between providers at the level of news mentions. The number of Articles in Table 2 and following tables do not correspond to the ones reported on Table 1. This is because the Table 1 only includes articles indexed in a platform, whereas Tables 2, 3, 4, 5 and 6 include articles in one or the other provider that have an altmetric event. This fact introduces slight differences in the total counts. Altmetric.com is the service that gathers more comments on news media sites. In average, it collects 6.79 more than PlumX and 6.82 more than CED. These differences cause a high standard deviation of the Altmetric.com’s distributions. This fact makes evident that media coverage in Altmetric.com is much wider than in the other services. Some possible causes could be that Altmetric.com can track the mention of articles in the media without a specific identifier (Liu 2013). Another possible reason could be its partnership with Moreover.com, a news data provider, which allows Altmetric.com to collect more news media (Williams 2015). According to CED, it is possible that many of these counts do not exactly belong to news mentions. Therefore, the percentage of real news mentions could be even lower. Differences between PlumX and CED are smaller (1.37), which could mean that both services have similar news sources.

Fig. 2
figure 2

Histograms of counting differences between providers according to news

Full size image
Table 3 Some parameters of the three comparative distributions according to news
Full size table
Table 4 Some parameters of the three comparative distributions according to tweets
Full size table
Table 5 Some parameters of the three comparative distributions according to Wikipedia citations
Full size table
Table 6 Some parameters of the comparative distribution of Altmetric.com/PlumX according to Mendeley readers
Full size table

Tweets

Figure 3 and Table 4 present the distribution of the counting differences between Altmetric.com, PlumX and CED regarding Twitter mentions. On average, Altmetric.com again counts 3.72 more tweets than PlumX and 9.21 more than CED. It is interesting to notice that there is a slight symmetry between Altmetric.com and PlumX. 13.07% of the articles have more tweets in PlumX than in Altmetric.com, 11.34% show the same number of mentions in both platforms and 35.39% have one more tweet in Altmetric.com than in PlumX. This symmetry could be due to Altmetric.com only counting tweeters, different tweets from the same account are hence not counted. By contrast, PlumX does count those tweets. This could explain why there are 13.07% of articles that have more tweets in PlumX than in Altmetric.com. However, the remaining 75.55% of articles with more tweets in Altmetric.com than in PlumX suggest that there is a systematic error. This error could be caused by the employment of different data providers or by the use of distinct criteria in order to count tweets. For example, a possible cause could be that PlumX did not use data from Gnip, the official Twitter data provider, until 2016. This could be the reason why some tweets were not properly matched. A manual inspection of some articles showed that the range of URLs that PlumX uses to identify mentions is much smaller than those used by Altmetric.com. This absence of completeness would produce the systematic underperforming of PlumX according to Altmetric.com. With regard to CED, it is interesting to notice the great difference with PlumX (14.32) and Altmetric.com (9.21). In this case, there are more systematic variations and these could be due that CED only compiles tweets that contain the DOI prefix or the main landing page of the article, dismissing other links to repositories, publishing platforms, etc. (CED 2017).

Fig. 3
figure 3

Histograms of counting differences between providers according to tweets

Full size image

Wikipedia

Figure 4 and Table 5 exhibit the distribution of the differences between the three data providers according to Wikipedia citations. In this case, differences are smaller than in the other metrics. This fact is demonstrated by the low averages and the high percentage of articles with the same count value. CED is the platform that gathers the most Wikipedia citations on average, with 0.6 more citations than PlumX and .24 more than Altmetric.com. In addition, PlumX obtains .73 more mentions than Altmetric.com. This metric displays the highest percentages of articles with the same count value, being 66.28% in Altmetric.com/PlumX, 36.66% in PlumX/CED and 36.66% in Altmetric.com/CED. The fact that Wikipedia citations shows such small differences across providers could be caused by the low number of references in Wikipedia, by the easy way with which the mentions are computed and by the fact that only one source is tracked. In spite of these facts, differences between CED and the other providers are most probably caused by Altmetric and PlumX only tracking mentions on the English, Finnish and Swedish-language Wikipedias (Altmetric Support 2017), while CED explores all languages on Wikipedia. Another possible reason is that Altmetric.com does not extract mentions from incomplete references or without an identifier. However, PlumX counts mentions only matching the title of the article (Plum Analytics 2016). This difference would explain the slightly higher number of mentions computed in PlumX according to Altmetric.com.

Fig. 4
figure 4

Histograms of counting differences between providers according to Wikipedia citations

Full size image

Readers

In the case of Mendeley readers, only Altmetric.com and PlumX include this altmetric indicator. CED does not offer information about this reference manager. Figure 5 depicts the distribution of the differences between Altmetric.com and PlumX compiling Mendeley readers, and Table 6 shows some parameters about this distribution. The histogram is nearly symmetric, with an important proportion of similar counts (28.28%). However, there is a slightly higher proportion of negative values, which means that PlumX regularly counts more readers than Altmetric.com. In fact, the mean of the differences (− 2.5) confirms this counting error. This result is significant because we are talking about only one source and the metric is already counted by the source, that is, providers do not have to aggregate readers. This error is explained by the fact that Mendeley allows to include references that were not previously registered in its Public Catalog. The risk of this practice is that it creates duplicate records. In this case, PlumX aggregates the readers of possible duplicate records from references with similar titles, years, authors, etc. Contrarily, Altmetric.com only counts the readers from a unique identifier, dismissing readers from duplicate items. This could be the reason why on average, PlumX shows more readers than Altmetric.com.

Fig. 5
figure 5

Histograms of counting differences between Altmetric/PlumX according to Mendeley readers

Full size image

Discussion

The results show that there are important differences between providers when they collect and count altmetrics. In general, Altmetric.com captures more altmetric impact than the other services. Except for the number of Mendeley readers and Wikipedia citations, Altmetric.com exhibits the highest percentage of papers with at least one altmetric event for each metric. Perhaps, the most significant difference is regarding the coverage of Twitter. Almost all the articles in altmetric.com have been mentioned on Twitter (91.38%), a slightly higher proportion than that found by Robinson-García et al. (2014). Meanwhile, only 34% of articles in CED and 22.28% in PlumX were tweeted. This last proportion is similar to the one found by Meschede and Siebenlist (2018) (21%) and it was already noticed by Jobmann et al. (2014), mainly with regards to PlumX. This variation could be due to the fact that this aggregator did not use Gnip, the official Twitter data provider, until 2016. Another reason could be that the number of identifiers used by PlumX and CED is lower than the ones used by Altmetric.com, causing a reduction of the number of captured tweets. Altmetric.com also has a better coverage of blogs and news, with more than twice as many mentions. However, there are some metrics in which PlumX and CED perform better than Altmetric.com. For instance, PlumX (98.66%) covers more articles with Mendeley readers than Altmetric.com (95.76%); and CED captures more Wikipedia citations (32%) than Altmetric.com (2.99%) and PlumX (2.21%). These coverage differences allow us to imply that, despite Altmetric being the comparative best provider, it would be recommended to select particular providers to analyze specific metrics. Thus, while Altmetric.com would be the best source for news, blog posts and tweets, PlumX would be more suitable to study Mendeley readers and CED to study Wikipedia mentions.

According to counting differences, the results also show that there are important variations between providers in the five metrics analyzed. Altmetric.com is the service that has a better coverage of blogs, news and tweets, a fact also confirmed by Baessa et al. (2015). In these metrics, Altmetric.com shows more news mentions than PlumX and CED, being more visible in news feeds, with 6.79 more news than PlumX and 6.82 more than CED. This result makes it clear that Altmetric.com is more concerned with having a complete list of blogs and news, which lets it capture the widest media impact possible. Altmetric.com also gathers more tweets than their competitors (3.72 more than PlumX and 9.21 more than CED), caused perhaps by the use of more unique identifiers such as DOIs, repository handles, landing pages, etc.

By contrast, PlumX achieves a better coverage of Mendeley readers (2.5 more) and Wikipedia citations (.73 more) in comparison to Altmetric.com, although these differences are smaller and probably due to technical criteria. For example, Altmetric.com obtains fewer Mendeley readers because it fails to aggregate duplicate records. In comparison with CED, PlumX has better figures in all metrics, except for Wikipedia citations (.6 fewer than CED). This difference is especially significant when counting tweets, where PlumX captures 14.32 more tweets than CED.

These results have also indicated that CED is still a beta service. The number of collected events is rather low in comparison with the other providers, whereas this difference is more noticeable in Altmetric than it is in PlumX. This fact is evident in all the metrics, with the only exception of Wikipedia citations. CED covers all Wikipedia languages, collecting slightly more citations than Altmetric.com (.24) and PlumX (.6).

These results have strong implications for research evaluation because they demonstrate that the counts are not uniform across providers. This means that the altmetric impact of scientific publications is different according to the provider used and it casts doubts on the reliability of these tools for measuring altmetric impact. These differences are mainly due to systematic errors caused by coverage problems (blog posts, news, Wikipedia citations) or technical limitations (tweets, Mendeley readers). Such results recommend using specific providers if we want to track particular metrics. In this sense, Altmetric.com is the most suitable tool for blogs, news and tweets, PlumX for Mendeley readers and CED for Wikipedia citations. On the contrary, if we want to analyze several metrics, these results, then, suggest to use several aggregators. In this way, altmetric studies could be more consistent selecting the most appropriate sources depending on the metric to be analyzed.

This study could be limited by the sample gathering process. Data were not randomly extracted from PlumX, which could influence the results about the coverage of Altmetric.com and CED in comparison to PlumX. However, this fact would be less relevant when the providers are compared according to counting differences. This is because these counting differences are independent of the way in which the sample was taken. This statement is reinforced when the obtained figures are comparable to previous studies.

Conclusions

There are differences in the amount of articles with altmetric impact in each provider. Altmetric.com is the service with the highest percentage of publications with some altmetric event, surpassing the other providers in almost all the indicators. The only exception is Wikipedia citations where CED obtains very high ratios and in Mendeley readers where PlumX slightly exceeds it. Otherwise, PlumX yields a significantly low proportion of tweeted papers and CED only stands out with extracting Wikipedia citations.

There are also important differences between providers when it comes to altmetric counts. Overall, Altmetric.com is the platform that collects more altmetric events, obtaining the best scores on blog posts, news and tweets. PlumX is only better than Altmetric.com when it compiles Mendeley readers, but it is limited when it comes to extracting blogs, news and tweets. CED is still a beta product with a short range of altmetrics and a deficient collection of news and tweets. However, it is the best service for extracting Wikipedia citations.

Finally, these results suggest that Altmetric.com, PlumX and CED cannot be used separately for research evaluation because there are important differences according to coverage and counting procedures. Therefore, it is recommended to select particular providers for the analysis of specific metrics (i.e. CED for Wikipedia citations, PlumX for Mendeley readers) or to combine several sources if one wants to perform a global analysis about the interaction of several altmetrics.