Keywords

1 Introduction

Scientific publications are the most important means for the research community to communicate findings. To advance, researchers build on such findings and, to this end, cite them [4]. Consequently, especially citation counts, average citations, and the number of published papers are often considered as indicators for importance, visibility, and impact of researchers. Thus, such metrics are used not only to evaluate research [6, 11], but also to rank publications, for example in Google Scholar [1]. However, there are discrepancies in how these metrics are determined [13, 21]. Moreover, due to the increasing number of publications [24] and technical limitations of digital libraries [27], such rankings can bias which publications are found. For example, researchers often rely on these metrics, especially citation counts [9], when they retrieve existing publications—threatening the completeness of their search and potentially missing important research. Still, some information that could indicate quality are publicly available, in contrast to the actual publications, allowing to compute metrics and, thus, facilitating the daily work of researchers. To this end, it is necessary to understand the publicly available information, how they are connected, and if they can indicate quality.

Utilizing such information can be particularly interesting in assessing the quality of retrieved primary studies for a literature analysis, especially systematic literature reviews [14]. While in some domains the citation count is used during this phase [9], there are ongoing discussions and criticisms on how well it represents the quality of research [4, 18, 22]. In particular, the context in which the research has been published, for example, the year, availability, and publication venue can have an impact. With an increasing number of publicly available information, quality assessments may be improved considerably. While a detailed quality assessment arguably includes reading the selected publications, such metrics can provide initial insights.

Using such metrics is especially interesting, because many digital libraries do not block the necessary information sources with pay-walls—in contrast to the full papers that usually are not open-access. In this paper, we investigate several of these libraries, namely the ACM Digital Library, CiteSeerX, DBLP, Google Scholar, IEEE Xplore, ScienceDirect, Scopus, Springer Link, and Web of Science. We derive an overview of the available information in each library that can be utilized to assess quality. Furthermore, we derive a taxonomy that displays the relationships between the information and discuss how they can indicate quality. Having such insights can help researchers to improve the automation of literature analyses and facilitate the identification of relevant, qualitative publications.

The remainder of this paper is structured as follows: In Sect. 2, we describe our research approach and results, providing an overview of the information available in digital libraries and of our taxonomy. Within Sect. 3, we discuss the implications concerning the quality of information in digital libraries as well as their suitability for quality assessments of publications. Afterwards, we discuss the major threats to validity in Sect. 4, describe related work in Sect. 5, and conclude in Sect. 6.

2 An Overview of Digital Libraries

We base our analysis on nine digital libraries that are regularly used for reviews in computer science [16, 30]. Note that we also include Scopus and Web of Science as a comparison, despite the fact that they do not provide free access to most of their information. For each of these libraries, we search for a set of publications on the web interface and investigate the information that are available. Moreover, we analyze author and publication venue pages to understand which additional information on these are provided. This procedure has been performed separately by the first two authors. Afterwards, we merged the results, checked them again, and analyzed the identified information as well as their relationships.

Table 1. Overview of the information provided () by digital libraries that may be useful for assessing quality of publications.
Full size table

2.1 Research Objective

In the remainder of this section, we present the results of our analysis concerning the information provided in different digital libraries that may be useful for quality assessments. We remark that we do exclude standard bibliographic data, such as a publication’s authors, year of publication, and page numbers. These information are provided by all the digital libraries and are already used as parameters to determine a publication’s quality. Our focus remains on more specific information that are supported and made available exclusively by some libraries, such as, citation information and metrics on publications, authors, and venues. Thus, we can summarize our research objective:

  • RO We aim to compare the publicly available information of different digital libraries to investigate their usability for quality assessments.

We remark, that some of the considered libraries are limited in the number of results they retrieve for a search query [27]. Furthermore, additional restrictions can apply that are also due to technical reasons. For instance, Web of Science can only report citations for up to 10,000 search results. If a query returns more publications, the citation count feature becomes inactive. Furthermore, there can be differences when using APIs [27] and it is important to consider that digital libraries evolve. Consequently, we provide an overview on the currently available information, but this may change over time.

2.2 Results

In Table 1, we summarize all information we identified in the investigated digital libraries. We can see that there are several discrepancies among these libraries. For instance, most of them (except DBLP) count the number of citations for a publication, while only few of them have other metrics, such as self-citations or Field-Weighted Citation Impact (FWC-Impact). This is depending on the purpose of each digital library: Does it only help identify literature (e.g., DBLP), aim to summarize existing research and provide an overview on publications and authors (e.g., Scopus, Google Scholar), or is it a publisher’s library (e.g., ACM Digital Library, IEEE Xplore)? For example, the h-index [3] is an author-level metric intended to measure the productivity and citation impact of a researcher that is only supported by Google Scholar and Scopus. Interestingly, none of the libraries seems to be “complete” in the sense that it supports all or at least most information that is available in other libraries. Closest to this may be Scopus, but it does not provide all its information for free.

The only information that is rather consistently provided is the citation count, partly with references to the citing publications. Only DBLP is an exception in this regard, due to its intended purpose and limitations [17]. In contrast, DBLP is one of the few libraries that allows users to link the search results either to an author’s or venue’s profile page and list corresponding publications.

We find several additional information that are interesting and can improve automated quality assessments, but are supported only by few libraries. Consider, as an example, self-citations, which refer to the number of citations that belong to one of the publication’s authors: They are reported separately in CiteSeerX, Scopus, and Web of Science. While such citations are important to refer to own, previous work, it is unclear to which extend they represent quality as the count could be biased if authors misuse self-citations [28].

Overall, the results seem to verify that the citation count is considered to be an important information and it is made publicly available in most libraries. As this number reflects on the impact of a publication, it may be possible to use this measure for an assessment of quality. However, there are several other information that are tangled with the number of citations, for example, publication year, self-citations, and index terms that are often neglected. In order to improve this situation, we need to refine our understanding of the relationships between the different information. To this end, we put the identified information from Table 1 into a taxonomy that helps us to structure them.

2.3 A Taxonomy of Available Information

We display our taxonomy in Fig. 1. The top-level represents the categories of information we already used in Table 1: Publication, author, and venue. However, we introduce two additional layers within the taxonomy: Measures refer to concrete metrics and numbers that are computed by the digital libraries and can potentially reflect on the impact of research. Meta-data are not metrics, but a collection of information that can indicate different quality aspects. For example, the list of references may indicate well-founded background, a library’s index terms can more precisely cluster publications, and the list of co-authors may show established, reliable cooperators. Again, we can see that across all information, the citation metric and its sub-metrics seem to be established. Still, there are some relationships (and fuzzy distinctions into our layers) between the information, for example, the number of downloads of a publication will somehow impact the number of its citations. In the remaining paper, we discuss the information shown in our taxonomy.

Fig. 1.
figure 1

Taxonomy of information provided by digital libraries.

Full size image

3 Discussion

In this section we discuss dependencies of the identified information and their potential to indicate quality. We further discuss some shortcomings of digital libraries, mainly highlighting the issues of incomplete information. Moreover, we find that inconsistencies of information across libraries makes comparative analysis between them challenging or even impossible.

3.1 Assessing Publication Quality

To assess the quality of scientific studies for a literature analysis, specifically systematic literature reviews, certain criteria should be defined [14]. For automated support of such an analysis, digital libraries provide some quantitative data that are easier to capture than those that have to be qualitatively captured by reading each publication. Such data can be used as indicators of the overall interest in the specific research work and, thus, may aid quality assessments. Our findings (cf. Table 1, Fig. 1) indicate that there are various quantitative metrics supported by specific libraries. Applying combinations of these metrics could be beneficial for researchers when assessing the quality of a publication. For instance, connecting publication count and publication year with the evolution of an author’s citations may better indicate the significance of their contributions for the scientific community than the sole citation count. Similar approaches to combine different metrics have been undertaken by some libraries and researchers, resulting in metrics, such as the FWC-Impact or h-index. Similarly, the number of downloads of a publication and those received by an author is useful to perceive the overall impact. However, it must also be noted that these counts highly depend on the availability of a publication for other researchers. Due to certain restrictions imposed by publishers and digital libraries, for example, IEEE Xplore, their content is not publicly available [27]. Consequently, it is possible that a good quality publication has a lower number of (measured) downloads and citations, as researchers are not able to access it.

During a literature review, researchers usually use some common factors for the quality assessment, including the impact of a publisher, citation counts, popularity of authors, and number of pages [26]. Citation analysis is one of the most important technique that could facilitate researchers for evaluating research [4]. However, there are other factors that influence the citation count, such as, the year of publication. Consequently, researchers can define different metrics to evaluate the quality of a publication that may include the citation count received by an author, the impact factor of the publishing venue, and the number of times a publication has been downloaded. Such information can communicate the overall interest of the scientific community and the role of such factors may be considered for the assessment.

The findings that we display in Table 1 show that digital libraries provide several information, especially about the citation count of the publications. However, there is inadequate support for the self-citations count, which may be an important factor that must be considered. Self-citations, if unbiased, fairly presented by authors and accurately citing the previous works performed by the co-authors and collaborators, could be a suitable quality indicator [12]. However, there is still extensive evidence on how poor the citation of previous work can be in the scientific literature [7, 23]. This may mislead researchers aiming to evaluate a research work and propagate invalid analyses. In this regard, researchers must aim to determine relevant references to minimize bias and improve reliability of the citation counts.

Certain factors can also influence the data for evaluating quality of a publication. The publication count of authors over the years is an indicator of their contributions, however, it is also possible that these may not necessarily reflect their impact [18]. In some situations, it is possible that authors keenly contribute towards a research topic through valid and reliable publications. However, they may become less active later on—which can negatively influence such quantitative data. Thus, the metrics depending on such factors must be examined and their effects on the information must be minimized for valid quality assessment.

3.2 Dependencies of Information

The open data supported by various digital libraries may provide an indication of the significance of a research work. However, we must also analyze the usability of available information for the quality assessment. As we mentioned earlier, the most consistent information that libraries support are the citation counts for a publication. Still, advanced citation metrics for authors and publishing venues are only provided by few libraries. The quality assessment criteria that researchers usually define to perform the evaluation are based on the cumulative effect of the authors, publishing venues, and publications.

Citation metrics that measure the influence of individual researchers and authors, such as h-index and i10-index, are also important information useful to assess a publication. It is evident in Table 1 that ACM DL, Google Scholar, IEEE Xplore, Scopus, and Web of Science support several useful items of information regarding the authors. In contrast, while being popular digital libraries in the field of computer science, CiteSeerX, ScienceDirect, and Springer Link lack the support of such features. These factors are a good indicator of the author’s credibility and the reliability of the performed research. Furthermore, meta-data of authors, such as their affiliation history, number of downloads and citations received per publication, can also reflect quality. These significant aspects represent the expertise of an author regarding the topic and how well their work is able to capture the attention of the audience.

Moreover, digital libraries may support features that allow users to obtain an understanding regarding the impact of publishing venues. Through our analysis, we conclude that there is inadequate, or no support at all, for information regarding publication venues across digital libraries (except Scopus). Although, it is partially possible to assess articles based on the publication type, for instance, journal papers are often considered to have a greater impact compared to conference and workshop proceedings or parts of a book. However, to improve the evaluation, additional information, such as, impact factor, SNIP, and acceptance rate, are useful. These information represent the importance and standard of a publishing venue, thus, supporting the quality assessment of publications.

3.3 Incomplete Data

The digital libraries we investigated are commonly used by scientific researchers to retrieve promising publications. However, during our investigation, we realized that information of the unique parameters are not always provided for all publications. The probability of incomplete data hamper the analysis even if only a single digital library is being examined [27]. For example, if the acceptance rate is under consideration for determining the quality and is missing for any publication, it would become unreasonable to investigate this parameter completely. Thus, developers of digital libraries must ensure completeness of the data provided to the users, allowing them to perform valid quality evaluations.

Similarly, it is also possible that the information about the author in some libraries, such as, Google Scholar, is missing. Due to such limitations, an evaluation based on data that is missing for some results makes it challenging to perform automated analyses. Similarly, only few libraries collect information on venues or connect them. In particular, there are often incomplete information, for instance, on acceptance rates or the proceedings series. The latter prevents time series assessments of conferences, as the publications are not linked to each other. We also realized that the computed metrics and meta-data provided by digital libraries are solely based on the content present in this database. Due to variations in the coverage of publications across digital libraries, for example, Google Scholar has a larger coverage compared to Scopus, using them in isolation does threaten consistency and completeness.

3.4 Uncertain Parameters

Generally, the citation statistics provided by libraries are based on the scholarly content available in the specific library’s database itself. Thus, overview libraries, such as Google Scholar, may be better sources for citation counts as they provide an overall count of a publication. While using most of the other libraries, the analysis of this parameter may lead to an incomplete result, as the counts are confined within a certain database only.

Furthermore, the author statistics are often only consistent in libraries if the authors maintain their profiles in each library. Thus, such information may be insufficient or not available for other authors without a profile. To obtain certain interpretation of the information metrics provided by digital libraries, it must be clearly stated which parameters are under observation.

3.5 Inconsistent Data

Based on our findings in Table 1, we notice the inconsistency of parameters supported by different digital libraries. Assessing the quality of publications obtained from a number of libraries for any literature analysis becomes quite challenging, or even impossible, due to inconsistent information provided. For example, analyzing the publishing venue is only possible by using Scopus, as the rest of the digital libraries have limited or even no support for such features.

The inconsistency in the features supported by different digital libraries is a limitation for automated quality assessments. For instance, the same search query may not be reused in different libraries, due to missing search fields [27]. Thus, to overcome this problem, the support for significant parameters that reflect quality must be enabled across libraries.

3.6 Indicating a Publication’s Quality

The research metrics can be used inappropriately: Using one metric in isolation may lead to undesirable outcomes. Thus, using multiple evaluation metrics based on the different types of measurements, such as, document, authors, and venues, is helpful. For example, the author’s measures and meta-data along with the document metrics could be used to indicate the credibility of a publication. Thus, the identified information metrics can be useful to answer important questions, despite their limitations:

  • How credible are the findings of a study regarding a research topic?

  • What is the impact of a study on the overall research being performed?

  • How relevant is a certain publication in a specific research domain?

Overall, we argue that using multiple metrics to assess the quality of literature is beneficial—at least improving over the citation count as a metric. A combination of such information provides a clearer understanding of the significance of research.

3.7 Limitation of Using Meta-information

On a final note of our discussion, we want to emphasize that a sole automated quality assessment based on meta-information is usually not feasible. It can help to indicate important and reliable research, helping to reduce the set of publications. However, only by carefully reading a publication, researchers can assess its relevance, importance, and quality. The purpose of automating such analyses is to facilitate the time-consuming and effort-full process of literature analysis, especially in the context of systematic literature reviews.

4 Threats to Validity

We see the following three major issues that may threaten the validity of our results: First, we focused on a subset of all available digital libraries. However, we relied on the arguably most important ones for software engineering, reducing this threat. Moreover, our research goal is to assess the available unique information (other than the standard bibliographic meta-data, such as, year of publication) and their suitability for quality assessments. Thus, we may have missed some information, but we arguably captured the most regular ones and our discussions on their suitability are not threatened.

Second, we derived all information and discussed them by ourselves. Consequently, other researchers may face more access restrictions (e.g., for Scopus and Web of Science), which limits the availability of these information. While our focus is on the openly available data (e.g., ACM Digital Library, DBLP, IEEE Xplore, and Google Scholar), we used such libraries as references. Thus, other researchers may come to different conclusions, depending on their access restrictions, the considered subject systems, and implications they derive. However, we carefully described our procedure and derived our results with caution, meaning that they can be reproduced by others.

Third, digital libraries evolve over time, with a prominent example being CiteSeer’s update to CiteSeerX [8, 20, 25]. Consequently, the available information may change over time, making our results obsolete. Of particular interest may be the extension with further metrics or the consolidation of digital libraries into single repositories—similar to Scopus, DBLP, and Google Scholar—to solve technical issues [27]. While we cannot overcome this threat, capturing the state-of-the-art is important to investigate the evolution of information in digital libraries. Additionally, several pieces of information are likely to be always relevant, for example, the citation count.

5 Related Work

Concerning related work, we are aware of several works that are discussing the suitability of different publication metrics [2, 3, 29]. Mostly discussed may be the citation count and its meaning [1, 15, 21], with particular attention being paid to self-citations [12, 28]. Other works focus on the evolution [8, 20, 25], overlap [5, 19], and technical issues [10, 27] of digital libraries. Still, none of these papers is based on an analytical comparison of established digital libraries, except for Neuhaus and Daniel [21]. Thus, this work may be the closest to ours, but does only focus on the search functionalities of the considered libraries. In contrast to existing works, we are focusing on—especially publicly available—information that can be used to assess the quality of publications. We are not aware of a similar overview and taxonomy as the one we derived.

6 Conclusion and Future Work

In this paper, we provide an overview of the available information in various digital libraries that can support the quality assessment of scientific publications. We present a first insight through our analysis and aim to continue further research in this regard. Our investigation shows that there are some useful features, such as, self-citation count and unified indexing, that are still not supported by all libraries. As indicated in previous works, the citation count and its subsumed metrics (e.g., self-citations, citations per year) represents an established metric and can be useful—if considered with care. Some other information can also be useful and assist quality assessments, in particular aiming to provide automation based on open data. We believe that further analysis and tooling to assess quality using publicly available information can have several benefits, for example:

  • Reduce the necessary time to analyze a large set of publications, as is necessary for literature analyses.

  • Improve the ranking of publications to not only rely on citations and, thus, identify potentially more relevant works.

  • Help researchers to find publications that are most suitable for their needs, for example, if they aim to explore a new research area.

  • Potentially extend the search capabilities of digital libraries to allow for more fine-grained searches.

To this end, we aim to extend our research to improve our understanding regarding the connection between the different information and quality of publications.

In the future, we plan to evaluate different metrics to interpret the quality of scientific research works. For the current paper, we mainly focused on highlighting the support for certain features of various commonly used digital libraries. The results will provide the basis for our research, allowing us to scope automated quality assessments. Furthermore, we aim to explore how well these information can support and guide systematic literature reviews, for which some approaches on automated quality assessment have been proposed [26].