Keywords

1 Motivation

The number of scientific publications increases steadily every year. More and more research results are published, so that there is an exponential increase in publications [6]. Consequently, it is more and more time-consuming to inform oneself in detail about the current state of research of a subject area, such that the importance of systematic reviews grows.

A systematic review is prepared on the basis of already published research work and presents a current and detailed summary and evaluation of several research results of a certain scientific topic. In this respect, the review offers the reader a suitable opportunity to obtain further information in his or her field of work and to bring it up to date [22]. For the authors of a review, on the other hand, the already enormous amount of time and effort required for its preparation increases due to the large number of available scientific papers. Taking several further barriers for the creation process of a systematic review into account, the support of a tool especially designed for systematic reviews is very useful in this step [1]. For this purpose, some tools are already available. However, since different steps are required to create a review and the overall work is very extensive, it is difficult to support the entire creation process, such that existing tools have gaps and do not address all the requirements of a review [1].

Furthermore, deep insights might become evident by visualizing the data of systematic reviews in a proper way, but existing tools provide only basic and very limited support for automatically generating visualizations for systematic reviews. For example, the following two types of visualizations are not generated by existing tools at all: 1) a flow chart of the publication selection process presenting the flow of information during the different phases of the review and 2) the citation graph illustrating the relationships between the included publications of a review, thus giving the user a new perspective on the work used.

Our contributions are:

  • A tool for automatically generating flow charts of and citation graphs for systematic reviews.

  • Our developed software for the creation of the visualizations (i.e., flow chart as well as citation graph) can be downloaded at https://github.com/l-hartung/reviz and is freely available to users worldwide by using docker containers.Footnote 1

  • Different variants of citation graphs for simplifying the presentation by summarizing nodes and edges, and introducing factors like direct and indirect citations as well as coloring publications with common authors for further analysis.

  • An extensive evaluation of the different variants of the citation graphs.

In the following Sect. 2, we introduce the basics of systematic reviews. In addition, previous work on the support of reviews as well as various visualizations and their selection for our work are presented. Subsequently, Sect. 3 details the methods and strategies for the flow chart and the citation graphs with its various variants for simplification and adding information for the purpose of in-depth analysis. Afterwards in Sect. 4, we evaluate the created visualizations and Sect. 5 provides a summary and the future work.

2 Basics

In this section, some basic principles are presented in order to shed more light on systematic reviews and their preparation. In addition, some already existing tools supporting systematic reviews are considered in order to select one of these tools for the present work. Furthermore, different types of visualizations in reviews are introduced, including the flow chart of the publication selection process and the citation graph.

2.1 Systematic Reviews

Systematic reviews provide an overview of selected scientific papers on a research topic. Firstly, all available publications relevant to the research area are identified, evaluated and interpreted [16]. A central advantage of a systematic review over other scientific papers is its high informative value. While individual research papers are often based on the expectations of the scientists and results that do not correspond to the desired results can also be omitted from the publication, a systematic review is fundamentally more objective and very comprehensive. By summarizing a large number of existing research results on a topic, gaps, contradictions, relationships, or inconsistencies in the research can be identified, thus providing clues and directions for future research [16].

There are guidelines for the development of a systematic review in order to create uniform and comparable results. In the medical field these are for example the Cochrane Reviewer’s Handbook [27] and the CRD Guidelines for those Carrying out or Commissioning Reviews [14]. In [16], these guides have been adapted for the research area of software engineering. In principle, however, the procedure of a review is mostly identical in every area.

2.2 Related Work for Supporting Systematic Reviews

As in other publications, the quality of a systematic review can vary greatly and depends to a large extent on the approach and thoroughness of the authors and the scope and quality of the publications included. In order to ensure uniform standards, the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement was specified as a guideline for the report of systematic reviews [22]. Although the statement is designed for medical reviews, it can also serve as a basis for reviews from other areas. The PRISMA statement consists of a checklist of 27 points that should be included in the review report and a 4-phase flow chart (see Fig. 1).

The extremely high effort required for the preparation of systematic reviews leads to an increased need for automatic support during this process. Many software tools are already available to help authors in this context. These range from basic word processing programs, Reference Management Tools and statistical programs to specially designed tools, which are intended to support the entire systematic review process - or large parts of it - [21]. Reference Management Tools, such as RefWorks and EndNote, are widely used by review authors [21]. Such tools are available in large numbers, but only take up a very small portion of the work in a review. The Cochrane Collaboration also offers several tools to support the management and analysis of systematic reviews, including Covidence [9], EPPI-Reviewer [29] and RevMan [28]. These tools are designed specifically for Cochrane medical reviews. However, Eppi-Reviewer and RevMan can also be used for other types of reviews, although in this case some features cannot be used, and provide good support in some areas of the systematic review process [21].

In [1] six other widely-used tools are compared and evaluated from different perspectives. These provide support throughout the systematic review process and are not limited to a specific application area, although Al-Zubidy et al. refer to the software engineering area. The authors identify various barriers during the review process and requirements for supporting tools, and examine the six selected tools in relation to these. From these tools, only Parsifal [23] is free and open source software, and is additionally among the three tools with the best overall results in [1], such that our proposed tool extends it for generating the flow chart of the publication selection process and the citation graph.

There are other works that deal with the comparison of tools with respect to the requirements during the systematic review process, such as [20]. However, most of the tools either overlap with those already mentioned or support far less of the overall systematic review process. Accordingly, there are promising approaches to support systematic reviews; nevertheless, there are gaps in the existing tools, and optimal support cannot be provided.

2.3 Visualizations in Systematic Reviews

Visualizations in scientific works enable a better understanding of data sets, provide deeper insights or facilitate the analysis and presentation of large amounts of data [11]. Furthermore, in systematic reviews, different types of visualizations are used, for example to present information about the included publications in a clear way. Table 1 presents some types of visualizations frequently used in reviews and some sample reviews that include these visualizations.

Table 1. Various systematic literature reviews (SLR) and their applied visualizations. The first four reviews are from the medical area, the following four from the software engineering area and the last two from other areas.
Full size table

Flow charts for the selection of the included publications during the entire review process are included in almost all of the reviews examined. It is part of the PRISMA statement and is displayed in all systematic reviews in the form specified there or in a very similar manner. Some of the systematic reviews display tables with the most important characteristics of the included publications to provide an overview of them. In most cases, different tables are also shown to illustrate the different properties and characteristics of the respective systematic review. Diagrams showing the distribution of the publication years of the included publications are only used in the reviews examined in the area of software engineering. Other distribution diagrams like the publications sources or their geographical distribution, are also frequently found in this area. In medical reviews, on the other hand, forest plots are always presented to summarize and support their clues of the respective studies (e.g. relative risk or odds ratio). Funnel charts are also used here, for example, which can provide information on publication bias.

The most commonly used presentation is the flow chart of the publication selection process, which obviously forms a fundamental part of a review. Therefore it is implemented as one of the visualizations in this work to allow a quick and easy representation of the diagram in all systematic reviews.

Citation Graphs: None of the examined systematic reviews includes a citation graph. A citation graph is a directed graph in which publications are the nodes and citations are the edges. Thus, the relationships between the publications are represented by the citations themselves [24]. With the analysis of citations and citation graphs, knowledge flows and the spread of ideas and perceptions as well as the relevance of information sources can be examined [34]. This can be of importance during the preparation of a systematic review. By illustrating the relationships of included publications in a graph, for example, the spread of different methods, ideas or conceptions among the works can be made visible. Furthermore, knowledge can be gained about the relevance of individual works and about which works serve as a basis for further work. The fact that a citation graph does not appear in any of the examined systematic reviews is therefore not necessarily an indication of its insignificance for systematic reviews, but could rather be an expression of the complexity and efforts behind the creation of a citation graph. Therefore, a semi-automatic creation of citation graphs is a useful addition to the tool support of systematic reviews.

Visualization Tools for Scholarly Datasets: Please see Table 2 for tools for visualizing scholarly datasets. Existing tools are not primarily designed for systematic reviews and offer many types of visualizations not necessarily including citation graphs suitable for systematic reviews. Hence our proposed tool ReViz offers some important unique features: Integrated in the tool Parsifal for conducting systematic reviews, the citation graphs of ReViz are automatically constructed (with possibility of manual correction) from a set of paper documents. ReViz further supports various simplification approaches for citation graphs.

Table 2. Various visualization tools for scholarly datasets.
Full size table
Fig. 1.
figure 1

Example flow chart of the publication selection process of systematic reviews as generated by our proposed tool

Full size image

3 Flow Chart of the Publication Selection Process, Citation Graphs and Variants

We introduce our tool to generate the proposed visualizations (i.e., flow chart of the publication selection process and citation graphs in different variants) in this section.

3.1 Flow Chart of the Publication Selection Process

Our proposed visualization generator (integrated into Parsifal) generates a flow chart for the publication selection process (see Fig. 1), which is based on the structure of the template contained in the PRISMA statement [22]. In our study of several reviews on different topics, we observed differences in the presentation of the flow charts, but the content is always based on the PRISMA statement. In these flow charts, nodes present publications found in various digital libraries or other sources. Furthermore, nodes for the number of publications after duplicate elimination, as well as after removal of publications by inclusion and exclusion criteria and quality criteria should be included. Thus, each step in the publication selection process of the review is shown in a summarized form.

3.2 Structure of the Citation Graphs

Citation graphs in the context of systematic reviews should provide an insight into the relationships between the publications included in the review. The aim is to create a meaningful graph for the reader that provides as much information as possible. In order to achieve this goal, a further component is included here in our graphs: the arrangement of the publications should be based on their respective year of publication. This means that all publications which appeared in the same year are arranged next to each other, making additional information visible. For illustration purposes, a timeline is printed below the graph itself, so that it is possible to see exactly which publication appeared in which year. In addition, it can be quickly determined in which period of time the included research work is conducted and from which years more or less work originates.

The basis for the citation graph is therefore primarily the timeline. It covers the period of all publication years of the included works. Based on the timeline, the works are then drawn as nodes at the respective position on the timeline and the works linked by quotations are connected to each other with edges. Furthermore, different components within the graph should be visibly separated from each other. From a graph-theoretical point of view, it is possible that the citation graph is not only represented by a single directed graph, but consists of several independent subgraphs. In the context of this work, however, it is useful to consider the citation graph always as a single unit and thus as a graph with several components. Here, a component is defined as a subset of nodes and edges in which each node has at least one incoming or outgoing edge to another node of this subset. In addition, each node that has no edges is also its own component. To make the independence of the components visible, they should be drawn one below the other.

3.3 Node Summaries

Although various methods are used here to obtain an illustrative result, the large number of nodes and edges in a citation graph can still lead to very confusing results. An example is shown in Fig. 2a. In order to reduce this complexity, further methods to simplify the graph are necessary.

Fig. 2.
figure 2

Example of a) a citation graph (for the systematic review presented in [2]), and b) the same graph after merging several nodes with respective corrections of the edges

Full size image

One way to reduce overlaps of many edges and the resulting confusion is to reduce the number of nodes. A smaller number of nodes results in fewer edges and a less dense overall graph. If many publications are included in a review, the only way to reduce the number of nodes is to merge several individual nodes. In this respect, a summary of the original graph is generated. The difficulty in creating small graph summaries is the minimization of the resulting errors [19], so that no information is missing from the original graph and no wrong information is added.

There are various approaches to summarizing graphs, but they are not necessarily applicable to such relatively small graphs like our citation graphs. A merging of several nodes to a supernode with superedges is a well realizable possibility. Several nodes with the same or very similar incoming and outgoing edges are merged into one large node. In order to minimize errors and to reconstruct the original graph exactly, the merge consists not only of the graph itself, but also of a list of corrections of the edges.

This method can quickly become very inefficient for large graphs, since numerous comparisons of the nodes with each other must be processed to find those with similar edges. In [15] such an algorithm is presented, where first similar nodes are searched for to avoid unnecessary comparisons between all nodes. Then a summary of the graph is iteratively generated by merging original nodes or already existing supernodes.

In case of the citation graphs, only merges of nodes in the same level, i.e. publications with the same publication year, are reasonable. Because of this, supernodes with a large number of merges are extremely unlikely, so it makes more sense to focus on good merges of two or three nodes each. For this purpose, all possible candidates for the merging of two nodes are found first. Using a weight based on common and different edges for each pair of candidates, the best possible candidates are then selected. Based on these results, the calculation of connection components with three nodes from the candidate pairs is performed to determine good merges of three nodes.

We present the above described example in Fig. 2b as a summary after merging nodes: A total of 5 nodes and 37 edges are removed.

3.4 Summarizing Transitive Edges

If the publications in the graph are scattered over a longer period of time, there are inevitably fewer ways to merge nodes, since there are far fewer nodes in a plane. We present an example in Fig. 3.

Fig. 3.
figure 3

Example of a citation graph covering a longer time period (for the systematic review presented in [17])

Full size image

In this case, the clarity is impaired by many very long edges. To avoid this, more edges would have to be reduced, regardless of the number of nodes. Transitivities are suitable for this purpose. After the summary of transitive edges, some information is lost in the graph. For example, the number of incoming edges is no longer a clear indication of the number of citations of this node. In order to keep this information, we propose to increase the width of the remaining incoming edges of a node for omitted transitive edges (see Fig. 4). Thus, depending on the size of the incoming edges, the actual number of citations can be better inferred.

As a further variant and in order to have a metric for the influence of single publications to other research contributions, we propose to display exact numbers in the nodes of the citation graphs for (direct) citations as well as for indirect ones. Indirect citations represent a path of direct citations (see Fig. 4), i.e., A indirectly cites B if A cites B, or A cites C and C indirectly cites B, where A, B and C are publications. This allows a direct comparison between the citations of the individual nodes, despite the omitted edges. In order not to enlarge the nodes too much by the two additional numbers in the label, they must be displayed relatively small. In order to enable a quick comparison of the quotations of the individual nodes at first glance, these numbers are additionally highlighted in color. By means of a color scale, nodes with many citations can be quickly distinguished from those with fewer citations.

Additionally, it should be possible to identify publications with many common authors, for example to recognize follow-up publications and related approaches more easily. We hence propose to draw these publications in the same color (see Fig. 4).

Fig. 4.
figure 4

Example citation graph after removing transitive edges, summarizing nodes, displaying direct and indirect citations and publications colored for joint authors

Full size image

4 Evaluation

The evaluation of the visualizations introduced in this work mainly focuses on different aspects of the citation graph. The developed flow chart, which is largely based on the 4-phase flow chart of the PRISMA statement, offers little scope for variation and therefore takes up only a small part of the evaluation.

For the citation graph, on the other hand, several decisions are made regarding the layout and the used elements. However, the evaluation of the results is complex and can only partly be calculated mathematically.

Since the evaluation of the visualizations is partly subject to the subjective perceptions of the reader, part of the evaluation consists of a user survey. Thus, different aspects of the created variants of the citation graph as well as the flow chart can be evaluated. In addition to the survey, some calculations regarding the number of nodes and edges in the citation graph are carried out for evaluation.

4.1 Reduction of the Complexity of Citation Graphs

The number of nodes and edges can vary greatly in the introduced variants of citation graphs. Fewer nodes and edges reduce the complexity in the graph. We present in Table 3 the results for different calculations regarding nodes and edges for two graphs in different variants.

The upper part of the table contains the number of nodes, edges, and edges per node. Overall, there is a very strong reduction of the evaluated values. In the first graph, only about half of the edges of the original graph are drawn using both the node summary and the summary of transitive edges, while in the second example, there is even a reduction of 73% in total edges and 67% in edges per node.

In the lower part of the table, three additional parameters are considered that make it difficult to track edges: the number of nodes hiding one or more edges, the number of edges hidden by nodes, and the number of edge overlaps. Since these parameters are sometimes difficult to measure visually and could only be examined manually, some of the values are approximate values. Here, too, the results with node summary and summary of transitive edges are to a large extent highly reduced. In the case of edge overlaps, even 90% and 95% lower values can be achieved in both graphs when using both functionalities (i.e., node summary and removing transitive edges).

Table 3. Results for the number of nodes, edges and edges per node, as well as the number of nodes hiding edges, edges hidden by nodes and edge overlaps for two graphs (Graph 1 contains the publications of [17], Graph 2 contains the publications of [2]). In the first line the original – normal – graph is taken as starting point. This is followed by the graph with a summary of similar nodes, where an edge deviation of two has been allowed for nodes to be combined, and the graph with a summary of transitive edges. In the last line, both functionalities are combined.
Full size table

Long edges, which run over a longer span on the timeline and thus across several layers, add to the confusion, as they are more likely to cross more other edges and nodes, making it more difficult to quickly capture the course of all edges. Therefore, in Fig. 5, we present the lengths of edges occurring in the two graphs considered earlier, so that a comparison of the edge lengths in the graph variants is possible. In addition to the general reduction in the number of edges, which is already shown in Table 3, it is also apparent that many of the longer edges are eliminated by removing transitive edges.

Overall, very high reductions for the evaluated values occur in the calculations presented here, whenever node summaries and summaries of transitive edges are performed in the graphs. Thus, a lower complexity of the graphs can be concluded. In order to determine whether better final results for the graphs can be achieved as a consequence, the results of the user survey follow in the next section.

Fig. 5.
figure 5

Comparison of the edge lengths without any reduction, using node summary, removing transitive edges and both functionalities. The edge length on the x-axis indicates how many layers (i.e., years) an edge passes over.

Full size image

4.2 User Survey

In a user survey created for the evaluation of the visualizations, the different implemented variants of the citation graphs as well as the flow chart are assessed by external persons. This allows to determine how well the results are understandable and appealing to outsiders and whether the desired goals for the visualizations have been achieved. The graphs are primarily examined with regard to the points “clarity” (Are all information quickly and easily grasped at a glance?), “comprehensibility” (Are all necessary information available to understand the overall picture?) and “layout” (Is the result visually appealing?). Due to space limits, we only discuss the results of the user survey here without going into detailFootnote 2.

With 22 participants, the sample is relatively small and the answers of the participants varied from one another. Nevertheless, a clear tendency in the answers can already be determined.

In general, the clarity of the citation graphs, which contain more edges and/or nodes, is rated as relatively poor. As confirmed by the evaluation, the clarity is improved by the different functionalities. However, even after the improvement, a “good” result cannot necessarily be assumed. In order to be able to combine many nodes and thus save many nodes and edges, a high number of nodes within one year is necessary. In this case, however, the graph is likely to be very confusing, so that the result is likely to be complex even after the summary is performed. If, on the other hand, fewer nodes are within a year, but are scattered over a longer period of time, there are fewer possibilities to summarize, so that the clarity of the result will also not change much. However, if you use the summary of transitive edges, you often save a lot of edges, which certainly has a positive effect on the clarity of the graphs. Nevertheless, much of the information is obscured and the presentation of the omitted edges in the legend enables to trace the citations of a publication without gaps, but it is very time-consuming. Nevertheless, both functions are considered useful and are in any case capable of creating a new, clearer, presentation method for many citation graphs. The use of the flow chart created by our tool is also a useful addition to the creation of a systematic review.

5 Summary and Conclusions

We introduce a tool for the creation of visualizations for systematic reviews. In particular, we integrated the generation of a flow chart for the publication selection process of the systematic review and different variants of citation graphs (with and without merging nodes, removing transitive edges, adding numbers for direct and indirect citations and coloring publications with common authors) for the analysis of the citations of the publications among each other. The basis for these visualizations is the data from the tool Parsifal, which supports the creation process of a systematic review. We verify good results in an extensive evaluation by determining and comparing the number of hidden edges and hiding nodes in the citation graph variants and by a user survey for assessing subjective opinions of users.