Introduction

Anterior cruciate ligament (ACL) injury has attracted substantial interest of clinicians during the last decade, allowing it to occupy a spotlight position in the field of sports traumatology [1]. The fundamental role of the ACL as a stabilizer of the knee joint together with the posttraumatic consequences of its insufficiency fuelled this trend with increasing evidence [4, 8, 17].

The rapid growth of the field was accompanied by a rising number of questions with which clinicians were confronted, resulting in a corresponding increase in research activity [3, 5, 9, 16, 18–24]. Changes in treatment algorithms were experienced when initial historical techniques of primary suturing of the ACL were abandoned after proved a failure, allowing for a shift towards reconstructive techniques to restore functional anatomy [6, 7, 22]. A steep increase in the volume of published material related to ACL reconstruction was subsequently perceived. Between the years 1980 and 1999, 1328 articles containing the words “ACL reconstruction” or “Anterior cruciate ligament reconstruction” in their titles were published in the scientific literature. This number increased to 10,948 articles published between the years 2000 and 2015 (based on figures from the Institute for Scientific Information (ISI Web of Knowledge).

Careful selection of outcome measures is of fundamental importance for adequate demonstration of effects in clinical studies [10]. The process could become challenging when selection from a large pool of available measures is necessary. More than 54 scores have been designed alone for the ACL deficient knee and applied in the literature [14]. There is a lack of investigations on the most frequently applied scores and outcome measures in ACL studies with high impact in the field. Such studies would aid clinicians and researchers in selecting appropriate outcome measures for patient follow-up and monitoring.

Prompted by the points mentioned above, the aim of this study was to (1) Identify and evaluate the 50 most cited level I randomized trials in the field of ACL research. (2) Determine the outcome measures reported by these studies. (3) Identify factors influencing citation metrics.

Materials and methods

The annual Journal Citation Report ® of the Institute for Scientific Information (ISI) was utilized for identification of journals under the subject categories “Orthopaedics”, “Sports Sciences”, “Radiology” and “General medicine”. Two hundred and eighty-nine journals (all 72 orthopaedic journals, all 125 radiology journals, all 81 journals of sport sciences and 11 of the general medical journals with the highest impact factors) were identified and selected for further manual allocation of highly cited trials in the field of ACL research. The search was performed on the 4th and 5th of January 2016.

Allocation of articles and eligibility criteria

The ISI database comprising “MEDLINE”, “Web of Science core collection”, “BIOSIS previews” and “SciELO Citation Index” was employed for the generation of article lists within the selected journal categories regardless of language, which comprised the first step of the search process. The second step was based on article pre-selection, intended to identify randomized trials. The following keywords were employed: “random”, “randomly”, “randomized”, “randomised”, “randomizing” and “randomising”. Articles cited less than 50 times were excluded, given the assumption that the 50th article to be included in the analysis would posses more than 50 citations. In the third step of the search process, potentially relevant titles and abstracts were manually screened to exclude studies with obvious irrelevance to the ACL or incompatibility of study design (Fig. 1).

Fig. 1
figure 1

Flow chart showing the process of allocation of relevant studies. *: “random”, “randomly”, “randomized”, “randomised”, “randomizing”, “randomising”

Full size image

Finally, the full texts of remaining articles were evaluated for eligibility by two independent epidemiologically trained investigators based on the following inclusion criteria:

  1. 1.

    The primary research question was set to address an injury or pathology of the ACL.

  2. 2.

    The study design complies with the definition criteria of a randomized trial based on the Oxford Centre for Evidence Based Medicine (CEBM) definition [12].

  3. 3.

    The study had a lost to follow-up rate of no more than 20 %, and a narrow confidence interval, to assure quality standards of a level I study.

Exclusion criteria included:

  1. 1.

    Articles dealing with multiple ligament injuries or with primary focus on meniscal pathology.

  2. 2.

    Non-clinical studies regardless of type.

  3. 3.

    Lower-quality randomized trials that would not comply with the standards of a level I study.

  4. 4.

    Systematic reviews and meta-analyses.

Data extraction

The 50 most cited eligible articles were subject for extraction of the following data: year of publication, geographic origin, number of citations, current citation rate (citations in the year 2014), citation density (citations/article age), authorship and outcome measures. The articles were further evaluated for quality using the Jadad scale [13] and the Cochrane risk of bias tool (addressing 7 domains namely sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting and “other issues”) [11]. The area of research and the conclusion of each study were noted. Disagreements between investigators were solved by consensus with the involvement of a third investigator.

Statistical analysis

Normally distributed data were presented as mean ± standard deviation. Analysis of variance (ANOVA) was performed for comparison between means. Univariate analysis was conducted to identify factors influencing article citation metrics, and multiple linear models subsequently performed for adjustments. The Mann–Kendall trend test was applied for time-dependent trends. Qualitative evaluation of intervention superiority was undertaken, based on the outcome of each included trial when homogeneity was present. Cohen’s Kappa coefficient was applied for inter-rater agreement. A p value of <0.05 was considered statistically significant.

Results

All 50 articles were published in eight orthopaedic journals, one sports sciences journal and one general medical journal (Tables 1, 2), between the years 1987 and 2012 (mean 2002 ± 6). They originated from 16 countries, mostly the USA and Sweden (nine articles each), followed by Japan (seven articles) (Table 2). Citations per article ranged from 56 to 223 (mean 122.2 ± 54.7). There were 9 authors each represented by two trials in the list. Ten of the 50 studies (20 %) were collaborative.

Table 1 List of the highly cited trials in ACL research
Full size table
Table 2 Journals and geographic origin of the selected highly cited trials
Full size table

Quality assessment

The mean Jadad score for all articles was 2.2 ± 0.9 (range 1–5). The risk of bias was lowest amongst studies for the domain “completion of outcome data” (44/50). On the other hand, a high risk of bias was most common for the domain “selective reporting” (19/50). Insufficient reporting was found for the domain “blinding of participants and personnel” (Fig. 2).

Fig. 2
figure 2

Cochrane risk of bias assessment of all studies

Full size image

Inter-rater agreement

The inter-rater agreement for inclusion of articles was high (κ 0.91, confidence interval (CI) 0.84–0.98). The agreement on the items of the Cochrane risk of bias tool was highest for the domain “sequence generation” (κ 0.77, CI 0.61–0.83), followed by “allocation concealment” (κ 0.58, CI 0.40–0.76). Lower inter-rater agreement was present for the domains “incomplete data” (κ 0.37, CI 0.29–0.46), blinding (κ 0.27, CI 0.18–0.37), “other sources of bias” (κ 0.34, CI 0.23–0.45), “overall risk of bias” (κ 0.30, CI 0.16–0.44) and “selective reporting” (κ 0.09, CI −0.07–0.22).

Citation metrics

Three factors showed to influence total citation count, namely article age, citation density and a North American origin of the article. The rate at which a trial was currently being cited showed to be influenced by the overall citation density of an article and the inclusion of the Knee Injury and Osteoarthritis Outcome Score (KOOS) as an outcome measure. Citation density itself was shown to be influenced by the inclusion of the pivot-shift test as an outcome measure. Table 3 illustrates the results of the multiple linear regression models. The Jadad quality score did not show to have an influence on citation count, citation density or current citation rate (p = n.s). There was no significant correlation between increasing article age and citation count (p = n.s). However, there was a significant negative correlation between current citation rates, citation density and age (r = −0.43, p = 0.002 and r = −0.60, p < 0.001, respectively) (Fig. 3).

Table 3 Multiple linear regression model demonstrating factors influencing citation metrics after adjustment
Full size table
Fig. 3
figure 3

Graph representing the correlation between age (x-axis), citation density (y-axis blue) and current citation rate (y-axis res). R: correlation coefficient

Full size image

Outcome measures

There were ten frequently reported outcome parameters, each constituting an outcome measure in more than 2 of the 50 highly cited trials, namely KT-1000 measures of anterior translation (n = 38), IKDC subjective score (n = 28), pivot shift (n = 21), Lysholm score (n = 21), reporting of graft failure (n = 17), range of motion (ROM) n = 17, Tegner score (n = 15), IKDC objective (n = 6), Cincinnati score (n = 6) and KOOS score (n = 6) (Fig. 4).

Fig. 4
figure 4

Radar chart illustrating the frequency of reports per outcome measure on each of the axis

Full size image

Areas of research

The 50 most cited trials covered six core research topics, mainly graft selection (n = 18), single- versus double-bundle graft utility (n = 10), rehabilitation (n = 9), surgical technique (n = 8), indication for surgery (n = 4) and injury prevention (n = 1).

Of the 18 articles related to graft selection, 16 dealt with the primary question: bone-patella tendon-bone (BPTB) versus hamstrings grafts. Qualitative illustration of the results is shown in Fig. 5. A general similarity in outcome between both graft types, however, the studies showed a tendency towards favouring hamstring graft reconstruction with regard to donor site morbidity. Laxity and pivot-shift outcome results were favoured in BPTB graft groups in 3/16 and 3/8 studies, respectively.

Fig. 5
figure 5

Graph presenting a qualitative illustration of the outcome of the 16 studies comparing bone-patella tendon-bone (BPTB) to hamstring grafts. The x-axis represents the outcome measures. The upper portion of the y-axis is BPTB—graft favouring, the lower portion hamstring-graft favouring. The size of the diamonds demonstrates the number of papers (also in numbers)

Full size image

Ten articles dealt with the question double-bundle versus single-bundle techniques. More than half the articles reporting on laxity (4/8 studies), pivot shift (6/8 studies) and graft failure (3/5 studies) favoured the double-bundle technique (Fig. 6).

Fig. 6
figure 6

Graph presenting a qualitative illustration of the outcome of the 10 studies comparing single-bundle and double-bundle techniques. The x-axis represents the outcome measures. The upper portion of the y-axis is double-bundle favouring, the lower portion single-bundle favouring. The size of the diamonds demonstrates the number of papers (also in numbers)

Full size image

Four trials dealt with the question of surgical versus non-surgical treatment of an ACL rupture, all originating from Sweden. The first article by Sandberg et al. in 1987 demonstrated the only difference of an increased abundance of a positive pivot-shift test in conservatively treated ACL ruptures, compared to ACL sutured knees. The second article was published by Andersson et al. in 1989 comparing three groups undergoing different treatment strategies (1) repair of all major injuries including suture and augmentation of the ACL using a strip of the iliotibial band, (2) ACL repair without augmentation and (3) repair of all major injuries except the ACL. The results favoured the group receiving augmented ACL repair in terms of stability of the knees and the need for subsequent meniscal repair. The third trial was published by Meunier et al. in 2007 proving the significant benefit of ACL repair in reducing secondary meniscal lesions. The fourth article by Frobell et al. published in 2010 showed that 40 % of patients receiving initial conservative treatment required ACL reconstruction within 2 years and demonstrated a higher need for subsequent meniscal surgery in the primary rehabilitation group compared to an early ACL reconstruction group.

Discussion

The most important finding of this study was that the inclusion of the pivot-shift test and the KOOS score in an ACL trial would influence acquisition of citations

It is apparent that there is a change in citation practice, as it was shown in this study that the rate at which an ACL trial is currently being cited is determined by factors differing from those influencing the total citation count (Table 3); a North American origin of an article no longer shows an influence on its likelihood of being cited.

The Jadad quality score of studies did not show to influence any of the citation metrics

The outcome measure that is frequently being reported in more recent trials is the pivot-shift test. This may be seen as a corresponding result to the increase in clinical and academic focus on restoring functional anatomy and rotational stability. There are, however, associated challenges due to the variety of described testing techniques and lack of consensus regarding the ideal testing manoeuvre [15]. This highlights the need for establishing consensus over a reproducible standardized testing manoeuvre that can ideally be quantitatively assessed. The results of this study do show that reporting of pivot shift is likely to influence acquisition of citations, emphasizing the need to include this outcome measure in all clinical studies.

It is to be affirmed that despite age being a factor influencing total citation count, increasing article age does not guarantee citations, as citation density and rates were shown to drop as the article ages. This underlines the fact that it is the quality of an article based on multiple factors, which determines its likelihood of being cited in the future.

The phenomenon of acquiring high citation counts is considered the natural consequence of the decision of researchers to cite one article [2]. The underlying rationale for this phenomenon was postulated in explanatory models by scientometric researchers based on both quality and visibility dynamics of citation practices [2]. Therefore, a high citation count represents a reflection of at least a methodological agreement amongst peers.

This in turns highlights the importance of the frequently reported outcome measures in highly cited ACL trials. The primary implication of these findings is on the planning of clinical study designs. Based on the results of this study, the inclusion of the following outcome measurers in ACL outcome studies can be recommended: (1) The use of KT-1000 for measurement of anterior translation. (2) Reporting of pivot-shift test results. (3) Inclusion of the subjective IKDC, Lysholm, Tegner and KOOS scores. (4) ROM and (5) Reporting of graft failure (Fig. 4). Both the pivot-shift test and KOOS score are likely to positively influence acquisition of citations. This recommendation should be used as a guide for planning ACL outcome studies of all types, since all therapeutic clinical studies, regardless of study design, ranging from level IV to level I evidence, mandate sufficient selection of outcome measures.

It is fair to mention that results deduced from the included studies are biased towards conclusions from highly cited articles, thereby accounting for a limitation of this type of analysis as a meta-study. However, the primary aim of this study was not to provide a meta-analysis, but to reflect on the conclusions of highly cited impacting studies dealing with the most common questions in the field of ACL research. Due to the dependence of the primary research question on citation metrics, the ISI Web of Knowledge database was utilized solely, being the only source for retrieving accurate citation information. Despite being comprehensive, the risk of missing articles cannot be excluded, therefore accounting for a further limitation of the study. It is also necessary to point out that recent qualitative trials that have not yet gathered sufficient citations to enter the list were not included in the study.

The results of this study should provide an aid for clinicians and researchers when choosing appropriate outcome measures to monitor patients postoperatively after an ACL intervention.

Conclusion

The vast increase in ACL research during the last decade led to the establishment of many measures used for outcome evaluation in clinical research. Based on the analysis of highly cited level I research, it can be recommended that an ACL outcome study should include KT-1000 measurements of anterior translation, should report on pivot shift, and include the subjective IKDC, Lysholm, Tegner and KOOS scores, ROM and report on graft failure. The inclusion of the pivot-shift test and KOOS scores is likely to influence whether or not an article will be cited in the future.