Highlights

  • Reporting quality of randomized controlled trials (RCTs) can be improved by adhering to the CONSORT guidelines.

  • RCTs for new oral anticoagulants (NOACs) in venous thromboembolism (VTE) present moderate reporting quality. Randomization and blinding are primarily deficiently reported.

  • Higher impact factor, greater number of authors and existence of participant flow diagram may be associated with CONSORT compliance. Abstract and article reporting quality present strong, positive, linear correlation.

  • It is important that RCTs for NOACs in VTE present more transparent and complete reporting, which will enable readers to appraise the study, as well as investigators to perform similarly designed RCTs.

Introduction

Randomized controlled trials (RCTs) constitute the optimum study design for the assessment of new medicinal interventions [1, 2]. However, inappropriate methodological features may yield biased calculations [3]. Similarly selective reporting might lead to distorted conclusions [4]. Therefore, accurate and transparent reporting, especially with respect to methodology and presentation of the results, is considered of utmost importance [5].

The Enhancing the Quality of Transparency of Health Research (EQUATOR) Network is involved in the institution of recommendations for reporting in health research [6]. The CONsolidated Standards Of Reporting Trials (CONSORT) statement consists a tool for the improvement and assessment of the reporting quality of RCTs [7, 8]. It was initially introduced in 1996 [9] and underwent two revisions, in 2001 [10] and 2010 [11]. The revisions were accompanied by a detailed explanation and elaboration document [12, 13].

The CONSORT statement comprises of a 25-item checklist and a flow diagram [14]. It constitutes a guiding tool for authors, which prevents omitting crucial information, ensuring complete and transparent reporting. Therefore, an increasing number of journals endorse compliance with the CONSORT statement to improve reporting standards. Nevertheless, it is imperative to remember that quality of reporting does not always correlate with methodological quality [15].

Venous thromboembolism (VTE) includes deep vein thrombosis (DVT) and pulmonary embolism (PE). Patients suffering from or at risk of developing VTE require anticoagulation therapy. Cumulative evidence from RCTs is capitalized to establish the relative efficacy and safety of anticoagulants. Direct oral anticoagulants (DOACs) tend to become an increasingly popular choice over traditional therapy with low molecular weight heparin (LMWH) and vitamin K antagonists (VKAs) [16].

DOACs consist orally administered agents, prescribed alternatively to VKAs, whose target is either thrombin or factor Xa. They were, initially, introduced in 2008 and their use is indicated in cases of both arterial and venous thromboembolism [17]. Dabigatran, rivaroxaban, apixaban and edoxaban are the four major representatives of this category, all of which have been extensively studied in the prevention and treatment of VTE [18].

Despite pilling of new evidence, the reporting quality of performed RCTs has yet to be established. Groff et al. [19] evaluated reporting of outcomes in studies for thromboprophylaxis after total joint arthroplasty, investigating the effect of commercial funding. To the best of our knowledge the present study is the first to endeavour appraisal of the reporting quality of RCTs for novel oral anticoagulants (NOACs) in VTE disease based on the CONSORT statement.

Methods

The present study constitutes a retrospective evaluation of RCTs for NOACs in the prevention and treatment of VTE disease.

Search method

MEDLINE was meticulously searched, in order to identify all relevant RCTs from inception to April 02, 2019. The implemented search strategy is quoted:

(((((((((((((((((((((((((((((((NOAC) OR new oral anticoagulant) OR novel oral anticoagulant) OR NOACs) OR new oral anticoagulants) OR novel oral anticoagulants) OR dabigatran) OR pradaxa) OR prazaxa) OR pradax) OR rendix) OR bibr 1048) OR bibr-1048) OR rivaroxaban) OR xarelto) OR bay 597939) OR bay-597939) OR bay 59-7939) OR bay-59-7939) OR bay59 7939) OR apixaban) OR eliquis) OR bms 562247) OR bms-562247) OR edoxaban) OR lixiana) OR savaysa) OR du 176b) OR du-176b)) AND ((((((((((thromboprophylaxis) OR deep vein thrombosis[MeSH Terms]) OR vein thrombosis) OR DVT) OR venous thromboembolism[MeSH Terms]) OR venous thromboembolism) OR VTE) OR pulmonary embolism [MeSH Terms]) OR pulmonary embolism) OR PE))

No language restrictions were applied. References quoted by retrieved RCTs were manually searched.

Eligibility criteria

Studies that fulfilled the following criteria were considered eligible:

  1. (1)

    they were classified as RCTs -RCTs were defined as prospective studies with random assignment of their human population to two or more intervention groups-

  2. (2)

    they were published before April 02, 2019

  3. (3)

    at least one intervention group was randomized to one of the four NOACs –dabigatran, rivaroxaban, apixaban, edoxaban- regardless of the administration regimen and comparators

  4. (4)

    the population under research was people either at risk (prevention) or suffering from (treatment) VTE (DVT or PE) -evaluation of VTE disease as an outcome measure was considered an adequate, independent criterion for inclusion.

Studies were excluded according to the following criteria:

  • reports not published in English

  • conference abstracts

  • studies performed on animals

  • pilot trials

  • other study designs (e.g. retrospective study design, prospective not randomized design)

  • study protocols

  • retracted papers.

We screened all titles and abstracts retrieved, as well as full texts in case of inability to establish if a study met the inclusion criteria.

Data extraction

The 2010 revised CONSORT statement comprises of 25 items, 12 of which are divided into two parts (37 items in total). Introduction (Background and Objectives) and Discussion (Limitations, Generalisability, Interpretation) items was decided not to be evaluated in view of their subjective nature. Each of the remaining 32 items was appraised equally by 1 point when adequately reported, 0 when either inadequately reported or absent and as not applicable according to certain features of the studies. The modified CONSORT 2010 checklist is presented at Table 1. Items reported more than once were assessed by 0 in case of inconsistency. Reporting of an item in a different section of the trial (title, abstract, introduction, methods, results, and discussion) was appraised as 0, with the exception of the ‘other information’ items (Registration, Protocol, and Funding were assessed as reported regardless of the section where they were described) and the reporting of item 14a in the section methods (dates for recruitment and follow-up). Reporting of an item in the appendix of a study was assessed by 1 only in case there was a relevant reference in the text. According to the CONSORT explanation and elaboration document, item 8a was evaluated as reported only if it was included in the body of the main article and not as a separate supplementary file, where it can be missed by the reader.

Table 1 Modified CONSORT 2010 checklist of information to include when reporting a randomised trial
Full size table

Items 3b (changes to methods), 6b (changes to trial outcomes), 7b (interim analyses and stopping guidelines), 11b (description of the similarity of interventions), 12b (subgroup analyses and adjusted analyses), 14b (why the trial ended or was stopped), 18 (results of any other analyses performed) were not assessed in case of non-applicability. The proportion of adherence to the CONSORT statement was determined without taking not applicable items into consideration. Consequently, each study was rated against a different number of items.

Item 1b (Structured summary) was assessed separately based on the CONSORT for Abstracts extension, which comprises of 17 items. A 14-item version was deployed after the removal of the contact details item (specific to conference abstracts) as well as of items with regards to objective and conclusions, according to the modification applied to the CONSORT statement for full texts. Reported items inconsistent with the full text were assessed by 0. Item 1b was assessed by 1 when ≥ 7 of the 14 items were satisfied. The modified CONSORT checklist for abstract is presented at Table 2.

Table 2 Modified CONSORT checklist of items to include when reporting a randomized trial in a journal abstract
Full size table

Further information collected included publication year, journal ranking for the publication year (according to the Journal Impact Factor (IF) published each summer by Clarivate Analytics (Thomson Reuters) via Journal Citation Reports), number of authors, sample size, interventions assigned, population under research, country and centre design, reporting of commercial funding and the presentation of a participant flow diagram per randomized group (according to the CONSORT explanation and elaboration document).

Two authors (L.I., C.A.) individually assessed abstracts and full-texts of each RCT retrieved. In case of discrepancies, a decision was reached by consensus.

Objectives

The primary endpoint of the present study was to determine the mean adherence of RCTs for NOACs in VTE disease to the CONSORT statement. Statistic measures of central tendency and dispersion were used to describe CONSORT compliance. Secondary objectives were the calculation of compliance per CONSORT item and the investigation for probable determining factors with regards to the reporting quality of RCTs.

Statistical analysis

All statistical analyses were performed with SPSS Statistics Software Version 24. Cohen’s kappa (point estimate) was determined to appraise inter-rater agreement per CONSORT item. A kappa point estimate between 0.60 and 0.80 was considered indicative of substantial agreement, while a figure above 0.80 was appraised as an almost perfect agreement. Statistic measures of central tendency and dispersion were used to describe CONSORT compliance. Compliance above 70% was defined as adequate and below 70% as inadequate.

Univariate analysis for possible determinants was performed. Journal impact factor (IF) and number of authors were analyzed as continuous variables, using independent sample t-test. Normal distribution was assumed according to the central limit theorem. Publication year (up to 2010, after 2010-year of CONSORT revision-), country design (single country, multinational), sample size ( ≤ 1000, > 1000-arbitrary threshold-), presentation of participant flowchart according to the CONSORT statement and reporting of commercial funding were analyzed as nominal variables, using Pearson’s chi squared test or Fisher's exact test. A relaxed p-value of 0.20 was established as a threshold for the variables to enter the binary logistic regression. A rigor p-value of 0.05 was set to be significant for the logistic regression analysis. Odds ratios (ORs) and 95% confidence intervals (95% CIs) obtained from the logistic regression analysis are presented.

An exploratory analysis was performed to investigate the existence of linear correlation between abstract and article reporting quality. Pearson Correlation Coefficient (Pearson’s r) was determined for this purpose.

Results

Literature search yielded a total of 3633 studies (Fig. 1). After initial screening of titles and abstracts 3251 studies were excluded. Finally, 83 studies were included following assessment of the full text. The manual search did not provide any additional RCTs. Among retrieved studies 13 assessed Dabigatran, 44 evaluated Rivaroxaban, 9 appraised Apixaban and 15 Edoxaban. The rest two studies investigated more than one intervention (NOAC). Interventions were evaluated as thromboprophylaxis in 62 (74.70%) studies and as treatment for VTE in 21 (25.30%) studies. Thromboprophylaxis was mainly investigated in orthopaedic patients (49/62).

Fig. 1
figure 1

Flow chart of the literature search

Full size image

CONSORT compliance

The primary purpose was to establish the mean proportion of adherence to the CONSORT statement. The Mean CONSORT adherence was calculated 61.84% with SD = 18.72. The Median was 65.38%, while the minimum and maximum adherence 20% and 92.59% respectively (range 72.59%). Among the retrieved studies 35 (42.17%) presented an adequate reporting (above 70%), while 48 (57.83%) registered an inadequate reporting (below 70%).

Adherence per consort item was evaluated (Fig. 2; Table 3). Inter-rater agreement was rather satisfactory with Cohen’s Kappa above 0.6 for all 32 items, and above 0.8 (almost perfect) for 31 of them. Item 19 (all important or unintended effects in each group) was the only item presenting agreement below 0.8 (Cohen’s kappa = 0.75).

Fig. 2
figure 2

Graphical presentation of adherence per CONSORT item

Full size image
Table 3 Adherence per CONSORT item
Full size table

Items 1a and 1b (title and abstract correspondingly) were evaluated as reported in 31.33% and 54.22% of the articles respectively. Among methodological items, randomization and blinding were mainly underreported. Randomization process (items 8a and 8b) and allocation concealment (item 9) were described in less than half of the studies (49.40%, 48.19% and 49.40% respectively), whereas implementation (item 10) was assessed as reported in 0% of the studies. Although 80.72% of the articles indicated who was blinded (item 11a), solely 44% of them described the similarities of interventions. The rest of the methodological features were reported in more than 60% of the RCTs, with the exception of items 3a (trial design) and 12b (methods for additional analyses) (38.55% and 45.10% correspondingly).

Results were generally adequately reported (above 65%). Only items 17a (results for each group), 17b (results for binary outcomes) and 18 (results of any other analyses) were positively assessed in less than 50% of the studies (45.78%, 13.25 and 39.22 respectively). Other information (trial registration, trial protocol, sources of funding) were rated as reported in 63.86%, 26.51% and 83.13% of the studies (respectively).

Determinants of reporting quality

The association of several factors with the overall reporting quality was investigated. Results provided by Univariate analysis are presented at Table 4. Apart from the publication date (before and after the 2010 CONSORT revision, p = 0.824), every other analysis provided a statistically significant result. Specifically, higher IF, greater number of authors, sample size larger than 1000, multinational design, reporting of commercial funding, existence of participant flow diagram were all associated with superior reporting quality (p < 0.001 for each variable).

Table 4 Univariate analysis of possible determinants of reporting quality
Full size table

Results provided by Multivariate Logistic Regression are illustrated at Table 5 (graphical presentation at Fig. 3). All of the aforementioned factors were analyzed (with the exception of publication date). None of the parameters was associated significantly with adequate reporting. Journal IF [p = 0.055, OR = 1.347, 95% CI (0.994, 1.826)], number of authors [p = 0.076, OR = 1.277, 95% CI (0.975, 1.672)] and existence of participant flow diagram according to the CONSORT guidelines [p = 0.055, OR = 55.358, 95% CI (0.914, 3351.765)], failed shortly to demonstrate statistical significance. None of the rest of the factors analyzed came close to establishing significance of the results.

Table 5 Multivariate analysis of possible determinants of reporting quality
Full size table
Fig. 3
figure 3

Determinants of reporting quality. Reference line set to 1.00. Participant flowchart not depicted due to large 95% confidence interval (0.914, 3351.765)

Full size image

An exploratory analysis was performed to investigate for linear correlation between abstract and article reporting quality. Pearson’s r was estimated r = 0.851, p < 0.001, which is indicative of statistically significant, strong, positive correlation. The scatter plot diagram (Fig. 4) graphically demonstrates the correlation between abstract and article reporting quality.

Fig. 4
figure 4

Trend between abstract and article reporting quality. Y: article X: abstract

Full size image

Discussion

CONSORT compliance

The present study constitutes an effort to determine the reporting quality of RCTs for NOACs in the prevention and treatment of VTE, based on a modification of the 2010 CONSORT statement, after the removal of subjectively assessed items. We reviewed 83 articles and the overall adherence to the CONSORT statement was as moderate. Only 35 of the 83 papers registered a reporting quality above 70%, which was defined as the limit for adequate reporting, while inter-rater agreement (assessed by Cohen’s kappa) was almost perfect.

Reporting of principal methodological features with regards to randomization and blinding was suboptimal. Implementation of randomization presented the lowest reporting, with none of the studies providing a sufficient description (0%). The rest of the items from the randomization and blinding section were reported in less than half of the studies apart from item 11a (who was blinded). Reporting of the said items is argued to be of utmost importance for the assessment of a trial’s methodological quality. Nevertheless, deficient reporting of them appears to be a general affliction [20,21,22], with implementation of randomization constituting the major issue more commonly [23,24,25,26,27]. The rest of the items in section methods are more adequately reported, with the exception of item 3a (trial design), which was evaluated negatively (not reported) in 61.45% of the studies, due to the absence of allocation ratio description, and item 12b (methods for additional analyses), reported in 45.10% of the articles. A similar reporting pattern was observed by Liu et al. and Chen et al. [22, 28] with the former attributing the negative evaluation of trial design to deficient reporting of allocation ratio.

Items regarding the results section were sufficiently reported (above 65%), with the exception of items 17a (results for each group), 17b (binary outcomes) and 18 (result of any other analyses). Item 17a (45.78%) was rated negatively due to insufficient presentation of estimated effect sizes and their precisions for each primary and secondary outcome, an issue previously addressed [22, 27, 29]. Similar results with respect to reporting of study results were reproduced by Gnech et al. [21], Huang et al. [23] and Chen et al. [26].

Items with respect to trial registration (23), protocol (24) and funding (25) are arguably the most objectively assessed, along with item 1a (title). Item 24 registered a poor reporting quality (26.51%), while items 23 and 25 were more sufficiently reported (63.68% and 83.13% respectively). Study protocol, generally, appears to constitute the item more frequently underreported among these items, as demonstrated by Nagai et al. and Rikos et al. [30, 31].

Item 1a was reported in 31.33% of the studies and item 1b (abstract, separately assessed using the CONSORT for Abstracts extension), in 54.22% of the studies. The reporting quality of abstracts acquires increasing importance owing to the rapidly increasing number of publications. A great number of articles focus on the adherence of abstracts to the CONSORT guidelines, since abstracts are commonly utilized as filtration tool for acquisition of the full text [32,33,34,35]. Results are heterogeneous depending on the medical research field.

It is worth reminding that reporting quality is not identical with methodological quality [15]. Nevertheless, it is not of inferior importance. An optimal methodological quality ensures adequate robustness and reliability of the results [3]. On the other hand, sufficient reporting facilitates evaluation of the methodological quality. Quality assessment tools (e.g. the Risk of Bias Cochrane tool for Systematic Reviews of interventions [36]) are dependent on reporting in order for reviewers to clarify the risk of bias regarding the findings and sequentially grade the quality of evidence [37]. Furthermore, appropriate evidence synthesis from different studies requires sufficient reporting of methodological features and results. It is important to highlight that the performance of similarly conducted trials to examine the reproducibility and consistency of outcomes, as well as to pool obtained results, is not feasible in the absence of adequate reporting. Therefore, although, compliance with the CONSORT statement does not improve the quality of a trial, it allows more satisfactory evaluation of RCTs which enables more efficacious capitalization of data in the clinical setting.

Determinants of reporting quality

Although Univariate analysis indicated that higher IF, greater number of authors, sample size larger than 1000, multinational design, reporting of commercial funding, and existence of participant flow diagram were associated with superior reporting quality (p < 0.001 for each variable), Multivariate analysis provided significant results for none of the above factors.

Publication year was the sole factor that did not reach statistical significance in the Univariate analysis (p = 0.824). The relationship between date of publication and CONSORT compliance has been investigated before with certain studies providing results compatible with superior reporting following publication of the CONSORT guidelines [29, 38,39,40]. Journal IF failed shortly to reach statistical significance. IF was previously studied and a number of studies achieved to demonstrate a significant association between IF and reporting quality [29, 31, 39, 41, 42].

Number of authors [27, 43], sample size [22,23,24, 44] and funding (commercial or not) [22, 28] were previously investigated, but concluding results were not obtained. The present study demonstrated an almost significant effect for the number of authors and non significant effects for sample size, as well as, reporting of commercial funding. The study of Parish et al. [45] associated scientific collaboration (number of authors) with higher citation impact. This finding appears consistent with our results, which correlate (approaching significance) scientific collaboration with superior reporting quality.

Abstract reporting quality is considered of crucial importance, owing to the fact that most readers base their decision to acquire or not a full text on its abstract [46]. An exploratory analysis was carried out to investigate for linear relationship between proportion of adherence to abstract and article CONSORT guidelines. A statistically significant strong positive correlation was established (r = 0.851, p < 0.001). We have not identified any other article that endeavoured the mentioned correlation.

We were not able to identify published studies analyzing the effect of participant flow diagram and country design. Our study indicated a considerable trend towards significance for the existence of a participant flow chart, while country design does not appear to exert significant influence. Other parameters occasionally suggested as possible determinants are CONSORT endorsement [22, 24] and centre design [41]. In the present study CONSORT endorsement was not investigated in view of the fact that only the presently published instructions for authors could be evaluated. We did not attempt to analyze centre design because our initial intention was to investigate the effect of country design and the simultaneous insertion of centre design in the multivariate regression analysis would induce multicollinearity in our model.

Conclusions

To the best of our knowledge, the present study is the first to evaluate the reporting quality of RCTs for NOACs in the prevention and treatment of VTE disease. For this purpose, we deployed a modified version of the CONSORT tool, after the removal of subjectively assessed items. Additionally, we examined two poorly investigated possible determinants of reporting quality, the existence of a participant flow diagram and country design, with the former presenting almost statistically significant results. Furthermore, the present study appraised the correlation between abstract and article reporting quality, which was as yet insufficiently evaluated. Considering the increasing number of publications it is essential that the reporting quality of abstracts is satisfactory and corresponding to the reporting quality of articles. Two authors separately rated each study and a rather satisfactory inter-observer agreement was achieved.

Our study has certain limitations. To begin with, literature search was performed only in one database, PubMed. Articles not published in English were excluded and the researchers were not blinded to author and journal information. Moreover, all items were equally assessed by 0 or 1, while certain items are generally considered of superior importance than others.

The results we obtained were compatible with moderate adherence to the CONSORT statement. It is important that the reporting quality of RCTs for NOACs in VTE disease is improved, especially with respect to randomization and blinding. Transparent reporting will enable readers to critically appraise the procedural quality and interpret the results of published studies.