Impacts on Practice

  • High heterogeneity in meta-analyses of pharmacy services is limiting the strength of their evidence.

  • Meta-analyses of pharmacy services report insufficient sensitivity and subgroup analyses and do not present prediction interval calculations to limit the effect of heterogeneity.

  • Overstated conclusions in pharmacy services meta-analyses are associated with poor methodological quality of evidence synthesis.

Introduction

Systematic reviews and meta-analyses represent a major advance at integrating available information, and they constitute the highest level of evidence in scientific literature. The number of systematic reviews published in recent years has been steadily increasing as a result of the growth in the number of primary studies, as well as the desire to utilise accruing evidence as early as possible to improve health care decisions [1, 2]. Systematic reviews and meta-analyses can enhance precision, provide robust estimates and answer questions for which single studies are underpowered [3]. However, the validity of the conclusions of systematic reviews depends on the quality of the individual studies and the methodological quality of the evidence synthesis process [2, 4].

Previous studies showed that an important number of suboptimal and conflicted systematic reviews and meta-analyses in biomedical area were published. They served mostly as easily produced publishable units or marketing tools [2, 3, 5,6,7]. Ioannidis et al. determined that one-in-three meta-analyses in health sciences are redundant or unnecessary, and approximately half of the remaining ones have serious methodological flaws. These deficiencies impair the replication and reliability of the results. Truly informative meta-analyses represent a small minority [1].

In last decades, the effect of pharmacy services on patient outcomes has been explored by means of systematic reviews and meta-analyses with conflicting conclusions [8]. Particularities of the interventions used in pharmacy services may be the origin of the heterogeneity identified in meta-analyses. Pharmacy services include complex interventions—with several components—frequently provided through educational, attitudinal or behavioural actions. Thus, standardising the dose of the intervention (i.e. the frequency in which the interventions are performed) and ensuring the fidelity of the service is much more complex than ensuring administrations of a certain dose of an active substance. Poor methodological quality of the evidence synthesis process is an easily modifiable limitation; it increases heterogeneity and compromises the conclusion of the meta-analysis [2].

Aim of the review

We aimed to map the characteristics of published meta-analyses of pharmacy services and their association with the study conclusions.

Methods

A systematic review of systematic reviews and meta-analysis (i.e. overview or umbrella review) was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement and the Cochrane recommendations [9, 10]. Two reviewers performed all the steps independently (screening of titles and abstracts, full-text appraisal, data extraction and quality assessment of the studies); discrepancies were discussed with a third reviewer.

Search and eligibility criteria

We searched for meta-analyses that reported the effect of clinical pharmacy services in PubMed (which includes MEDLINE and PubMed Central), Scopus and Web of Science without timeframe or language limits (last update in January 2019). The search strategies were designed to be highly sensitive and are available in Supplementary Material 1. We also manually searched the reference lists of the included studies.

Meta-analyses of interventional and observational studies that compared a service provided by pharmacists versus usual care or that provided by other health professional were included. Usual care was defined as patients who received the usual treatment in regular practice. Articles were excluded if they met any of the following criteria: (a) articles written in non-Roman characters; (b) systematic reviews without reporting meta-analysis; (c) outdated meta-analyses (we included only the most recent version of the meta-analysis to avoid duplicate results); (d) interventions or services provided by a multidisciplinary team without differentiating a role for the pharmacist.

Data extraction and quality assessment

A standardised form to extract data was used. It comprised: (a) the studies’ metadata and general characteristics, such as authors’ names, publication year, journal impact factor (using 2018 Journal Citation Reports), country, sample size (i.e., number of included trials and population), pharmacy service, type of studies included and patients' clinical conditions; (b) methods used in the systematic review, including databases, use of MeSH terms, description of full search strategies, performance of manual and grey literature searches, statement of using reporting guidelines (e.g., PRISMA statement, Cochrane recommendations and GRADE), previous publication of the protocol and the studies’ quality and publication bias assessment (i.e., Jadad Score, Cochrane Risk of Bias Tool, funnel plot, Egger’s or Begg’s test); (c) description of the statistical model used for the meta-analyses (i.e., random, fixed or both), meta-analysis statistical method (i.e., inverse-variance, Mantel Haenszel or Peto), additional analyses (i.e., subgroup, sensitivity or meta-regression analyses) and software used for calculations; (d) report of results (i.e., effect size measures of primary outcome and its results with lower, heterogeneity and p value); (e) conflict of interest and funding source declarations. To identify the primary outcome of the meta-analysis, the following sequential criteria were defined:

  1. 1.

    Outcome defined by the author as primary;

  2. 2.

    Outcome was evaluated through a meta-analysis (i.e. only one meta-analysis available);

  3. 3.

    Outcome prioritised by GRADE;

  4. 4.

    Outcome as a core outcome set;

  5. 5.

    Outcome described in title, objective or conclusion;

  6. 6.

    An “ECHO” (economic, clinical, humanistic) outcome (i.e. ECHO outcomes have greater consequence on patient’s healthcare when compared to isolated process outcomes);

  7. 7.

    Outcome with more studies in the meta-analysis;

  8. 8.

    Outcome first mentioned in the study.

The methodological quality of the included meta-analyses was evaluated using R-AMSTAR, in which each of the 11 questions should be assigned a score from 1 to 4. The summation of all scores represents the overall quality score of the systematic review [11].

Statistical analyses

To evaluate potential time trends, the median of the publication year distribution was used as a two-period cut-off. The normality of the variables was assessed using the Shapiro–Wilk test, with additional visual inspection of the Q–Q plot. For continuous variables with a non-normal distribution, results were reported as median and interquartile range (IQR), and the Mann–Whitney U test was used for group comparisons. Categorical variables were reported as absolute and relative frequencies, and the chi-square test was used to identify associations.

Results

A total of 1239 records were retrieved from the electronic databases. After the screening process, 151 articles were included for full-text analysis, of which 85 were considered for data extraction and analysis (Fig. 1). The list of included and excluded studies (with the reasons for exclusion) are available in Supplementary Material 2 and 3, respectively.

Fig. 1
figure 1

Flowchart of the literature selection process

Full size image

These 85 meta-analyses were published between 1989 and 2018, with a median year of 2016. Most studies were conducted in only one country (n = 49, 57.6%), especially the United States (n = 29), United Kingdom (n = 19) and Australia (n = 12). International collaboration among authors did not differ before and after 2016 (p = 0.118); it accounted for 42.3% of all included studies. The median number of authors (5, IQR 2 for the entire period) did not statistically differ before and after 2016 (5 [IQR 2] and 6 [IQR 3], respectively; p = 0.475).

The meta-analyses were published in 47 different journals, with a median impact factor of 3.23 (IQR 1.7) and 2.96 (IQR 4.5) for records published before and after 2016, respectively (p = 0.801). The most prevalent journals were Annals of Pharmacotherapy (n = 8, 9.4%), British Journal of Clinical Pharmacology (n = 7, 8.2%), Cochrane Database of Systematic Reviews (n = 6, 7.1%) and Journal of Clinical Pharmacy and Therapeutics (n = 6, 7.1%). Only 4 (8.5%) of these journals, which published 8 meta-analyses (9.4%), were considered to be a part of the pharmacy practice area in the map of pharmacy journals, namely Research in Social and Administrative Pharmacy (n = 4), American Journal of Health-System Pharmacy (n = 2), International Journal of Clinical Pharmacy (n = 1) and International Journal of Pharmacy Practice (n = 1) [12].

Each study included a median of 3 meta-analyses (range from 1 to 17). Additionally, despite the increase in pharmacy services publications over the years, there was no difference in the number of studies included in meta-analyses published before and after 2016 (p = 0.475), with an overall median value of 11 (IQR 10; range from 3 to 771). The most commonly used software was Review Manager (n = 28, 34%), followed by Comprehensive Meta-analyses (n = 16, 19%). The most frequently used statistical model was random (n = 75, 90.4%).

Pharmacists’ interventions evaluated by the meta-analyses were mainly focused on a specific medical condition, especially cardiovascular diseases (including diabetes), with 34 meta-analyses (73.9%). The majority of the meta-analyses did not identified the primary outcome (n = 57, 67%). The results of the algorithm used to establish a primary outcome, together with general characteristics of the included studies, are presented in Supplementary Material 4. Additionally, most of the primary outcomes reported in the meta-analyses were clinical outcomes (n = 32, 37.6%), followed by medication adherence (n = 13, 15.3%) and hospital admissions (n = 13, 15.3%).

The median number of databases used in the meta-analyses was 5 (IQR 4), in which PubMed/MEDLINE (98.8%), Scopus/Embase (78.8%) and Cochrane (including Cochrane Database of Systematic Reviews and Cochrane Central Trials Register; 72.9%) were the most prevalent. There was no difference between databases used before and after 2016, except for International Pharmaceutical Abstracts that presented an important decrease over the years (Table 1).

Table 1 Bibliographic databases used in the included meta-analyses
Full size table

Concerning the methodological quality of the meta-analysis, the overall median r-AMSTAR evaluation was 30 (IQR 8). This score represents 68.2% of the maximum possible score. Less than half of the meta-analyses (n = 47, 55.3%) reported the full search strategy. Most of the included studies performed the manual search (70, 82.4%), with a decreasing trend after 2016 (p = 0.049). A minority of meta-analyses searched for records in grey literature (35, 41.2%). A protocol registration for systematic reviews and meta-analyses was provided by 30 studies (41.2%), with a slight increasing trend over the years (p = 0.008; Table 2).

Table 2 Reporting characteristics of meta-analyses of pharmacy services
Full size table

Among the guidelines for conducting and reporting systematic reviews and meta-analyses, only the PRISMA statement was reported to be followed by 45 (60%) meta-analyses. There was a positive trend before and after 2016 (p = 0.022). Following Cochrane recommendations or using GRADE were substantially less reported (23 [27.4%] and 15 [19.2%], respectively), and their use did not show positive trend over the years (Supplementary Material 5, 6 and 7).

Heterogeneity and quality analyses were performed for more than 80% of the meta-analyses; there was no difference before and after 2016 (see Supplementary Material 8). However, when considering the studies in which the sensitivity analyses should be performed, due to their moderate-high heterogeneity, only 41 (55.4%) presented these analyses. There was no difference before and after 2016 (p = 0.322). The raw data used in the meta-analyses were made available in less than 10% of studies, with no difference between the publication periods. Other reporting of characteristics of meta-analyses are described in Table 2.

A total of 63 meta-analyses (74.1%) showed significant results in favour of pharmacists’ interventions based on the pooled effect size with a 95% confidence interval. However, only one study calculated the prediction interval [13].

Among the meta-analyses that did not provide the full search strategy, 89.5% favoured the pharmacists’ intervention (p = 0.004). From the 71 meta-analyses that did not report an analysis of the quality of evidence and the primary studies in their conclusion sections, 78.9% favoured the pharmacist’s interventions (p = 0.024). Meta-analyses that either did not express any doubt about potentially inconclusive results or comment about the necessity for more studies to support their conclusions most favoured the pharmacists’ intervention (p = 0.038). However, there was no statistical association as to whether or not GRADE was performed, or whether quality assessment was reported, with favouring pharmacists’ intervention (Table 3).

Table 3 Descriptive characteristics of the included studies by overall results of the meta-analyses
Full size table

Discussion

We identified an increasingly growing number of systematic reviews with meta-analysis that synthetize evidence about pharmacy services. They were mainly published after 2010, and most of them reported positive conclusions of the impact of these interventions. A similar growth in meta-analyses publication was observed in other biomedical areas [14, 15]. This increase can be partially explained by the scientific community’s awareness of the need for these filtered studies [14]. However, this general proliferation of meta-analyses has been associated with poor quality and duplicated results [16]. Ioannidis considered that only 3% of current meta-analyses are decent and clinically useful, 20% of them are flawed beyond repair and 27% are redundant and unnecessary [6]. Meta-analysis redundancy means that substantial numbers of these studies continue to be conducted on some topics without clear evidence for the additional value of the newer publications [5].

We also observed great dispersion in journals where the meta-analyses were published; less than 10% classifiable as pharmacy practice journals. Articles published in other scientific areas have editorial processes run by experts in those areas; these experts may have limited knowledge about the topic under discussion [12]. This situation means that editors and peer-reviewers may have “information about the topic plus some understanding, meaning and sense-making”, but they may have insufficient capacity to integrate it and adapt their knowledge to the context and situational shifts [17]. Consequently, publication scattering has become a common pattern in the pharmacy practice field [8, 18]. This phenomenon probably occurs because authors assign great importance to a journal’s impact factor, and this practice weakens pharmacy practice as a discipline [19].

The quality of meta-analyses about pharmacy services has been partially explored by previous studies. Melchiors et al. analysed a total of 151 systematic reviews and concluded that their quality varied from moderate to poor, especially due to weak conduct and reporting of methods [20]. Lipovec et al. performed an umbrella review to summarize systematic reviews and meta-analyses investigating the effects of pharmacist-led interventions and showed considerable heterogeneity between the studies. Additionally, authors found that the effect on health resources use was inconsistent, with a significant variability among study’s quality. However, authors included only interventions performed in transitions of care (n = 14 studies) and restricted their search to English papers. No correlation on the characteristics of the included studies and meta-analyses results were performed [21].

Similarly, MacLure et al. identified common pitfalls, including the lack of a detailed protocol and focus on the research question(s), bias and drift from the primary outcome. The authors noted that these situations may contribute to generation of systematic reviews that are not really systematic [22]. None of these studies performed a comprehensive analysis of meta-analyses. Similarly, studies in other areas have also described suboptimal reports, low reproducibility and lack of transparency [14, 23,24,25,26,27]. An analysis of 441 systematic reviews in biomedical literature showed that only four studies could be replicated, just one provided a full protocol and none made the raw data available [28].

Systematic searches using well-constructed search strategies, which ensure sufficient sensitivity to identify all the relevant literature on a topic, represent a critical step of the systematic review [29]. However, in our study only 55.3% of studies provided the complete search strategies. This factor hampers the reproducibility of the systematic review. As expected, PubMed/Medline and Scopus were the most commonly used bibliographic databases. The total number of databases used among the meta-analyses was surpsingly high, with a poor utilisation of manual and grey literature searches. Due to the well known overlapping coverage, the optimal number of databases that should be searched remains controversial [30]. PubMed and Scopus are the two most recommended for biomedical areas [31]. The best combination of databases should be established in field-specific studies, but manual and grey literaure searches are crucial in any scientific area.

Considering that clinical pharmacy services usually represent complex interventions, the poor reporting of the intervention components in primary studies, and the lack of standardisation of pharmacy services [8, 32, 33], the heterogeneity of the meta-analyses in this area are a major concern. This deficiency reduces the methodological quality and increases heterogeneity. Meta-analyses should recognise the influence of the different intervention components, their dose and the fidelity of the intervention in primary studies on the obtained heterogeneity [34]. Although there are several guidelines available to report the components of pharmacy interventions, their use is not generalised among primary studies researchers [28, 35, 36]. Given this current uncertainty, as well as the availability of a robust primary study quality assessment, using GRADE as a tool to evaluate certainty in the evidence should be mandatory. However, we identified very poor use of this instrument.

Several strategies should be consistently used, especially—but not only—in high heterogeneity meta-analyses. Sensitive analyses attempt to ascertain the potential influence of a few studies on the pooled effect size. Subgroup analyses evaluate the potentially different effect sizes obtained within the different sub-populations or among the various settings or interventions included in the overall meta-analysis [14]. In our analysis, only 55% of studies performed sensitivity analyses. In addition to these analyses, reporting prediction intervals (PIs) is particularly important in highly heterogeneous meta-analyses because PIs allow more informative inferences [37,38,39]. IntHout et al. demonstrated that in 20.3% of 479 meta-analyses, the PI showed that the effect could be completely opposite to the point estimate reported when using the 95% confidence interval. Thus, PIs should be routinely reported to confirm the study conclusion regardless of the obtained heterogeneity [39]. In our study only one meta-analysis reported the PIs [13].

Almost 85% of the meta-analyses in our study included a statement of competing interests, with the vast majority reporting the absence of any conflict. The pharmaceutical industry is commonly associated with major sources of potential conflicts of interest [40]. It is important to be aware of other origins of potential conflicts of interest [41]. Although implicit bias (also called unconscious bias) is commonly associated with negative evaluation of a person or group (e.g., gender or racial bias) [42], positively exaggerated and overstated evaluations might come from an implicit bias [43]. Being a member of a guild, which could be among the origins of an implicit bias when evaluating the role of those professionals, is rarely declared as a potential conflict of interest. In our study, 63% of the pooled estimates were favourable to the pharmacists’ intervention. We identified several significant associations between poor reporting or conduct indicators and positive conclusions about the impact of pharmacist’s intervention.

This systematic review demonstrates the importance of editors and peer-reviewers further increasing requirements beyond merely mentioning complying with PRISMA and Cochrane recommendations. Early (a priori) registration of the protocol in registry should be a mandatory requirement for publication. PROSPERO (https://www.crd.york.ac.uk/PROSPERO/) is one of the commonly used registries for health and social care systematic reviews, although registration is limited to systematic reviews that report health outcomes, a factor that may limit its interest. Other healthcare professionals have created field-specific registries like PEDro (https://www.pedro.org.au/). Reproducibility should also be a major objective for editors and peer-reviewers [44]. Providing as online supplementary material the complete search strategy for all the databases used and a spreadsheet with all the retrieved articles detailing the reason for exclusion should also be a requirement for the submission. Moreover, raw data used to compute the effect size measures should also be available to avoid errors in calculations [45] or effect size measures with unclear primary data [46]. Finally, meta-analyses should be conducted by trained unbiased personnel using only the highest quality standards, which should include sensitivity and subgroup analyses to explore the causes of heterogeneity. Furthermore, the prediction interval should always be reported to account for the heterogeneity in the pooled effect size measure.

Limitations

As with any systematic review, our study has some limitations. Search strategies may not retrieve all the relevant literature in the field, although we did not identify any additional meta-analysis in our manual search. We attempted to be language inclusive, but we had to exclude two meta-analyses (in Chinese) due to the limited language capacity of the research team. The final methodological quality of meta-analysis can be influenced by several variables. We evaluated different items that were considered as most important to be reported by authors of systematic reviews; however, further variables could be investigated.

Conclusion

Our map of characteristics of meta-analyses published about pharmacy services demonstrated several weaknesses related to poor reporting and poor conduction that was slightly associated with intervention efficacy overestimation. Weak conclusions were more frequent in studies that favoured the pharmacist’s interventions. Journal editors and peer-reviewers should consistently apply reporting guideline checklists and other conduct requirements to ensure the quality of the evidence published as meta-analyses.