FormalPara Key Points

The method developed in this study—the Competition Index (ComIn)—relies on the proportion of reports of adverse events (AE) that mention the drug class at an adequate level of drug grouping for identifying competitors.

From a sample of five AEs from the French spontaneous reporting database, the ComIn could greatly minimize the competition bias in signal of disproportionate reporting (SDR) detection, notably when considering potential competitors at the pharmacological drug class level.

The performance for SDR unmasking of ComIn was similar to that of existing methods when applied to the same dataset.

1 Introduction

Signal of disproportionate reporting (SDR) is widely used for the identification of drug safety issues from pharmacovigilance databases [1]; however, it can be affected by several biases, mostly related to underreporting and also to selective reporting (e.g. due to media coverage of a safety alert [2]) which can lead to competition bias or the masking effect [3–5]. The theoretical principle of this bias relies on the assumption that drugs or drug classes strongly associated with an adverse event (AE) will raise the baseline reporting rate of that AE in a spontaneous reporting database. When using this baseline reporting rate or a related indicator as the reference for SDR detection, this increase relating to existing associations could reduce the ability to detect safety signals relating to other drugs associated with the AE in question [5, 6].

To date, two automated methods have been developed for minimizing the competition bias in SDR detection, both of which are based on a two-steps process performed before the SDR detection: (1) identification of potential drug competitors via an index of competition/masking—the masking factor (MF) [7] and the masking ratio (MR) [8]—and (2) removal of reports related to these competitors from the database. These methods have both focused on the strength of the disproportionality for identifying potential competitors of the AEs of interest. However, as noticed by Maignen et al. [8], the proportion of reports regarding the AE of interest which mention the drug or drug class could be sufficient for identifying the potential competitors. In addition, they have only been tested in relation to potential competitors at the drug level, yet competition bias can also be caused by drugs sharing common properties (drug class effect) [9, 10]. Although they achieve some SDR unmasking [7, 11] it is likely that they fail to reveal SDRs that would be masked by the drug class effect.

The objective of this pilot study was to develop a new method that relies on identifying competitors by considering the proportion of reports of AEs that mention the drug class at an adequate level of drug grouping to increase sensitivity (Se) for SDR unmasking, and to compare its performance with that of MF and MR.

2 Methods

2.1 Data Source

The French spontaneous reporting database includes all AEs reported to the 31 French regional pharmacovigilance centers by health professionals during the period, but not those directly reported to manufacturers. Each report is reviewed and assessed by experts in pharmacovigilance before being recorded into the database. Drugs are coded according to the Anatomical Therapeutic Chemical (ATC) classification, and AEs are coded according to the Medical Dictionary For Regulatory Activities (MedDRA®) classification (version 13.1) [12].

All reports of AEs recorded in the French spontaneous reporting database between 1 January 2000 and 31 December 2005 were selected, with the exception of those mentioning only vaccines, herbal medicines, homeopathy, oligotherapy, food supplements, illicit drugs, cosmetics, or organophosphates as the suspected products [12].

2.2 Adverse Events of Interest and Related Case Identification

This study focused on five serious AEs that are considered important enough to monitor in pharmacovigilance: acute myocardial infarction (AMI), acute pancreatitis (AP), aplastic anemia (AA), convulsions (CO), and gastrointestinal bleeding (GIB) [13]. These were selected as (1) they were serious; (2) they related to different systems or organs, or involved different mechanisms; and (3) the reports relating to cases were retrievable from standardized MedDRA® queries (SMQs; version 15.0), used according to the broad version and the algorithmic approach.

2.3 Identification of Potential Competitors

2.3.1 Competition Index

The new method developed in this study—the Competition Index (ComIn)—measures the ability of drug D to compete with other drugs for an AE of interest by computing the proportion of reports relating to this AE that mention drug D; it considers drug D as a potential competitor when this proportion is higher than a given threshold. The ComIn is expressed in percentages and is calculated using the following formula (based on Table 1, MF):

$$ {\text{ComIn}} = \left( {\frac{a}{a + c}} \right) \; \times \; 100 $$
Table 1 Contingency tables associated with the computation of Competition Index and masking factor or masking ratio
Full size table

To determine the best configuration of ComIn for SDR unmasking, combinations of several values of the two following parameters were tested: (1) the competitor drug definition according to the ATC classification (ATC levels 2, 3, 4, and 5); and (2) the value of the ComIn threshold (from 5 to 20 % in increments of 1 %).

2.3.2 Masking Factor

MF looks at the expected rate of reporting of the AE of interest, which represents the denominator in disproportionality analyses. Precisely, MF measures the ability of a given drug D to compete with other drugs for an AE of interest by computing the ratio of this expected rate of reporting when all the reports relating to drug D are present or removed from the database, and for which a constant of shrinkage κ is applied [7]. A ratio lower than 1 means that drug D can compete with the other drugs for the AE of interest. In this study, the MF formula was adapted from Juhlin et al. [7] to express (as a percentage) the ability to compete (based on Table 1, MF):

$$ {\text{MF}} = \left( {1 {-}\frac{c + \kappa }{{\frac{(a + c)(c + d)}{(a + b + c + d)} + \kappa }}} \right) \times 100 $$

To determine the best configuration of MF for SDR unmasking, combinations of several values of the three following parameters were tested: (1) the competitor drug definition according to the ATC classification (ATC levels 2, 3, 4, and 5); (2) the value of the constant of shrinkage κ (50, 70, 90, and 100); and (3) the value of the MF threshold (5, 10, 15, 20, 25, and 30 %).

2.3.3 Masking Ratio

MR measures the ability of drug D to compete with another drug X for an AE of interest by computing the ratio of the values of the metrics of the disproportionality analysis performed for drug X, before and after removing all reports related to drug D from the database. A ratio higher than 1 means that drug D can compete with drug X for the AE of interest [8]. In this study, the MR formula was adapted from Maignen et al. [8] to express (as a percentage) the ability to compete [based on Table 1, MR; e.g. with the proportional reporting ratio (PRR)]:

$$ {\text{MR }}_{\text{PRR}} = \left( {\left( {\frac{(c + d)}{{(a_{D} + b + c + d)}}\left( {1 + \frac{{a_{D} }}{c}} \right)} \right){-} 1} \right) \; \times \;100 $$

To determine the best configuration of MR for SDR unmasking, combinations of several values of the two following parameters were tested: (1) the competitor drug definition according to the ATC classification (ATC levels 2, 3, 4, and 5); and (2) the value of the MR threshold (from 10 to 100 % in increments of 10 %).

2.4 Signal of Disproportionate Reporting Detection (SDR) Procedures

SDR detection considering AEs of interest, as identified using SMQs and drug exposure only at ATC level 5, was performed using the PRR; drug/AE pairs were considered as SDRs when (1) PRR was >2; (2) n > 3; and (3) χ 2 > 4 [6]. For each AE of interest, SDR detection was first performed in the complete database; to correct for competition bias, it was then performed after removing all reports mentioning the potential competitor drugs of that AE (ComIn, MR [8]) or only reports relating to potential competitor drug/AE of interest pairs (MF [7]). Unmasked SDRs corresponded to those identified only after removing reports.

2.5 Performance for SDR Unmasking

All unmasked SDRs were included in a reference dataset serving as the gold standard for computing ComIn, MF, and MR unmasking performances. Each SDR was classified as validated if the AE was mentioned in the corresponding Summary of Product Characteristics (SPC), and non-validated otherwise. The last available European SPCs [14] (as of February 2014 for drugs still marketed, and last published SPCs for withdrawn drugs), or French SPCs [15] when no European SPCs existed, were used for this classification. It was assumed that the interval between data entry and SPC consultation (up to 9 years) was sufficient to have all true AEs identified in the SPCs. Classification was carried out by two experts (FS, AP) blinded to (1) the method used for SDR unmasking (ComIn, MF, or MR); and (2) the opinion of the second expert. In cases of disagreement, final assessment of the SDRs was obtained by consensus.

Area under the curve (AUC) for binary diagnostic tests [16] and corresponding 95 % confidence intervals (CIs) [17] were used for estimating the performance for SDR unmasking. The AUC relies on Se and specificity (Sp). Se corresponded to the proportion of validated SDRs that were detected after competition correction, while Sp corresponded to the proportion of non-validated SDRs not detected after competition correction. To demonstrate the ability to differentiate between validated and non-validated SDRs, the method for competition correction needs to obtain an AUC higher than 50 %, with an AUC of 100 % representing the perfect ability to discriminate between these two types of SDRs.

AUC, Se, and Sp, as well as corresponding 95 % CIs, were estimated for each combination associated with ComIn, MF, or MR; for equivalent values of AUC between two combinations, preference was given to the one with the highest Se value. The performance for SDR unmasking of ComIn was compared with that of MF and MR, when all three methods were used at the best combination. All analyses were performed using the R statistical software, version 3.0.3.

3 Results

3.1 Study Dataset

Among the 82,885 reports recorded in the French spontaneous reporting database that were eligible for the study, 5253 were related to one or several AEs of interest: 1499 reports related to GIB, 1058 to CO, 981 to AMI, 911 to AP, and 825 to AA. Initial SDR detection identified 274 SDRs for these five AEs of interest.

3.2 Identification of Potential Competitors

Application of the ComIn with the ATC level 2/ComIn threshold of 5 % combination identified 15 potential competitor drug classes for AP, nine for AA, eight for AMI, eight for GIB, and seven for CO. At the same ATC level at the ComIn threshold of 20 %, the number of potential competitor drug classes identified decreased for each AE studied: three for AA, three for CO, two for AMI, two for AP, and two for GIB. A similar reduction in function of the ComIn threshold was observed when applying the ComIn at the other ATC levels (Table 2). Similar trends were observed with the application of MF and MR, although they identified less potential competitors than ComIn at equivalent combinations. With regard to MF, the value of the constant of shrinkage did not influence the results for the identification of potential competitors [electronic supplementary material (ESM) Tables ESM1 and ESM2].

Table 2 Number of potential competitor drugs/drug classes of each adverse event, and related reports (n = 82,885), identified by ComIn when tested with some ATC classification level/ComIn threshold combinations
Full size table

3.3 SDR Unmasking

A total of 174 SDRs were unmasked after using the ComIn, the MF, and the MR, and this constituted the reference dataset (Table ESM3); ComIn unmasked 151 SDRs (including 26 SDRs by this method only), MF unmasked 142 SDRs (18), and MR unmasked 112 SDRs (2). Among these, 46 SDRs related to AP, 46 to AA, 34 to AMI, 25 to CO, and 23 to GIB. Classification considered 73 unmasked SDRs as validated and 101 as non-validated, according to the SPCs of the associated products.

Using the ComIn with the ATC level 2/ComIn threshold of 5 % combination unmasked 30 SDRs (including 14 as validated): 11 (6) for AMI, 8 (2) for AP, 5 (3) for CO, 4 (1) for GIB, and 2 (2) for AA. A greater number of SDRs (total and validated) were unmasked at higher ComIn thresholds. When testing the ComIn with ATC level 3 combined with ComIn thresholds of 5–12 %, the total number of SDRs unmasked was steady but the number of validated SDRs increased to reach the top at the ComIn threshold of 12 %; both these numbers fell when combining ATC level 3 with higher ComIn thresholds. For ATC levels 4 and 5, the higher the combined ComIn threshold, the lower the number of SDRs unmasked (total and validated) (Table 3). Similar trends were observed with the application of MF and MR, with the highest numbers of SDRs unmasked (total and validated) obtained when they were used at ATC level 2 and, to a lesser extent, ATC level 3 (Tables ESM4 and ESM5).

Table 3 Number of signals of disproportionate reporting detected after removing reports related to potential competitors identified by ComIn when tested with some ATC classification level/ComIn threshold combinations
Full size table

3.4 Performance for SDR Unmasking of ComIn, MF, and MR

ComIn provided the best performance for SDR unmasking when used with the ATC level 3/ComIn threshold of 12 % combination (AUC of 62 % [54 %; 70 %]) (Fig. 1; Table 4), and the lowest performance when used with the ATC level 4/ComIn thresholds of 12–20 % combinations, or with the ATC level 5/ComIn thresholds of 8–20 % combinations (AUC of 50–52 %) (Fig. 1). Similarly, MF and MR provided the best performance when used with the ATC level 2/shrinkage of 50–100/MF threshold of 15–20 % combinations (AUC of 57 % [49 %; 66 %]) and with the ATC level 2/MR threshold of 70 % combination (AUC of 58 % [50 %; 67 %]), respectively (Fig. 1; Table 4). Performance decreased when using MF or MR with combinations including another ATC level, in particular ATC level 5 (AUC of 50–52 %) (Fig. 1). However, performance was not statistically different regardless of the method and combination used (Tables ESM6–ESM8).

Fig. 1
figure 1

Performance for signal of disproportionate reporting unmasking of the ComIn, MF, and MR according to the different ATC classification levels used for identifying the competitors and to the AUC criterion. Dots correspond to values of AUC obtained when using the ComIn, MF, or MR with different ATC levels/threshold value combinations; plots for MF were obtained using a shrinkage value of 50. ComIn Competition Index, MF masking factor, MR masking ratio, ATC Anatomical Therapeutic Chemical, AUC area under the curve

Full size image
Table 4 Performance for signal of disproportionate reporting unmasking of the ComIn, MF, and MR when used at the combination according to AUC criterion
Full size table

When focusing on the Se for SDR unmasking, ComIn still provided the best performance with the ATC level 3/ComIn threshold of 12 % combination (Se of 52 % [45 %; 59 %]) and the lowest with the ATC level 5/ComIn thresholds of 12–20 % combinations (Se of 4 % [1 %; 7 %]) (Fig. 2). MF and MR provided the best Se for SDR unmasking when used with the ATC level 3/shrinkage of 50–70/MF threshold of 5 % combination (Se of 51 % [43 %; 58 %]) and the ATC level 3/MR threshold of 10 % combination (Se of 45 % [38 %; 53 %]), respectively. The lowest Se was obtained at ATC level 5, both for MF and MR, and at ATC level 4 with the highest value of MR threshold for MR (Fig. 2). Performance in terms of Se for SDR unmasking of ComIn, MF, and MR was significantly better when used at ATC level 3 than at ATC level 5 for all three methods, but was not different between the methods (Table 5).

Fig. 2
figure 2

Performance for signal of disproportionate reporting unmasking of the ComIn, MF, and MR according to the different ATC classification levels used for identifying the competitors and to the sensitivity criterion. Dots correspond to values of AUC obtained when using the ComIn, MF, or MR with different ATC levels/threshold value combinations; plots for MF were obtained using a shrinkage value of 50. ComIn Competition Index, MF masking factor, MR masking ratio, ATC Anatomical Therapeutic Chemical, AUC area under the curve

Full size image
Table 5 Sensitivity for signal of disproportionate reporting unmasking of the ComIn, MF, and MR when used at best combination with the ATC classification level 3 and ATC level 5 as definition for the competitors
Full size table

4 Discussion

This pilot study found that using the ComIn with a definition of competitors considering drugs according to ATC level 3 combined with a threshold value of 12 % for the proportion of reports of the event mentioning the competitors could greatly minimize competition bias in SDR detection. In this combination, ComIn provided similar performance for SDR unmasking to that of MF and MR when applied at the best configuration, which also considered competitors at ATC level 3.

Competition bias in SDR detection is mainly caused by drugs sharing common properties and, in a few cases, by individual drugs [9, 10]. In this study, ComIn, MF, and MR all three provided the highest performance in terms of Se for SDR unmasking at ATC level 3, and the lowest at ATC level 5. Importantly, this difference of performance between the two levels of definition of the potential competitors was significant, which is in agreement with the hypothesis that identifying potential competitors at the drug class level could provide better Se for SDR unmasking than considering them at the individual drug level. For instance, no antidepressant considered individually was identified as a potential competitor for SDR detection with regard to COs. Conversely, the antidepressants drug class constituted a competitor for COs; removing reports associated with this class contributed to unmasking of several SDRs.

The ComIn relies on a simple principle that considers the proportion of reports of the AE mentioning the drug class for identifying potential competitors, contrary to MF and MF, which rely on the strength of the disproportionality [7, 8]. Despite that, no difference of performance for SDR unmasking was found between the methods. Although point estimates were in favor of the ComIn (for both AUC and Se), and this pilot study is likely to suffer a lack of power due to the limited number of events considered, it is difficult to distinguish one method from another based only on the performance for SDR unmasking. Nevertheless, the ComIn presents some advantages over the two others methods. Its principle is simpler to understand, and its implementation is easier and requires far less computation time.

The ComIn was developed with regard to a pragmatic criterion for pharmacovigilance systems, i.e. the ability to unmask relevant SDRs efficiently. In this perspective, the aim of the ComIn is different to that of the MR, which is, first, to evaluate the masking potency of drugs and, second, to allow unmasking in SDR detection. In addition, in this study priority was given to Se instead of Sp in case of equal values of AUCs, as the ability to detect potential SDRs can be considered of prime interest for systems of pharmacovigilance. Nevertheless, Sp, even if less important, cannot be fully ruled out as it is likely methods with low Sp will detect a high number of false associations, examination of which can itself delay the identification of true signals. For this purpose, both the identification of potential competitors, as well as SDR detection, were performed at the report level. In addition, SDRs were considered according to the criteria of Evans et al. [6], and not to the lower limit of the 95 % CI [18], as they require a minimum of three cases that reduce the detection of spurious associations based on a very few number of reports. In this pilot study, the Sp for SDR unmasking obtained with the ComIn at the best combination appeared correct, with a value slightly higher than 70 %, and was similar to that of MF and MR.

This pilot study presents certain limitations. First, the number of AEs considered was limited, and to what extent the results would apply to a more general set of conditions is not known. Case retrieval was carried out using SMQ broad searches in order to yield Se; however, a recent study has demonstrated that such queries can vary in their ability to correctly identify AEs in the French spontaneous reporting database [19]. Three of the AEs selected in this study—AMI, GIB, and COs—are easily recognizable, which reportedly increases the reliability of coding and thus that of case retrieval [19]. The remaining two AEs—AP and AA—rely on biological values, which is reported to favor the presence of false positive and false negative cases [19]. Nevertheless, there is no reason to believe that they would be preferentially related to a particular drug, and thus this limitation is not likely to alter the results. Another aspect to consider is that the performance for SDR unmasking was only assessed when using the PRR for SDR detection, and thus cannot be fully presumed for the other methods of disproportionality analysis. However, it is likely to be similar for case/non-case and reporting odds ratio [20], for which computation is very close to that of PRR. Bayesian methods [21, 22] are based on a more complex computation and the Gamma Poisson Shrinker might provide better Se for SDR detection than PRR [23]; however, as they seem to be similarly affected by competition [7], the results should also be transposable to these methods. Another limitation could be the definition of the reference dataset used in this study; all unmasked SDRs were assessed using SPCs only and not by using other complementary sources of validation, such as a literature review as carried out by other authors [24]. This approach might have increased the number of unmasked SDRs considered as true positives, which would favor the Se for SDR unmasking.

5 Conclusion

According to this pilot study, the ComIn could minimize competition bias in signal detection from spontaneous reporting databases, notably when considering potential competitors of AEs at the pharmacological drug class. Further investigations on simulated data would be needed to precisely determine its optimal configuration and related performance when used at a larger scale, and whether it can be an interesting tool for minimizing competition bias in an automated SDR detection procedure.