FormalPara Key Points

Application of an appropriate signal-detection method and careful analysis of spontaneous reporting systems supports the monitoring of quality defects and can identify substandard medicines (SSMs).

Important challenges in the identification of SSMs include missing data from individual case safety reports (ICSRs) as well as a lack of samples of suspected SSMs for verification testing, the latter being a direct result of the research being conducted retrospectively.

1 Introduction

By law, both innovative and generic medicines must be manufactured in accordance with regulatory requirements [1, 2]. A detailed specification for the finished product is set down in the marketing authorization [3]. Substandard medicines (SSMs) that do not conform with the specification—and therefore may compromise patient safety because of defects in the quantity of the active substance—may occur with both proprietary and generic medicines [4]. The use of SSMs is a poorly researched public health concern worldwide [5, 6]. SSMs are not counterfeit, falsified, or fraudulent but are poor quality and represent a significant risk to patients. There is published evidence that the use of such medicines can result in treatment failure [7] or even death [8].

1.1 Challenges Underlying the Detection of Substandard Medicines (SSMs)

In total, 42 analytical technologies are available for identifying SSMs or falsified medicines, both devices for laboratory testing, such as the gold standard high-performance liquid chromatography (HPLC), and in-field testing devices such as Raman spectroscopy [9]. The disadvantages of many laboratory testing devices are that they require laboratory facilities and highly trained personnel and that costs for these devices range from $US50,000 to 300,000. These instruments are not appropriate for routine product quality assessment in many of the low- and middle-income countries most affected by SSMs [9]. Field devices are less expensive but also less sensitive. This study discusses an inexpensive and sustainable statistical detection method that can be applied in routine product quality assessments in all markets.

1.2 Spontaneous Reporting Systems

Spontaneous reporting systems represent the most common method of pharmacovigilance in the postmarketing phase. They help generate hypotheses that could result in regulatory warning letters or changes to safety labels [10]. Although it is generally not possible to establish absolute proof of failure to meet the authorized specification of a medicine from individual case safety reports (ICSRs) in VigiBase® alone, as it is not possible to retrieve samples for confirmatory analysis testing, this data source can support the identification of hypotheses about potential poor-quality medicines associated with adverse events [11, 12].

The Monitoring Medicines project coordinated by the Uppsala Monitoring Centre (UMC) in 2011 demonstrated that spontaneous reporting could provide an indication of the presence of substandard/spurious/falsely labelled/falsified/counterfeit (SSFFC) medical products in healthcare systems. The UMC developed a signal-detection method in a retrospective setting using 24 MedDRA® preferred terms (PTs) indicative of inferior product quality within VigiBase®. A data-mining approach with three algorithms was applied to identify medicinal products associated with a higher-than-expected number of ICSRs. The main determinant was the lower 95% confidence interval (CI) of the comparative information component IC∆ exceeding 0. Several clusters of medicinal products with excess reporting of potential quality issues were highlighted and confirmed by information on product recalls or deficiencies. In 2014, the UMC implemented the developed algorithms on national pharmacovigilance data. Some of the identified clusters of the suspected SSMs could be validated by national regulators. Limitations of the survey included late ICSR submissions to Vigibase® and lack of data quality [13].

This pilot study used a data-mining approach broadly analogous to that of the UMC Monitoring Medicines project, but we applied a different disproportionality algorithm to detect potential SSMs. We employed the three stratification strategies in the pilot study on all five active substances and compared the results, whereas the UMC Monitoring Medicines project used these data-mining approaches independently. The other main point of difference was that the Monitoring Medicines project focused on the detection of falsified medicines, whereas our study targeted the identification of potential SSMs.

1.3 Objectives

The primary objective was to evaluate whether disproportionality analysis applied to individual case reports, accompanied by statistical stratification techniques, could be used for the detection of potential SSMs. Furthermore, we aimed to validate these techniques by comparing the results against examples from the literature of known and previously evaluated cases of SSMs reported to Vigibase®.

2 Materials and Methods

2.1 Data Source

Vigibase® was selected as the basis for research to identify potential safety hazards associated with SSMs without identifying individual patients or the original source of the reports [14]. We used the EB05 ratios produced by the Empirica Signal system, data-mining software (version 7.3.3.0.354, ORACLE) applied to ICSRs in VigiBase®.

2.2 Empirica Signal Application

Empirica Signal is a high-performance implementation of the multi-item gamma Poisson shrinker (MGPS) algorithm, which is linked to the marketing authorization holder (MAH) safety database, Argus Safety. For a drug–event combination (DEC), the adjusted value of an observed/expected ratio is denoted as the empirical Bayes geometric mean (EBGM) value [15]. The MGPS data-mining algorithm includes the computation of two-sided 90% CIs (EB05 < EB95) for EBGM. In general, MAHs and regulatory authorities use an EB05 or EBGM > 2 as a screening threshold for observations of disproportional reporting (ODRs) [16].

2.3 Data-Mining Analysis

We used 24 MedDRA® (version 17.0) PTs considered indicative of potential SSMs for the adverse event data-mining queries. The PTs were the same as those applied by the Monitoring Medicines project in 2011 [11] (see Electronic Supplementary Material 1).

Three different stratification strategies for detecting potential cases of product defects were assessed:

  1. 1.

    Medicines with an excess number of reports on the selected PTs relative to all other products in Vigibase® in the specified timeframe.

  2. 2.

    Medicines with an excess number of reports on the selected PTs relative to other products containing the same active pharmaceutical ingredients.

  3. 3.

    Medicines with an excess number of reports on the selected PTs relative to other products containing the same pharmaceutical substances in a specific country and year.

The third data-mining run was generated based on the stratification variables country and year of occurrence assuming the first and second data-mining runs showed an EB05 ≥ 2.

We excluded all ICSRs that did not specify the name of the medicinal product (‘NOS’ [not otherwise specified] or generic names in VigiBase®) as the hit could refer to multiple trade names. To evaluate the statistical significance of the disproportional reporting ratios for each DEC, we analyzed reports of trade names with N ≥ 1 ICSRs; EBGM ≥ 2, and EB05 ≥ 2 [15]. N was a significant index for monitoring the emergence of an adverse event but was independent of the signal score [17].

The entire dataset within VigiBase® was systematically screened using the specified MedDRA® PTs for higher-than-expected DECs. Specific medicinal products with an EB05 ≥ 2 (Fig. 1) were evaluated further. We then performed a pilot study on five medicinal products originally licensed by a single manufacturer but no longer patent protected: valsartan, methylphenidate, rivastigmine, clozapine, and carbamazepine (Fig. 2). In the analysis of this pilot study, the results for the names of the generic medicines as well as the respective equivalent proprietary products containing the five active substances were further investigated, represented in Figs. 3, 4 and 5, and the corresponding EB05 values summarized in Table 1. Each of the figures demonstrates the excess reporting rates of one of the stratification strategies.

Fig. 1
figure 1

Excess reporting of medicinal products in Vigibase® based on selected MedDRA preferred terms (EB05 ≥ 2). Output for all medicinal products (proprietary and generic as well as pharmaceuticals indicated by generic name or ‘not otherwise specified’) that were associated with a higher than expected number of individual case safety reports (EB05 ≥ 2), which warranted further investigation. EB05 fifth percentile of the confidence interval for the Empirical Bayes Geometric Mean  

Full size image
Fig. 2
figure 2

Excess reporting in Vigibase® (EB05 ≥ 2) of 23 proprietary and generic marketed medicines containing valsartan, methylphenidate, rivastigmine, clozapine, or carbamazepine. EB05 fifth percentile of the confidence interval for the Empirical Bayes Geometric Mean

Full size image
Fig. 3
figure 3

Excess reporting of generic forms and corresponding Novartis products of five selected active pharmaceutical ingredients. 14 pharmaceuticals with excess reporting rates (EB05 ≥ 2) via application of stratification strategy 1. Terms such as ‘valsartan 1’ refer to the 14 trade names of Novartis brand and Novartis generic medicines identified in this study. Methylphenidate 1,4,5; carbamazepine 2, and rivastigmine 1 show reports with multiple MedDRA® terms. Novartis valsartan, rivastigmine, and clozapine are not included in this figure, as the reporting rates of these products did not meet the threshold (EB05 values < 2). EB05 fifth percentile of the confidence interval for the Empirical Bayes Geometric Mean, N number of occurrences

Full size image
Fig. 4
figure 4

Assessment of excess reporting of generic forms and respective Novartis equivalent of selected active pharmaceutical ingredients with the same substance. Seven medicinal products with excess reporting rates (EB05 ≥ 2) via application of stratification strategy 2. The other seven medicinal products (Fig. 3) revealed EB05 values < 2 or were not reported (see Table 1). The data on the y axis are shown in logarithmic form. EB05 fifth percentile of the confidence interval for the Empirical Bayes Geometric Mean, N number of occurrences

Full size image
Fig. 5
figure 5

Country–year excess reporting of rivastigmine patch in 2 consecutive years. Multiple MedDRA® terms were reported for the same medicinal product. EB05 fifth percentile of the confidence interval for the Empirical Bayes Geometric Mean, N number of occurrences

Full size image
Table 1 Summary of identified trade names with excess reporting rates of the three stratification strategies on five active pharmaceutical ingredients
Full size table

We performed a literature research for advisory letters and product or batch withdrawals for each of the identified proprietary and generic products through official health authority and national pharmacovigilance center websites (independent of the MAH or manufacturer) concentrating on product quality issues or defects. We did not conduct a systematic review of all publications, as the research for advisory letters published by independent researchers or by the competent authorities was deemed adequate for the purposes of this study. Search terms included combinations of the following keywords: advisory letter, drug removal, substandard, quality, and trade name with country of occurrence of potential SSMs from retrospective analysis. Searches were limited to publications after 2001. The keywords were applied to the following websites: US FDA, Medicines and Healthcare products Regulatory Agency (MHRA), European Medicines Agency (EMA), and selected pharmacovigilance center websites, e.g., Netherlands Pharmacovigilance Centre. From the resulting alerts, we extracted the product name, dosage form, year of the alert, and description of the product defect [4, 18–20]. In addition to the literature research, we applied the three stratification strategies to two confirmed SSMs categorized by the UMC in 2014 as confirmed SSFFC clusters [12].

3 Results

Based on the search strategy and threshold criteria described, 2506 DECs were generated (see Fig. 1).

We excluded 664 potential target SSMs from further analysis, as the precise trade name of the product was not specified. The 664 excluded hits contained 478 medicinal products denoted as NOS, and 186 reports using generic names (e.g., fluoxetine) or broad therapeutic categories or drug class (e.g., ‘protectives against ultraviolet radiation for topical use’ or ‘centrally acting sympathomimetics’). The excluded hits could not be associated with a particular proprietary medicine or with a specific manufacturer if the product was a generic. Thus, warning letters were not applicable, as they always involved a specific medicinal product.

In total, 23 trade names of generic and proprietary medicinal products resulted when we filtered the data to include only reports associated with pharmaceuticals containing valsartan, methylphenidate, rivastigmine, clozapine, or carbamazepine (Fig. 2). Table 1 shows the stratification strategy results (EB05 values) of 23 proprietary and generic marketed medicines containing valsartan, methylphenidate, rivastigmine, clozapine, or carbamazepine. Excess reporting for both stratification method 1 and 2 was determined for nine medicinal products, whereas three pharmaceuticals (clozapine 4, carbamazepine 4, and rivastigmine 1) showed high EB05 values for all three data-mining operations. The three products with potential quality defects identified originated in Italy and the Netherlands.

In a subset analysis, we filtered the trade names of generic medicines produced by Novartis and the equivalent proprietary medicines containing valsartan, methylphenidate, rivastigmine, clozapine, or carbamazepine. This resulted in 21 ODRs involving 14 proprietary medicines, as occasionally there were reports of products with multiple MedDRA® terms (Figs. 3, 4).

The results of the stratification analysis are shown in Figs. 3, 4 and 5. Figure 3 shows excess reporting rates for all generic forms, including Novartis’ own generic products, valsartan and methylphenidate. The reporting rates of Novartis valsartan, rivastigmine, and clozapine did not meet the threshold (EB05 values <2) and are therefore not included in this figure. In analogy to Fig. 3, Fig. 4 illustrates excess reporting rates of 7 of 14 medicinal products, as the EB05 value of the other seven pharmaceuticals for product substance stratification did not meet the threshold (EB05 <2).) The criterion for excess reporting for stratification analysis 1 and 2 was fulfilled for seven medicinal products, whereas the rivastigmine patch met all the criteria, including the country–year stratification operation (Fig. 5).

The literature search for the rivastigmine patch revealed two letters relating to quality defects and associated safety concerns [18, 19]. Details of ICSRs provided in direct health professional communications could be matched with specific case descriptions provided in the VigiBase® records, including PT, country, and year of occurrence (Fig. 3). This medicinal product showed the highest value of EBGM and EB05 score with PT “Therapeutic response unexpected with drug substitution” (EBGM = 528,689; EB05 = 391,054; N = 32) compared with other compounds under study (Fig. 4). It was evident from Vigibase® and the published ‘dear healthcare provider letters’ that a case series had been identified. The literature research for advisory letters for the other two pharmaceuticals (clozapine 4, carbamazepine 4), which also met all three stratification criteria was impeded because the identified proprietary names could refer to the international non-proprietary name (INN) or to multiple generic brands. No advisory letter on safety or quality was found for the other medicinal products under study.

Two medicinal products, salbutamol and ethinyl estradiol/norgestrel combination tablets, were identified as confirmed SSFFC clusters by the UMC in 2014 because of excess reporting rates. The ethinyl estradiol/norgestrel tablets were referenced in an FDA warning letter in 2012 about a recall of 14 batches because of the possibility of inexact tablet counts or “out of sequence” tablets [21]. In our study, this medicinal product showed ODRs that exceeded thresholds for stratification strategies 1 and 3. In 2012, eight reports in the USA of “product quality issue” with an EB05 value of 26.77 for this combination product were submitted to Vigibase®.

Quality defects, including lack of effect due to inadequate administration technique and use of expired products, were documented with a salbutamol solution in the USA in 2012 [12]. In this report, the medicinal product showed ODRs that exceeded the thresholds for stratification strategies 1 and 3. There were 96 reports of ‘drug ineffective’ (EB05 4.5); 102 cases of ‘product quality issue’ (EB05 20.28), and nine reports of ‘therapeutic response decreased’ (EB05 2.04) submitted to Vigibase®. The stratification results for both products are presented in Table 2 [12].

Table 2 Summary of pharmaceuticals with excess reporting for all three stratification strategies for two confirmed substandard products [17]
Full size table

4 Discussion

Our study presents a new and effective way to detect potential SSMs. The data-mining approach used in this pilot study resembled the method presented in the UMC Monitoring Medicines project [11], but we applied a different disproportionality algorithm to detect potential SSMs. In the first sub-analysis, we identified active substances where reporting for selected PTs exceeded the threshold when compared with all other medicinal products in Vigibase®, whereas the Monitoring Medicines project started by selecting active substances within a particular country. Our second step was to evaluate disproportionality results for products containing the same active substance. Finally, for each medicinal product where reporting above the threshold occurred, we identified the country and year of occurrence. The UMC group analyzed the top 30 medicinal products with the highest disproportionality scores and assessed a further randomly selected dataset for comparison, whereas we analyzed the entire dataset for five active substances selected as the basis for this research and applied all stratification strategies on these five active substances and compared the results. The application of the three data-mining stratification strategies on ICSRs using the Empirica software discovered medicinal products with quality defects that were then confirmed by advisory letters from official health authority and pharmacovigilance centers. VigiBase® proved to be useful reference point for the identification of clusters of potential defective medicines.

The research presented here has augmented and extended previous work conducted by the UMC [11, 12]. This study included all marketed medicines in Vigibase® based on the 24 MedDRA® terms indicative of product quality defects containing valsartan, methylphenidate, rivastigmine, clozapine, or carbamazepine. Tables 1 and 2 illustrate that there were excess reporting rates (EB05 ≥ 2) for both proprietary and generic medicines.

After extensive investigation of the 23 identified trade names with excess reporting rates in Vigibase®, one potential SSM fulfilled the criteria for all stratification strategies, and a product defect was confirmed via an independent report from the Pharmacovigilance Centre in the Netherlands [22] via distribution of two letters to healthcare professionals. In Figs. 1, 2, 3, 4 and 5 and Table 1, the ODRs for the rivastigmine patch demonstrated high EB05 values and were demonstrable outliers. Similar to rivastigmine 1, the other two pharmaceuticals clozapine 4 and carbamazepine 4 also met the EB05 threshold for all three stratification techniques, but the literature research for advisory letters was hampered by the absence of specific product details.

The identified excess reporting rates for two of three stratification strategies on confirmed substandard clusters of salbutamol and ethinyl estradiol/norgestrel reinforces the potential utility of this data-mining approach. Compared with detection of SSMs with analytical devices, this technique is a non-destructive and reproducible method that can support non-governmental centers, healthcare professionals, manufacturers, and health authorities in low- and middle-income countries to triage for confirmatory analysis testing of medicinal products.

The findings in this study support the need for further research to refine the algorithm so this exploratory research becomes a matter of routine programming within the competent authorities and MAHs. It is our intention to optimize the sensitivity and selectivity of the method described. It is clear from this initial study that public health benefits could result from the early detection and reporting of quality defects associated with SSMs.

We recognize there are limitations applicable to this systematic analysis for the detection of SSMs. Safety data collected by MAHs include reports of product complaints related to quality defects. According to the existing regulations, these two datasets (safety and quality) are governed rather differently, with quality under Good Manufacturing Practice (GMP) and safety under Good Pharmacovigilance Practice (GPvP). There are multiple areas of overlap between safety and quality defects. For example, there is significant duplication across aggregate safety reports and submission of these data within periodic quality reports. Nevertheless, there are gaps in the analyses of these data in combination in order to form potentially important conclusions that may impact public health. An illustration of this is provided in Fig. 6. Manufacturers take great care to reconcile the two datasets according to the regulations. Regulatory inspections often focus on this area, and this has resulted in findings [25], warning letters [26], and more serious sanctions [27]. Our recommendation is that both regulatory authorities and MAHs could consider the following.

Fig. 6
figure 6

Inter-relationships of good manufacturing practices and good pharmacovigilance practices, and aggregate regulatory reports. AE adverse event, APQR aggregate product quality reviews (including but not limited to annual product reviews and product quality reviews), DSUR development safety update reports [23], LoE lack of efficacy, PSUR periodic safety update report [24], QD quality defects, RMP risk management plan

Full size image

  • Application of this method to all medicines using large safety databases (e.g., EudraVigilance) to aid the detection of adverse patient outcomes related to suspected SSMs. The results could help improve public health by earlier identification of products with quality defects.

  • Recommend targeted analytical testing in developing countries or regions based on the results of disproportionality analyses to detect SSMs.

Juhlin et al. [12] faced the same challenges during their survey. This was a retrospective study, therefore we have not been able to obtain samples of the suspected SSMs for testing as they were no longer available. In Vigibase®, the sensitive personal health information of patients and the contact details for patients and primary reporters are anonymized to prevent the identification of individuals [28]. Consequently, it was not possible to contact the report sources to obtain follow-up data and thereby consolidate and potentially extend our preliminary findings. In addition, the majority of drugs in VigiBase® were described by their active ingredients in a non-specific manner (NOS).

Under-reporting [29], particularly by resource-limited countries, meant only a relatively small number of ICSRs were associated with lack of efficacy events. Most of the ICSRs in this study originated from Europe (Italy, Netherlands, Denmark) and from Canada. Relatively few ICSRs originated from Brazil, Mexico, or South Africa. Healthcare professionals play a very important role in spontaneous reporting and, particularly in Europe, patient reporting has been actively promoted [29]. This could be augmented by requesting that patients and caregivers take action to report possible quality defects and lack of efficacy.

We determined there was no international consensus regarding MedDRA® terms describing SSMs. Initially, we started with the 24 PTs [11]. In contrast, the UMC publication from 2014 [12] included 77 PTs. We propose that the pharmaceutical industry and regulatory authorities collaborate with the MedDRA® Maintenance and Support Services Organization (MSSO) to develop a standardized MedDRA® query (SMQ) for SSMs. Perhaps the most important adjunct to the research described is the essential activity of conducting field-based sampling and testing. New portable devices will allow rapid and accurate assessment of samples purchased from suppliers to further assess the viability of signals generated from the screening of VigiBase®.

5 Conclusions

We have provided evidence of an effective method for the detection of SSM signals using a large pharmacovigilance dataset. The signal that was generated from the rivastigmine patch was confirmed by two independent publications in the Netherlands, both of which emanated from a pharmacist-based monitoring program [18, 19]. Furthermore, we confirmed the results of the data-mining technique that the ODRs for two medicinal products were related to SSMs as originally shown by the UMC. Our findings using this novel method of detecting potential SSMs is a positive step towards addressing the supply of poor-quality medicines. Further validation would enable the routine use of this approach by competent authorities and MAHs.