Introduction

Because the patient populations included in clinical trials are small and uniform, the safety information gained from the pre-marketing phase is limited [1]. Therefore, during the post-marketing phase, to compensate for the lack of safety information obtained during the pre-marketing phase, it is essential to identify and follow-up on previously unnoticed risks when the relationship between a drug and adverse events (AEs) is uncertain [2].

Spontaneous reports play a major role in the identification of post-marketing safety issues [3,4,5,6,7,8]. They can provide important information related to not only rare AEs that were not detected in the pre-marketing phase but also known serious adverse drug reactions concerning at-risk groups, risk factors, and clinical features [3]. In the first step of risk evaluation, spontaneous reports provide triggers for detecting a sign of risk, which is often called a signal [9, 10]. A signal does not determine the relationship between drugs and AEs, so we need to examine whether the signal truly indicates a risk. Based on further examinations, we make a decision to take safety regulatory actions in response to the signal. The workflow from signal detection to decisions regarding safety regulatory actions is called “signal management” and an overview of signal management is included in periodic benefit-risk evaluation report (PBRER) [9,10,11].

Some guidelines and reports related to signal management recommend considering several factors during signal prioritization and assessment to rationally determine which actions to take [9, 10, 14]. Module IX of the Guideline on Good Pharmacovigilance Practices, a procedure manual for signal management created by European Medicines Agency (EMA), recommends considering factors such as strength of evidence, previous awareness, and clinical relevance/context [9]. Some European countries have introduced a method of prioritizing signals by scoring these factors, which is called impact analysis, in their signal management process [12, 13]. However, it is unclear how strongly these factors contribute to decision-making regarding safety actions in response to signals.

Previous research investigated which factors play a role in the subjective process of signal selection [15, 16]. According to one of those, presence of “serious reports”, AEs designated as a World Health Organization (WHO) “critical term”, AEs that were unlabeled, and presence of a disproportionate association were all independently associated with signal selection [15]. Other showed that 4 characteristics of drug safety signals (presence of evidence in multiple types of data sources, mechanistic plausibility of the drug-event association, seriousness of the event, and a drug age < 5 years) were associated with product information (PI) updates [16].

The factors that influence decisions to take safety action in signal management have presented in previous research; however, all the previous research was based on the outcomes of signal management in the European Union (EU). To investigate different aspects of previous results, we focused on the outcomes of signal assessment in the USA (US) and considered the association between regulatory actions and potential factors including those which were not included in the previous studies. This study aimed to identify the factors that were associated with the decision to take safety actions in routine signal management based on the Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) by analyzing the safety signals identified from the FAERS and their relevant information.

Method

Selecting Signals for this Study

In accordance with Title IX, Section 921 of the Food and Drug Administration Amendments Act of 2007 (FDAAA), the FDA creates a quarterly report called “Potential Signals of Serious Risks/New Safety Information Identified from the FAERS” (the FAERS quarterly report) [17, 18]. The FAERS quarterly reports give information about what signals FDA identified from FAERS and what actions FDA took for them.

From the FAERS quarterly reports between 2008 1Q and 2014 4Q, we extracted the signals that resulted in regulatory action and those that were closed with no action. We determined that the signal resulted in regulatory action if the following information was contained in the column of additional information: labeling update, Risk Evaluation and Mitigation Strategies (REMS), product recall, withdrawal. We determined that the signals that were closed with no action if the following statements or similar contents were contained in the column of additional information:

  • The FDA decided that no action is necessary at this time based on available information,

  • The FDA has determined that the current labeling is adequate and that no further regulatory action is needed at this time.

In addition, several signals had multiple AEs in different fields of disorders. Among the drug and AE pairs for such signals, some did not clearly indicate whether regulatory action was taken. In such cases, we determined that the pairs resulted in no regulatory action. We determined that the assessment of the signal was ongoing and excluded the signal from the present study if the following statement or similar contents were contained in the column of additional information:

  • The FDA is evaluating the need for regulatory action.

Factors that are recommended for consideration in signal management

According to some guidelines related to signal management, factors that are potentially important during signal assessment were defined and classified broadly into 3 categories: strength of evidence, previous awareness, and clinical relevance/context [9]. A guideline for pharmacovigilance practices and pharmacoepidemiologic assessment created by the FDA contains similar contents for evaluating signals [19]. Specific factors in each category were extracted from the FAERS quarterly reports and the FAERS data released in the same quarters when the signals were published. The rationale for and definitions of classification of the collected factors are explained below.

Strength of Evidence

Presence of Disproportionate Reporting

The guidelines for good pharmacovigilance practices in the EU and the USA recommend employing statistical or mathematical approaches (called data mining) to examine reported AEs [19,20,21]. We calculated reporting odds ratios (RORs) to confirm the disproportionate reporting of the signals using the published FAERS data set [22, 23]. The signals were considered disproportionately reported if 2 or more cases were reported and the lower bound of the 95% confidence interval of the ROR was greater than 1. The disproportionality analysis is explained in detail in our previous article [24].

Positive Dechallenge or Rechallenge

The presence of positive dechallenge or rechallenge provides consistent evidence that the reported AE was caused by the suspected drug. The percentage of the cases with positive dechallenge or rechallenge in total cases of the signal was calculated.

Temporal Relationship

In evaluating the causal relationship between suspected drugs and AEs, temporal relationship (timing of administration of the suspected drug and the occurrence of AE) is an important factor. We defined that temporal relationship existed if the following conditions were satisfied:

The date of AE onset was within the period from the start date until the end date of the suspected drug’s administration. In addition, the period from the drug’s administration to the AE onset was reasonable in light of the time to develop the AE, which was commonly known.

The percentage of cases with temporal relationship in the total cases of the signal was calculated.

Mechanistic Plausibility

The presence of mechanistic plausibility offers supporting information to explain causality in terms of drug action mechanisms. In this study, three authors with different backgrounds (CF, YH and MN) independently judged the mechanistic plausibility of each drug and AE pair of signals based on their biological and pharmacological knowledge. To prevent a big difference in the interpretation of mechanistic plausibility, the following criteria were set in advance; mechanistic plausibility was considered present if the causality was able to be hypothesized or explained in terms of the drug’s mechanism of action; mechanistic plausibility was not considered present only if the risks related to the AEs had been listed in the drug’s label or if the AEs had been considered as class effects. The inter-rater reliability among the authors was good (kappa = 0.77). If their answers were not unanimous, presence of the mechanistic plausibility for the signal was decided by consensus.

Number of Cases in the Last One-Year Period Before the Signal Indication

An exponential increase in reports over a short period may indicate an increasing risk. We counted the number of cases that were reported in the last one-year period before the publication of the FAERS quarterly report indicating the signal. The percentage of the cases accumulated in the last one-year period before the signal indication in the total cases of the signal was calculated.

Previous Awareness

Previous awareness is defined as whether the signal relates to an adverse reaction has already been included on the label for the active substance of interest or other medicinal products containing the same substance, or whether the association was assessed in the initial application for marketing authorization or any other regulatory procedure based on information held or known by any organization [9]. To determine the presence of previous awareness of the signals, we referred the previous label of the suspected drug updated before the quarter when the signal was posted to check whether information relevant to the signal was already included.

We defined that the signals had previous awareness if any of the following conditions were satisfied: 1. The AE with the same term was already included in the label, 2. The AE was similar to the risk which was already described in the label, but their severity was clinically significantly different (e.g., even if hepatic disorder was already described in the label, hepatic dysfunction and hepatic failure were identified as signals), 3. The AE was an identified risk, but it had novelty in terms of frequency or population (e.g., even if the risk for pregnant women and fetuses was already described in the label, the risk for children was identified as a signal). We collected the labels of suspected drugs from the Drug@FDA database (https://www.accessdata.fda.gov/scripts/cder/daf/). If no label before the signal identification was available, we checked the relevant information using the edition of the Physician’s Desk Reference published before the signal was identified. If the relevant information could not be confirmed in anywhere, we excluded the signals from the analysis.

Clinical Relevance and Context

Importance of the AEs

AEs considered to be clinically important are likely to be reported [25]. We assessed the importance of the AEs for the signals based on the EMA Important Medical Events list (the IME list) [26]. This list aims to facilitate the classification of suspected adverse reactions as well as aggregated data analysis and case assessment in the frame of the day-to-day pharmacovigilance activities of stakeholders in the EU. It is also widely used for the judgment of important AEs in other countries. If the AE was listed on the list, it was considered “important”.

Serious Outcome

Outcomes after the occurrence of AEs should be considered during signal assessment. In accordance with the international definitions of seriousness [27], we defined that the case had serious outcome if one or more of the following outcomes existed: death (DE), life-threatening (LT), necessity of hospitalization or prolongation of existing hospitalization (HO), persistent or significant disability/incapacity (DS), congenital anomaly/birth defect (CA), other medically important events or reactions (OT). The percentage of the cases with serious outcome in the total cases of the signal was calculated.

Risks for Special Populations

Consideration of whether the risk occurs in a special population (e.g., pediatrics, pregnant women, patients who have a certain risk factor) is recommended during signal assessment. We determined whether the signals were specific to a special population based on the descriptions of the FAERS quarterly reports.

Other

Age of Drug

Spontaneous reports increase immediately after a drug is marketed (Weber effect) [28], so it is assumed that signals for new drugs are likely to be identified more easily than those for old drugs. We calculated the period of time for the drug on the market (the time from the marketing approval of the suspected drug to the identification of the signal). We classified the signals into the following 2 categories: time of marketing authorization < 5 years or ≧5 years.

Data Analysis

The univariate and multivariate logistic regression were employed to assess the influence of various factors on the decision to take regulatory action. The dependent variable was the regulatory action (action or no action). The explanatory variables were the disproportionate reporting (yes or no), positive dechallenge or rechallenge (%), temporal relationship (%), mechanistic plausibility (yes or no), number of cases in the last one-year period since the signal report (%), previous awareness (yes or no), importance of the AEs (yes or no), serious outcome (%), risks for special populations (yes or no), and age of drug < 5 years (yes or no). At first, we performed a univariate logistic regression analysis. Factors associated with regulatory actions that had a p value < 0.2 were included in the multivariate logistic regression model. We determined that factors with p < 0.05 were statistically significantly associated with the regulatory actions. The results are expressed as odds ratios (ORs) with 95% confidence intervals (CIs). Furthermore, as a supplemental analysis, the ordered logistic regression was conducted by a similar procedure as described above when the dependent variables were divided into the following three categories: high-severity (REMS, product recall, withdrawn, label changes in warnings and precaution, boxed warning, contraindications), low-severity (label changes in post-marketing experience, adverse reactions, any section other than mentioned above), and no action. The analyses were performed using StatsDirect version 2.7.9 (StatsDirect, Ltd., Altrincham, Cheshire, UK) and R software, version 3.4.0 [29].

Result

Between 2008 1Q and 2014 4Q, 258 signals were listed in the FAERS quarterly reports (Fig. 1). The temporal trend of the number of the signals is shown in Fig. 2. Among these signals, signals for which evaluations were ongoing (N = 17), and signals for quasi-drugs (a category between medicine and cosmetics such as sunscreen) (N = 1), combination drugs (N = 1), drug-drug interactions (N = 9) and comprehensive AEs not definable by the Medical dictionary for Regulatory Activities (MedDRA) terms (N = 14) were excluded. A total of 216 signals were examined in this study. Among them, 165 led to regulatory actions and 51 were assessed and closed with no action (Fig. 1). The regulatory actions taken for the 165 signals were labeling changes (N = 159), REMS (N = 4), product recall (N = 1), and withdrawal (N = 1) (Table 1). For the 51 signals that were closed with no action, the reasons were “no action is necessary at this time based on available information.” (N = 31), “the current labeling is adequate and no further regulatory action is needed” (N = 16) and unknown (N = 4) (Table 2).

Fig. 1
figure 1

Flowchart of selecting the signals for evaluation in this study. Among 216 signals to be evaluated in the present study, 165 led to regulatory actions and 51 were assessed and closed with no action

Full size image
Fig. 2
figure 2

The temporal trend of the number of the signals. From 2012, the number of the signals markedly decreased

Full size image
Table 1 Safety actions in response to the 165 signals from the FAERS
Full size table
Table 2 Reasons for no-action judgments in response to 51 signals
Full size table

To assess an association between previous awareness and regulatory actions, 6 signals were excluded because the relevant information could not be confirmed anywhere. As a result of the univariate logistic regression analysis, positive rechallenge (unadjusted OR 1.09, [95%CI 0.97–1.23]), number of cases accumulated in the last one-year period before the signal indication (unadjusted OR 1.01, [95%CI 1.00–1.02]), previous awareness (unadjusted OR 2.33, [95%CI 1.20–4.54]), serious outcome (unadjusted OR 0.98, [95%CI 0.96–1.01]) and risks for special populations (unadjusted OR 3.92, [95%CI 0.50–30.92]) showed p values < 0.2 (Table 3). These factors were included in the multivariate logistic regression model. Finally, number of cases accumulated in the last one-year period before the signal indication (adjusted OR 1.01, [95%CI 1.00–1.03]) and previous awareness (adjusted OR 2.43, [95%CI 1.22–4.85]) were associated with regulatory action. For the 88 signals with previous awareness (Table 3), relevant information was already described in the pre- or post-marketing AEs section (N = 85), the warning and precautions section (N = 45), or the contradictions (N = 12) of the previous labels of the suspected drugs (Fig. 3). As a result of the multivariate ordered logistic regression analysis, positive rechallenge (adjusted OR 1.09, [95% CI 1.00–1.18]), number of cases accumulated in the last one-year period before the signal indication (adjusted OR 1.01, [95% CI 1.00–1.02]), and previous awareness (adjusted OR 2.99, [95% CI 1.69–5.31]) were associated with regulatory action (Supplementary material 1).

Table 3 The association between the characteristics of the signals from FAERS and the regulatory actions
Full size table
Fig. 3
figure 3

Number of the relevant information by section of the label described in the previous labels of the suspected drugs. Among the 165 signals for which regulatory actions were taken, relevant information was already included in the previous label of the suspected drug in 88 cases. Major of those were described as AEs that occurred in pre- or post-marketing phases

Full size image

Discussion

We found that number of cases accumulated in the last one-year period before the signal indication and previous awareness of the risk were associated with the decision to take regulatory action. The similar results were obtained from the supplemental analysis.

In general, increase of AE reporting during a short period indicates an expansion of the risk [9]. The number of AE reports are influenced by various factors (e.g., immediately after a drug is marketed, additional indications or changes of dosage form), and it is necessary to regularly monitor the changes in the number of cases. We should be careful when a large number of AE cases accumulate in a short period.

Among the 165 signals for which regulatory actions were taken, 88 cases were already described in the label or were similar to the AEs in the label except for the severity, frequency, and population. The information was primarily described in the adverse reactions section (pre- and/or post-marketing experience) and the warnings and precautions. ICH E2C, which is intended to be a common standard for periodic benefit-risk evaluation reporting, explains that identified risk which is indicative of a clinically significant difference in the severity or frequency of the risk are regarded as new signals [27]. Rather than completely unknown risks, the risks with previous awareness which are indicative of a clinically significant difference from identified risks may lead to further regulatory actions. When assessing safety signals, we should pay attention to new information on not only unknown risks but also previously identified and potential risks.

The FDA manages significant safety issues that are identified during the evaluation of new drug application (NDA) or biologic license application (BLA) in an integrated fashion; it may require REMS or post-marketing requirement (PMR) to address the safety issues that are identified during the pre-marketing period as tracked safety issues (TSI) [30]. It was assumed that signals related to TSIs arising from spontaneous reports accumulated in the FAERS led to further regulatory actions. In the pharmacovigilance activities in FDA, for the safety issues of concern, the process of the continuation of collecting information, obtainment of new evidence, and consideration of a further regulatory action has been established.

Although previous awareness was identified as the factor that most strongly affected the decision to take regulatory action in this study, it is noted that the unknown risks without previous awareness (N = 71) also resulted in regulatory action. Routine signal management based on spontaneous reporting plays a role not only in the follow-up of risks for which there is previous awareness, but in the identification of unknown risks.

Other factors included in the multivariate logistic regression model in this study were positive rechallenge, serious outcome, and risks for special populations. Those factors may be potentially associated with the decision to take regulatory action during signal management. Positive rechallenge is a strong evidence for establishing causality. Our supplementary analysis showed that positive rechallenge was associated with regulatory actions. Serious outcome is a reference to evaluate the impact of the AE on patients. Although that information was essential to evaluate causality, they are often unavailable from spontaneous reports. Our study also showed that the quantities of available information from spontaneous reports were not different between the group for which regulatory actions were taken and the group for which no regulatory actions were taken (see supplementary material 2). A study based on spontaneous reports of adverse drug reactions from the Catalan Pharmacovigilance Center reported that more than one third of the reports from manufacturers did not include information that was considered a limiting factor to evaluate any causal relationship [30]. To make a decision on regulatory actions efficiently, it is needed to check the important factor to evaluate causality and improve quality of spontaneous reports.

During the pre-marketing stage, safety information for special populations, such as pediatrics, pregnant women, and patients with a particular risk factor, is usually lacking. It is difficult to include those populations in clinical trials due to the limited number of patients and ethical concerns. As an alternative, spontaneous reports are often used to examine drug safety in those populations [31,32,33,34]. Pharmacovigilance for special populations still owes a great deal to spontaneous reports.

Our results differed from those of previous study on some points [16]. The presence of evidence in multiple types of data sources was among the signal characteristics related to PI changes in the previous study. However, we could not perform a similar analysis because signal assessment evidence from sources other than the FAERS was not available from the FAERS quarterly reports. Among the signals we evaluated in our study, 24 led to FDA Drug Safety Communication actions. Based on the description in the Drug Safety Communication data summary, we organized the evidence used to evaluate these 24 signals. The assessment of these 24 signals considered evidence from the literature, observational studies, epidemiological studies, clinical trials, and the results of mini-sentinel pilot studies (see supplementary material 3). Recently, methods that complement routine drug safety signal analysis based on spontaneous reports using other sources, such as administrative claims data and electronic health data, have become widely used; e.g., the sentinel initiative [35]. To strengthen the evidence levels of signals from spontaneous reports, the activity verifying signals using other data sources will become popular in the future.

Mechanistic plausibility was associated with PI changes in the previous study; however, our results did not show an association between mechanistic plausibility and the decision to take regulatory actions. The standard of judgment in the EMA may not be the same with us. It seems to be difficult to be absolutely on the same page about this factor. Mechanistic plausibility is left to the judgment of the evaluators at the time. Austin Bradford Hill gave the following explanation of “plausibility” in his criteria for determining causation [36]: “What is biologically plausible depends upon the biological knowledge of the day.” It is difficult to consistently judge mechanistic plausibility, although it is a helpful factor in signal assessment [9, 10].

Signals for serious events and age of drugs (≤ 5 years) tended to lead to the PI changes in the previous study. However, those factors were not shown to be associated with the decision to take regulatory action in our study. Among the signals to be evaluated in our study, some for serious events or relatively new drugs led to no action because available information was probably limited [18]. Regulatory action may not be taken when supportive evidence is lacking even though it was a signal for a serious event or related to a recently marketed drug.

This study has some limitations. First, all the explanatory variables in this study were obtained from the published information, and we did not conduct systematic literature reviews to obtain information related to the signals. We could not consider information that was not available publicly; thus, other potential factors that were not examined in this study were left. Second, we could not explain all the FDA’s decision only by the factors considered in the present study. Other than the factors considered in this study, FDA considers information such as class effects, co- or pre-morbid medical conditions, concomitant medications, follow-up information, clinical and demographic characteristics, exposure duration, and route of administration [19]. In addition, the methods in this study were not along the FDA’s manner in some points. As the measure of disproportionality, we selected ROR, while the FDA employs Empirical Bayes Geometric Mean (EBGM). Mechanistic plausibility in this study was based on an assessment by us, not the FDA’s staff. From 2012, the number of the signals markedly decreased, and it seems that the criteria in the screening of signal from the FAERS has changed in the FDA (Fig. 2). However, we did not consider the possibility for a change of the role in FDA’s signal management. Therefore, the results in this study did not reflect all the FDA’s intensions for signal management. Third, in general, we suspect the signal when several factors are present in the cases (e.g., dechallenge is positive, temporal relationship is positive, and previous awareness is present). Necessity of a regulatory action for a signal is not decided only with the presence of one factor. We did not analyze the association between factors and regulatory actions when the presence of more than 2 factors were confirmed.

When assessing safety signals, we should be careful of the AEs with a large number of cases accumulated rapidly in a short period. In addition, we should pay attention to new information on not only unknown risks but also previously identified and potential risks. This knowledge will help the signal assessment in the periodic benefit-risk evaluation.