Introduction

In pharmacovigilance activities, spontaneous reports of adverse drug reactions (ADRs) are an essential data source to identify safety signals and assess the safety profile of a marketed drug. However, under-reporting is one of the disadvantages of spontaneous reporting, considering that not all adverse events (AEs)/ADRs suspected to be associated with drug use are reported to regulatory authorities [1]. Under-reporting can lead to a bias in statistical signal detection and a failure to identify potential drug-associated risks [2]. Although under-reporting has been recognized for a long time, it continues to occur [3, 4].

Previously, studies pinpointed the existence of under-reporting and attempted to estimate its degree by comparing the number of ADR reports submitted to the regulatory authorities with the number of ADRs estimated from various data sources [5,6,7]. Hazell and Shakir systematically reviewed studies on under-reporting and confirmed that > 90% of ADRs were likely to be unreported [5].

Most of the previous studies in this field derived the estimated number of ADRs from various data sources, such as data from prescription-event monitoring, hospital admission data, and claim databases, and were not based on all cases treated with a drug. Unless a registry system is implemented or a prospective clinical study is performed, it is impossible to collect ADR information from all drug-treated cases. Therefore, most of these studies do not adequately represent the entire patient population, and it is difficult to obtain a complete picture of the extent to which ADRs have been under-reported. Additionally, differences in the medical systems and practices among countries or regions affect the behaviors related to ADR reporting, which may result in differences in under-reporting among different countries or regions.

In Japan, at specific instances, the regulatory agency requires a marketing authorization holder to conduct a single-cohort study on all patients to whom the drug is administered over a certain period after its launch, which is called “all-case surveillance” [8, 9]. An all-case surveillance study is a type of observational cohort study that is performed for newly approved drug based on the instruction by the regulatory authority, in which all patients to whom the drug is administered are enrolled until a predetermined sample size is reached [8, 10]. Theoretically, all data for the patients enrolled in all-case surveillance are collected using a case report form, and the number and incidence rate of ADRs based on all patients treated with the drug can be obtained. In contrast, drug-safety information of the patients who started taking the drug after the enrollment period of the all-case surveillance is collected using routine spontaneous reports, which may be under-reported, i.e., the periods of all-case surveillance and routine spontaneous reports are clearly distinguished. In addition, in Japan, information on all serious ADRs (sADRs) for approved drugs collected from spontaneous reports, post-marketing clinical trials, and observational studies (including all-case surveillance studies) is reported to the Pharmaceuticals and Medical Devices Agency (PMDA) within the period specified in the relevant guidelines by the International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use [11]. “Serious” ADRs are defined as those that “results in death”, “is life-threatening”, “requires hospitalization or prolongation of existing hospitalization”, “results in persistent or significant disability or incapacity”, “is a congenital anomaly/birth defect” or “is a medically important event or reaction” [12]. Such information on sADRs is recorded in the Japanese Adverse Drug Event Report database (JADER). Therefore, we hypothesized that the effect of under-reporting of ADRs can be estimated by comparing the number of sADR reports obtained in the all-case surveillance period with that obtained in the spontaneous report period in the JADER.

Thus, this study aimed to estimate the extent of under-reporting of ADRs in the Japanese medical system and practice using the scheme of all-case surveillance.

Materials and Methods

In this study, we compared the number of sADR reports for the investigated drugs between the period of the all-case surveillance and that after the all-case surveillance compiled in JADER.

Study Periods

In this work, we defined two different study periods; one was the enrollment period for the all-case surveillance, which was termed the “all-case surveillance period,” and the other was the period that followed the enrollment period for the all-case surveillance, which was defined as the “spontaneous report period.” We set the maximum length of the spontaneous report period to 3 years (36 months).

Data Sources

We used the JADER database to acquire sADR information on the investigated drugs. Data from JADER used in this study were those reported during the 2nd Quarter (April) of 2004 and the 3rd Quarter (September) of 2018. ADR name, indication, and medical history name are coded as Preferred Terms (PTs) of the Medical Dictionary for Regulatory Activities/Japanese version (MedDRA®/J; version 23.0).

Investigated Drugs

Considering that this analysis aimed to compare the number of ADR reports of a drug between the all-case surveillance period and the spontaneous report period, we selected drugs with new active ingredients that were newly approved from 2010 to 2016 and for which all-case surveillance was conducted. In addition, we set the following criteria for the selection of drugs to be investigated in this study, to improve the reliability of our study outcomes: (i) all-case surveillance period > 3 months, (ii) mean number of sADR reports per month > 4, and (iii) availability of the month of completion of the all-case surveillance period. The all-case surveillance period for each drug was confirmed based on the Risk Management Plan in the PMDA website or on the analysis report of the all-case surveillance published in the website of the pharmaceutical company [13]. Some drugs obtained an additional indication in the study period. However, the ADRs related to the first indication were used in the analysis.

Outcome

The outcome of this study was the number of reported patients with sADRs in JADER (termed “sADR case” hereafter). Patients who had multiple sADRs were counted as one case. The method of collecting ADRs changes according to the date of the first administration of the drug to the patient; therefore, sADR cases were counted based on the first administration month of each investigated drug.

Regarding the main outcome, we counted all sADR cases. Furthermore, we conducted a sub-analysis using the following two outcome measures: (a) sADR for which the PT corresponded to the definition of the “Important Medical Event (IME)” list (Version 23.0) issued by the European Medicines Agency [14], and (b) sADR with death.

Statistical Analysis

The characteristics of the investigated drugs were summarized according to the year of approval, type of indication, and duration of the all-case surveillance period.

As a descriptive analysis, the monthly average of the sADR cases in the all-case surveillance period and the spontaneous report period were calculated for each drug. In addition, the reduction rate of sADR cases for each drug was calculated using the following formula:

$$Reduction \, rate \left(\%\right)=1-\frac{Monthly \, average \,of \,sADR \,cases \,in \,spontaneous \,report \,period}{Monthly \,average \,of \,sADR \,cases \,in \,all-case \,surveillance \,period}.$$

The number of patients exposed to a drug has a secular trend, which affects the sADR cases in each period. To estimate the magnitude of “under-reporting” by controlling the secular trend caused by a change in the number of patients exposed to a drug, we performed a segmented regression analysis of an interrupted time series (ITS) [15]. The ITS study design is a quasi-experimental design that is frequently used to assess the impact of policy changes or experimental interventions. Using this study design, the repeated outcomes over time before and after the implementation of an intervention was measured and the changes in the level (intercept) and slope between before and after the intervention periods was compared to evaluate the intervention effects quantitatively [15, 16].

In the ITS design, outcomes measured over time were divided into two or more segments before and after the intervention. Subsequent changes in the level (intercept) and slope of the segments were compared to assess the intervention effect. Changes in level and slope represent immediate and gradual changes in the outcome, respectively.

In this analysis, the following log-linear regression model with a negative binomial distribution as a linked function and month as a time interval was used.

$${\text{ln}}{Y}_{t}={\beta }_{0}+{\beta }_{1}T+{\beta }_{2}D+{\beta }_{3}TD.$$

The abovementioned model includes the following three variables: T is the time elapsed since the drug launch per month, D is a binary variable indicating the all-case surveillance period (D = 0) or spontaneous report period (D = 1), and Yt is the number of sADR cases that started receiving the drug at month T. The coefficient of β0 is the baseline level at T = 0, β1 indicates the overall secular trends over the whole period, β2 indicates the level (intercept) change following the completion of the all-case surveillance period, and β3 indicates the slope change following the completion of the all-case surveillance period. As an example, ITS analyses using a log-linear regression model for Everolimus and Panitumumab are depicted in Fig. 1.

Fig. 1
figure 1

ITS analysis for Everolimus and Panitumumab. Black dots and line: number of sADR cases per month and log-linear regression line in the all-case surveillance period. Blue dots and line: number of sADR cases per month and log-linear regression line in the spontaneous report period. Red line: β2: level (intercept) change

Full size image

To measure the immediate and gradual effect of the completion of the all-case surveillance on the outcome quantitatively, we performed a statistical test of changes in the level (β2) and slope (β3). As the method of collection of ADRs was divided into before and after the end of the all-case surveillance period, we hypothesized that the change in the number of sADR cases caused by the completion of the all-case surveillance occurs immediately. Therefore, we also calculated a reduction rate in level (1-exp(β2)) to examine the magnitude of under-reporting.

Supplementally, the number of prescriptions for the investigated drugs was counted based on the claims database of JammNet Co., LTD., which receives data from health insurance societies in Japan. We compared the monthly average of the number of prescriptions in the all-case surveillance period with that in the spontaneous report period using Student’s t-test.

R version 3.6.0 (R Foundation for Statistical Computing, Vienna, Austria) was used for the analysis. Statistical significance was set at P < 0.05.

Results

Drug Characteristics

In total, 96 drugs with new active ingredients had been newly approved between 2010 and 2016 for which all-case surveillance was conducted. Among them, 35 drugs met the selection criteria and were included in the analysis. Table 1 enlists the characteristics of the investigated drugs (approval year, type of indication, and duration of the all-case surveillance). Details of the investigated drugs are provided in Supplementary Table 1. Drugs indicated for malignant tumors (including hematological malignancies) accounted for 87.5% of the drugs. For 19 drugs, the duration of the all-case surveillance period was < 1 year.

Table 1 Summary characteristics of the investigated drugs
Full size table

Descriptive Analysis

The monthly average number of sADR cases for the 35 drugs in both the all-case surveillance period and the spontaneous report period is reported in Supplementary Table 2. For all 35 investigated drugs, the monthly average number of all sADR cases was lower in the spontaneous report period than in the all-case surveillance period. Figure 2 depicts a histogram of the reduction rate of sADR cases for the investigated drugs. The median of the reduction rate was 78.9%.

Fig. 2
figure 2

Histogram of the reduction rate of sADR cases between the all-case surveillance period and spontaneous report period

Full size image

ITS Analysis

The results of the ITS analysis of all sADR cases for each investigated drug are shown in Table 2. The results of the ITS analysis with log-linear regression models for each investigated drug are reported in Supplementary Fig. 1. The coefficients of the level change (β2) for all 35 investigated drugs were negative. For 24 (68.6%) of the 35 investigated drugs, a statistically significant decrease in the level of all sADR cases between the all-case surveillance period and the spontaneous report period was observed. The results obtained for the coefficients of the slope change (β3) were not consistent, with negative values recorded for 18 drugs and positive values for 16 drugs. Figure 3 provides a histogram of the reduction rate of level in the ITS analysis of all sADR cases for each drug. The reduction rates of level for all investigated drugs were larger than 0%. The median (first quartile (Q1) − third quartile (Q3)) of the reduction rate in level was 60.1% (35.3–70.7%).

Table 2 Results of the ITS analysis of all sADR cases for each investigated drug
Full size table
Fig. 3
figure 3

Histogram of the reduction rate of level in the ITS analysis of all sADR cases for each drug

Full size image

Table 3 provides the summary results of the ITS analysis according to the different outcomes. The details of the ITS analysis according to outcome are reported in Supplementary Tables 3a, b. The number of drugs with a statistically significant decrease in the level of sADR cases with PT terms corresponding to the definition of the IME and the median of the reduction rate were 19/35 (54.3%) and 54.1%, respectively, which is similar to the result of the main outcome (all sADR cases). Conversely, when the outcome was sADR with death, the number of investigated drugs with a statistically significant decrease in the level was 6/35 (17.1%), which was lower than that of the main outcome.

Table 3 Summary of the results of the ITS analysis according to outcome
Full size table

Number of Prescriptions (Reference)

For the 34 investigated drugs, excluding Selexipag—data for which were not available—the number of prescriptions in the all-case surveillance period and the spontaneous report period and the results of the t test (P value) are presented in Supplementary Table 4. Furthermore, 2 of the 34 drugs showed a statistically significant increase in the number of prescriptions in the spontaneous report period compared with the all-case surveillance period. No drug showed a statistically significant decrease in the number of prescriptions.

Discussion

This study aimed to estimate the degree of under-reporting of ADRs in the post-marketing stage by comparing the number of sADR cases obtained in the all-case surveillance period with that recorded in the spontaneous report period in Japan. ITS analysis of all sADR cases revealed that 24 (68.6%) of the 35 investigated drugs showed a statistically significant decrease in the log-linear regression intercept (level) in the spontaneous reporting period compared with that in the all-case surveillance period. The median (Q1–Q3) of the reduction rate of the level was 60.1% (35.3–70.7%). This result indicates that the number of sADR cases reported in the spontaneous report period is lower than that reported in the all-case surveillance period. No statistically significant decrease in the number of prescriptions for all the investigated drugs (excluding one drug for which the data were not available) was observed in the spontaneous report period compared with the all-case surveillance period. These results suggest the existence of under-reporting of sADRs in spontaneous reports in Japan.

To the best of our knowledge, this was the first study to examine the existence of under-reporting using an ITS analysis. Moreover, as this study calculated the magnitude of under-reporting for all the investigated drugs using a single data set (JADER) and the same analytical method without extrapolating other data sources, a bias caused by differences in research methods did not occur. Considering that under-reporting was quantitatively observed in the spontaneous reporting systems of countries other than Japan in previous studies [5, 6], the results of this study support the view that a certain degree of under-reporting exists in spontaneous reports, regardless of the reporting system.

The median of the reduction rate of the level was 60.1% in this study, whereas previous studies indicated a median under-reporting rate > 90% [5]. As the reduction rate of the level is not equal to the under-reporting rate, it is difficult to compare these values directly. Nevertheless, for the following reasons, we considered that the under-reporting rate in this study was lower than that reported in previous studies. First, all drugs investigated in this study were subject to all-case surveillance. All-case surveillance is conducted for drugs for which there is a need to collect post-marketing safety and efficacy information early and comprehensively, such as when the drug was approved based on clinical trial data with a limited number of patients, or when there is great concern about the occurrence of sADRs [10]. Therefore, it is possible that medical doctors and healthcare professionals monitor the safety of these drugs more carefully and report their safety information spontaneously more frequently than other drugs. The second reason is a characteristic of the JADER, which was used in this study. JADER compiles data of serious ADRs exclusively. Some previous studies indicated that severe events were more likely to be reported compared with mild or trivial events [5, 6, 17, 18]. Third, we examined the existence of under-reporting at the end of the all-case surveillance period. For most of the investigated drugs, the end of the all-case surveillance period was within 3 years after their launch, which is a relatively early stage of marketing. For these reasons, the magnitude of under-reporting for the investigated drugs in this study tended to be lower; in contrast, the magnitude of under-reporting for other drugs, or that in the mid to late stage of marketing, may be high.

As a sub-analysis, we conducted an ITS analysis of sADR that corresponded to the IME. The result was similar to that obtained for the main outcome of all sADRs (Table 3) because most sADRs reported in JADER met the sADRs listed in IME. Conversely, the result of the ITS analysis of sADRs with death showed that both the number of drugs with statistically significant decreases in the level and the median of reduction rate in the level were lower than those obtained for the main outcome. Although the number of cases with sADRs leading to death was small, and despite the difficulty in evaluating the results from a statistical viewpoint, we concluded that sADRs with death were reported appropriately compared with other sADRs.

Although the case reports with high priority, such as sADRs leading to death, were reported appropriately, the existence of under-reporting for sADRs as a whole was confirmed. Therefore, there is scope for improvement in the process and system of spontaneous ADR reporting, including enhancing the accessibility to the reporting system and awareness of healthcare professionals of the program. Lopez-Gonzalez et al. pointed out that busyness, ignorance, and indifference to ADRs were the main reasons for their under-reporting [18]. Some intervention programs aimed at improving spontaneous reporting rates have been attempted [19,20,21,22]; however, large-scale implementation of these programs may impose a burden on healthcare professionals. Additionally, a considerable amount of time and effort is necessary to report high-quality ADR information in all instances. Therefore, we considered that the selective collection of ADR information according to the risk profile of each drug rather than collecting all ADR information will lead to improvements in under-reporting, e.g., preferential collection of ADR information for “important potential risks” and/or “important identified risks” identified in the Risk Management Plan.

There is a system for all-case surveillance in Japan in which under-reporting is considered to be low. However, the implementation of all-case surveillance requires enormous work from investigators and expenditure of substantial financial resources by pharmaceutical companies [9]. Furthermore, it was reported that the contribution of post-marketing surveillance, including all-case surveillance, to labeling changes was limited [23, 24]. Therefore, all-case surveillance should not be solely performed to address under-reporting.

This study had several limitations. First, the number of patients exposed to the investigated drugs was not considered in the ITS analysis. As these changes would affect the number of sADR cases, this may have led to bias in the analysis. Second, theoretically, there is no under-reporting during the implementation of all-case surveillance; however, some medical institutions may not be able to conduct all-case surveillances. Moreover, omissions by medical personnel are also a possibility. Third, the ITS analysis performed in the present study compared the information from different periods (all-case surveillance period vs. spontaneous report period); thus, the effects of factors affecting only one period could not be removed or adjusted. Fourth, most investigated drugs were for malignant tumors because we selected drugs for which all-case surveillance was conducted. Anticancer drugs have different safety profiles and are monitored more closely than other general drugs. Therefore, generalization of our findings to other indications would be limited. Fifth, the number of prescriptions in the spontaneous period was lower than that in the all-case surveillance period for 13 drugs. The impact was considered minimal because none of the decreases were statistically significant, but the magnitude of under-reporting may be smaller due to less use of these drugs.

Conclusion

This study demonstrated the existence of sADR under-reporting in spontaneous reports in Japan quantitatively. The under-reporting rate observed in this study was lower than that described in previous studies, and the degree of under-reporting of sADRs with death was lower than that detected for other sADRs, suggesting that sADR information was reported appropriately according to their level of severity.