1 Background

Pharmacovigilance involves the detection and evaluation of adverse drug reaction (ADR) ‘signals’, which may arise from any source (e.g. spontaneous reports, clinical trials, observational studies). However, in practice, a large proportion of ADR signals arise from spontaneous reports, and the Council for International Organizations of Medical Sciences (CIOMS) VIII definition of a signal is as follows:

“Information that arises from one or multiple sources (including observations and experiments), which suggests a new potentially causal association, or a new aspect of a known association, between an intervention and an event or set of related events, either adverse or beneficial, that is judged to be of sufficient likelihood to justify verificatory action.” [1]

Statistical disproportionality methods are now routinely used to aid the detection of signals from spontaneous reports by identifying drug–event combinations that are occurring more frequently than would be expected based on a background rate derived across the spontaneous dataset [2–6]. These methods detect large numbers of signals, some of which are real and require further evaluation and regulatory action, and some of which will turn out to not be true ADRs.

At the UK Medicines and Healthcare products Regulatory Agency (MHRA), signals from spontaneous reports are routinely detected with the aid of disproportionality analyses and then initially prioritised using the Impact Analysis method [2]. Impact Analysis is used as an interim step between signal detection and detailed signal evaluation and is designed to focus the detailed evaluation on those signals that are the strongest and most likely to impact on public health [7, 8]. Signals for which further action or evaluation is to be carried out following Impact Analysis and discussion at a weekly signal meeting are considered as potential pharmacovigilance issues. Following the identification of a new drug safety issue, the nature of the particular ADR will affect the urgency with which the evaluation and consequential action is carried out. At the MHRA, potential pharmacovigilance issues (arising from any source) are discussed at a separate weekly signal management meeting where, in addition to determining what regulatory action is required, a priority is determined with a timescale for which the action is to be completed. The meeting is attended by experienced medical and scientific pharmacovigilance staff, and decisions on action and priority are made by consensus. The MHRA signal detection and prioritisation process is summarised in Fig. 1.

Fig. 1
figure 1

Medicines and Healthcare products Regulatory Agency signal detection and prioritisation process. ADR adverse drug reaction, RPPS Regulatory Pharmacovigilance Prioritisation System

Full size image

During the development of integrated systems, a need was identified to develop a more robust and less subjective system of issue prioritisation to aid in the management of multiple dynamic issues ensuring that acceptable timelines are defined and resources are appropriately allocated to meet public health and other obligations (e.g. European obligations) of the MHRA. The aim of the system was to aid the prioritisation process, but with the members of the signal management meeting retaining responsibility for the final priority allocated. Impact Analysis was built around the principle that two broad factors should drive the resource put into signal evaluation, i.e. strength of evidence and potential public health implications. These are also the major drivers of pharmacovigilance issue management within the MHRA, but a variety of other factors may additionally be important, for example, public interest or concern about an issue, and regulatory, governmental and international obligations of an agency.

We therefore developed from first principles a quantitative system of issue prioritisation (Regulatory Pharmacovigilance Prioritisation System [RPPS]) that builds on the concepts of signal Impact Analysis but also includes other factors important to the prioritisation of drug safety issues and links the outcome to timelines by which regulatory action should be completed. In contrast to Impact Analysis, which is designed to triage signals arising from spontaneous reports, the RPPS is designed to also prioritise issues arising from any data source. The RPPS is not intended to replace Impact Analysis and is used at a different stage in the process. Figure 1 illustrates how the RPPS fits within the MHRA signal detection and prioritisation process. This paper describes the concept of the RPPS and the results of a pilot and implementation of the RPPS tool at the MHRA.

2 Methods

2.1 Development of the Regulatory Pharmacovigilance Prioritisation System (RPPS)

The RPPS was developed as a mathematical issue prioritisation tool based on information that falls into four broad categories that are considered as most relevant to the priority to which the MHRA gives specific pharmacovigilance issues:

  • Potential public health implications

  • Regulatory obligations

  • Strength of evidence for a causal effect

  • Public perceptions

An initial system was developed empirically with a scoring system that gave a greater weight to inputs in the potential public health implications and regulatory obligations categories. This initial system was piloted in the weekly signal management meeting at the MHRA that discusses potential pharmacovigilance issues and, through consensus, determines their priority and a timescale for action. So as not to bias the decision of the meeting, the RPPS scores were not presented until an initial view had been reached. The concordance between the RPPS priority and the priority decided upon in the meeting was investigated and found to be low. The results from this initial pilot indicated that the original scoring system that gave greater weight to all inputs in the potential public health implications and regulatory obligations categories was too simplistic and that some inputs from the other two categories (evidence and public perceptions) were more important for issue prioritisation than some of those in the public health and regulatory obligations categories. The RPPS scoring system was therefore modified on the basis of the initial pilot results and judgement of the members of the meeting as to which inputs were the most important, with the aim to achieve maximum concordance between the RPPS and the meeting results. Data from the initial pilot were used to ascertain which inputs were more frequently associated with a higher meeting-assigned priority. Additionally, the meeting members were asked to rank the inputs in order of importance. This information was then used to develop a system that weighted the inputs individually to achieve maximum concordance between the RPPS and the initial blinded meeting priority using the data collected from the initial pilot. Once the method was established, a purpose-built Microsoft® Excel (Microsoft Corporation, Washington, USA)-based program was designed to automate the process.

The finalised RPPS score determines the priority of an issue by considering four different inputs within each of the four broad categories above. The inputs included in the system were chosen to provide information on each of the broad categories using readily available data sources. The inputs are individually weighted according to importance and may contribute between 1–4 points to an overall category score. The final priority is determined by the sum of the four category scores, which translates into a priority of ‘standard’, ‘increased’ or ‘top’ priority.

2.1.1 Calculation of Scores and Overall Priority

The 16 different inputs are used within the RPPS to answer a series of questions to which the answer can be ‘yes’, ‘no’ or ‘unknown’. The questions are worded such that to answer ‘yes’ requires certain criteria to be fulfilled or a threshold to be reached. Thresholds and criteria used in these questions were derived from first principles and signal detection experience from the MHRA. The 16 inputs and the RPPS questions are described in Table 1.

Table 1 Inputs, criteria and number of points associated with a positive response for the Regulatory Pharmacovigilance Prioritisation System
Full size table

If an input qualifies the answer of ‘yes’, a corresponding number of points (between 1–4) is awarded. An answer of ‘no’ or ‘don’t know’ is not awarded points. The number of points awarded is pre-determined for each individual input and was determined based on the results from the initial RPPS pilot. Data collected during the pilot of the initial system were used to determine which inputs are generally considered to be the most important by the experienced members of the signal management meeting. The scoring system was then developed with the aim of maximum concordance with the meeting. Details of the points attributed to each of the 16 inputs are given in Table 1.

An overall score for the issue is calculated from the sum of all the points from each input (where awarded). An additional criterion outside the 16 inputs is also considered at this stage: whether the issue has previously been identified and whether the proposed action is likely to change prescribing practice. This additional criterion was included to prevent ‘administrative’ signals being treated as higher priority issues than potentially new pharmacovigilance issues. This type of issue is often to align the product information for a generic product with that of the brand leader for a known ADR. If this additional criterion is met, the overall score is halved. The rationale for this is that issues where the ADR is well known, and the action is not therefore likely to change prescribing practice, are likely to score highly using the RPPS because many of the inputs will qualify for a ‘yes’ answer, particularly in the strength of evidence category. However, they are not issues that should warrant a higher priority, as they generally concern ensuring consistency of information rather than resulting in new or changed advice.

According to the final overall score, the final priority for the issue is determined as follows:

  • 0–5 points: standard priority

  • 6–13 points: increased priority

  • ≥14 points: top priority

To facilitate management and audit, the priority determined by the RPPS is linked to a maximum temporal target to reach a pre-defined end-point (e.g. regulatory action, communication, etc.), bearing in mind the need for interim steps (e.g. further evaluation and seeking the advice of expert committees during the process). The time targets in relation to the set priorities were as follows:

  • standard priority: 12 months

  • increased priority: 6 months

  • top priority: 3 months

An example of the output from the RPPS Excel program for an issue with an increased priority is given in Fig. 2.

Fig. 2
figure 2

Regulatory Pharmacovigilance Prioritisation System (RPPS) program output: finasteride and male breast cancer

Full size image

2.2 Pilot Study

A pilot study of the modified RPPS was carried out over 10 months between August 2007 and May 2008 at the MHRA. The aim of the pilot was to investigate the performance of the RPPS against the opinion of members of the weekly pharmacovigilance signal management meeting that discusses issues and determines their priority and a timescale for action.

Over the 10-month study period, all pharmacovigilance issues that arose were prioritised using the modified RPPS tool and taken to the pharmacovigilance issue meeting the following week. Following discussion of the issue and blinded to the RPPS priority, the members of the meeting were asked to give the issue a priority using their collective judgement and on the basis of the discussion as normal. The results of the RPPS priority were then presented and a final decision on the priority was then made. The three priorities (initial meeting priority, RPPS priority and the final priority) were recorded for each issue.

Kappa statistics were used to assess the levels of agreement between the three priorities, with the results of the kappa analysis categorised into the groups proposed by Landis and Koch [10].

2.3 Post-Implementation Evaluation

Following the results of the pilot study, the decision was made to implement the RPPS into routine use for issue prioritisation at the MHRA. All new pharmacovigilance issues were prioritised using the RPPS prior to the weekly meeting in which the issue and the RPPS priority were discussed. However, the RPPS priority was intended to be a guide, and therefore the priority may have been changed by consensus at the meeting.

The performance of the RPPS once integrated into routine issue management at the MHRA was investigated over the first year of implementation (July 2008–June 2009). Data on the RPPS and final issue priorities were collected over the 1-year period and analysed to verify whether the level of agreement between the RPPS and the meeting observed in the pilot remained.

3 Results

3.1 Pilot Study Results

A total of 73 pharmacovigilance issues were prioritised during the study period and included in the pilot study. The distribution of the RPPS, meeting and final priorities for the 73 issues is shown in Fig. 3.

Fig. 3
figure 3

Distribution of priorities for the 73 issues included in the pilot study. RPPS Regulatory Pharmacovigilance Prioritisation System

Full size image

The overall exact agreement between the RPPS priority and the collective judgement of the meeting was 60.3 % of all issues prioritised. The weighted kappa measure of agreement between the RPPS priority and the meeting priority was 0.31, which indicates a fair agreement. Where exact agreement was not obtained (n = 29), the RPPS generally prioritised the issues slightly higher than the meeting, with 19 of the 29 issues prioritised higher by the RPPS. However, the differences were only one category apart, with no two-category differences observed. These results are shown in Table 2.

Table 2 Pilot study results (August 2007–May 2008): concordance between the priority assigned by the Regulatory Pharmacovigilance Prioritisation System and the initial collective meeting judgement (n)
Full size table

A slightly higher agreement between the RPPS and the final priority (64.4 % exact agreement) was observed, indicating that subsequent knowledge of the RPPS priority, and the rationale behind it, had some influence over the final agreed priority.

3.2 Post-Implementation Evaluation

A total of 128 issues were prioritised over the first year of implementation. The distribution of the RPPS and final priorities is shown in Fig. 4.

Fig. 4
figure 4

Distribution of priorities for the 128 issues prioritised post implementation. RPPS Regulatory Pharmacovigilance Prioritisation System

Full size image

An overall exact agreement between the RPPS priority and the final agreed priority of 82.8 % was observed for all issues prioritised. The weighted kappa measure of agreement between the RPPS priority and the final priority was 0.69, which indicates a good agreement as defined by Landis and Koch [10]. Similarly to the pilot, where exact agreement was not obtained (n = 22), the RPPS generally prioritised the issues slightly higher than the meeting, with 12 of the 22 issues prioritised higher with the RPPS. The majority of differences were one category apart, with only one issue two categories apart. These results are shown in Table 3.

Table 3 Post-implementation evaluation (July 2008–June 2009): concordance between the priority assigned by the Regulatory Pharmacovigilance Prioritisation System and the final meeting priority (n)
Full size table

4 Discussion

The RPPS tool was developed to aid the management of multiple dynamic pharmacovigilance issues that arise from any data source. The aim of the RPPS is to provide a systematic and reproducible method for issue prioritisation that takes into account information considered to be the most important for prioritising pharmacovigilance issues.

The RPPS tool was designed initially using the Impact Analysis tool as a basis but including further criteria considered to be important when allocating timescales and resource to pharmacovigilance issues, i.e. agency regulatory obligations and public perceptions. The scoring system was developed according to early pilot results and subsequently modified on the basis of experience.

The pilot study investigating the use of the RPPS at the MHRA demonstrated that the system had good agreement with the collective judgement of the team of physicians and scientists, and where exact agreement was not obtained, the RPPS generally prioritised issues higher. This is encouraging, as it is generally preferable to have a system that is slightly over-cautious than one that risks under-prioritising issues. The RPPS is based on 16 inputs considered to be the most important for prioritising pharmacovigilance issues. These are not an exhaustive list but provide a framework for decision making that is considered at the meeting, with the meeting retaining overall responsibility for the final priority. Areas of discordance were due to additional factors brought up by experienced assessors at the meeting that are not considered by the RPPS. A limitation of the RPPS and any system of this kind will be that it can only consider a finite number of criteria, and the discordant results highlight the need to retain the final responsibility for signal prioritisation with the meeting rather than moving to a fully automated system.

The further analysis carried out 1 year following the implementation of the RPPS achieved a higher level of agreement with the final priority assigned to each issue, indicating that generally the meeting agrees with the RPPS priority and does not often change it. Again, where an exact match was not achieved, the RPPS generally prioritised issues higher than the final agreed priority. This increased agreement with RPPS post-implementation is not unexpected as, during the pilot study meeting, attendees were asked for their priority prior to revealing the RPPS result, whereas post-implementation, the RPPS result is presented first. People are likely to require more reason to change their initial opinion once given than to concur with a priority presented to them first. The results from the pilot study and post-implementation investigation (Figs. 3, 4) also show that over the two study periods, the distribution of RPPS priorities is very similar, with around 39 % of issues prioritised as ‘Standard’, 54 % as ‘Increased’ and 6 % as ‘Top’. Such similar results obtained over two different study periods demonstrates that the RPPS is consistent in its approach to issue prioritisation. Overall, the results of both the pilot study and the post-implementation investigation show that the RPPS provides a consistent, systematic approach to issue prioritisation that removes some of the subjectivity of reliance on the judgement of members of a meeting.

Whilst the RPPS was developed specifically for use at the MHRA, there is potential for it to be adapted for use by other regulatory authorities for issue prioritisation with minimal modification. For example, Table 1 describes the inputs required by the RPPS for a UK setting but these could easily be modified for another country/region. Adaptation for use by pharmaceutical companies would require more extensive modification as some of the criteria are not applicable to industry (i.e. the regulatory obligations criteria), but the general concept could be utilised and a set of industry obligations could be developed to be used instead.

Limitations of the RPPS include biases inherent with spontaneous ADR data, as some of the criteria rely on spontaneous data, which may result in falsely elevated or decreased evidence or public health scores. Although the RPPS relies on 16 different criteria and has been developed to include information considered to be the most important for issue prioritisation, it cannot include all information relevant for every potential issue and therefore can never completely replace the need for review by scientific and medical assessors. For this reason, the RPPS is intended to support and guide judgement on issue prioritisation rather than provide a definitive answer.

We did not specifically analyse the impact of scoring ‘yes’ or ‘don’t know’ during the pilot study. As the inputs are weighted differently, the impact will be greater for those inputs with a higher attributed weight, e.g. the impact of scoring ‘yes’ for health consequences will be greater than the impact of scoring ‘yes’ for the total number of spontaneous reports. The intention was that the data required for the RPPS would be reasonably easy to obtain and therefore each RPPS assessment would not take long to perform. The majority of data required will be readily available from regulatory/drug utilization databases, with some inputs potentially requiring a simple literature search (e.g. data sources, biological plausibility). It is therefore envisaged that the number of inputs scoring ‘don’t know’ will be very low. Indeed, during the pilot study, all answers were either ‘yes’ or ‘no’ with no occurrences of ‘don’t know’.

5 Conclusion

The RPPS provides a systematic, consistent approach to pharmacovigilance issue prioritisation in a regulatory setting to enable resource to be allocated within an appropriate timescale. The method removes some of the reliance on subjective decision making for prioritising issues and provides a valuable audit trail. The RPPS has the potential to be adapted for use within other regulatory agencies and the pharmaceutical industry.