FormalPara Key Points for Decision Makers

There is a high level of redaction in current National Institute for Health and Care Excellence (NICE) single technology appraisal (STA) guidance, both for academic and commercial reasons, and this may also be increasing over time.

The level of redaction present is variable between STAs, in many cases leading to decisions that do not appear justified, and in others, inadvertently revealing supposedly confidential discounts.

An improvement is needed to the status quo to ensure appropriate redaction is applied so as not to undermine the transparency of health technology appraisal.

1 Introduction

When health technology assessment (HTA) agencies such as the National Institute for Health and Care Excellence (NICE) assess the cost-effectiveness of new interventions, the incremental cost-effectiveness ratio (ICER) (or cost per quality-adjusted life year [QALY] gained) plays a pivotal role. For interventions associated with an ICER greater than the willingness-to-pay threshold (λ) specified by NICE, it is unlikely that NICE would issue a positive recommendation. This is because an ICER higher than λ implies a displacement of treatments or services within the National Health Service (NHS) that offer equivalent (or better) value for money, and would result in a decrease in population health.

Manufacturers of new interventions faced with this situation have two distinct options to consider (outside of revisiting the evidence and/or undergoing an appeal). One of these is to simply walk away, accept a negative recommendation by NICE and therefore forfeit the prospect of securing patient access via routine commissioning. The other is to propose a commercial arrangement between the NHS and the company to allow NICE to arrive at a positive recommendation—in other words, provide the intervention at a reduced price such that the ICER may fall below λ.

A commercial arrangement (or Patient Access Scheme [PAS]) will be proposed by a company for consideration by the Patient Access Scheme Liaison Unit (PASLU) at NICE [1]. The nature of the PAS may be public knowledge or confidential—the most common of which is a (confidential) simple percentage discount on the list price of the intervention [2]. The confidentiality of a simple PAS discount may provide an attractive option for companies seeking to procure patient access while also protecting the nature of any commercial agreements made in order to do so.

Ultimately, the final decision made by NICE (i.e. a positive or negative recommendation issued by the committee) is publicly available. Within the NICE guides to the methods and processes of technology appraisal, transparency is described as essential to ensure appropriate and robust decision making [3, 4]. The guide to the methods of technology appraisal [3] states:

The credibility of the guidance produced by the Institute is dependent on the transparency of the Appraisal Committee’s decision-making process. It is crucial that the Appraisal Committee’s decisions are explained clearly with reference to all the available evidence, and that the contributions of clinical specialists, commissioning experts, patient experts and the views of people who responded to consultation during the appraisal are considered. The reasoning for the Committee’s decision will be explained, with reference to the factors that have been taken into account, in the ‘Considerations’ section of the guidance.

Because many factors are taken into account when the committee makes a decision, it is possible that some factors bring benefits that outweigh other barriers. While a λ of £20,000 per QALY gained may have originally been set to maximise health (i.e. ensure that the benefits of a new technology outweigh the opportunity costs), NICE has been explicit in its tolerance of higher values of λ when other factors are evident. For instance, when considering ‘end-of-life’ therapies, NICE has been willing to issue positive recommendations when the ICER clearly exceeded the estimated opportunity cost (i.e. λ of £20,000 per QALY gained) [5]. This reflects the ‘social value judgement’ that end-of-life treatment might be more valued than other care. In effect, it means that we (society) may be willing to accept a lower aggregated ‘population’ health in order to see end-of-life given priority. The important point, though, is that we (society) should be able to fully understand the trade-offs that are being made on our behalf.

If ICERs used to inform decision making (i.e. those including the confidential PAS discount) are withheld from the public due to commercial reasons, then the population is being asked to give up an unspecified amount of health. In such cases, it is impossible for this to be a truly informed social value judgement, at least from the perspective of those outside committee membership. Therefore, while members of the public may not always agree with decisions made by NICE, it is important that the public is able to understand the rationale behind these decisions. As such, there is a clear conflict of interest for NICE to make decisions transparently while also protecting the confidentiality of PAS discounts (and thus be able to recommend treatments which offer value for money).

The objective of this study was to investigate how NICE has balanced these two ideals when providing evidence used to support its decision making. A review of all recommendations made by NICE that included a confidential, simple PAS discount was carried out to identify instances of reduced transparency (and thus decision making that appears ill-founded in relation to the presented evidence) as well as any documentation wherein confidential information may be back-calculated or estimated based on the evidence presented. By performing this study, the subtleties associated with balancing conflicting principles may be better understood and recommendations made to improve the transparency of decision making.

2 Methods

Guidance for technologies assessed by NICE are published on the NICE website (http://www.nice.org.uk), which includes guidance from the single technology appraisal (STA), multiple technology appraisal (MTA), and highly specialised technology (HST) appraisal programmes (in addition to other types of guidance relating to medical devices, surgical procedures and diagnostic tools). While the majority of documents are publicly available, parts of these are redacted owing to the incorporation of academically or commercially sensitive information—the latter of which is typically due to the specification of a commercial arrangement (or PAS).

In addition to the guidance issued for NICE-assessed technologies, a list of all medicines approved with a PAS is also available on the NICE website [2]. Approved technologies including a commercial arrangement were extracted from the NICE website, and categorised by type of commercial arrangement. The key types of commercial arrangement permitted by NICE include simple discounts (i.e. a straightforward percentage discount), complex discounts (e.g. including a limited amount of free stock, or outcomes-based dose caps) or commercial access agreements (e.g. when recommended for use within the Cancer Drugs Fund while further data are collected).

The primary focus of our study was to consider those technologies with an approved, confidential, simple PAS discount. Consequently, technologies with an approved complex PAS discount scheme or commercial access agreement were excluded.

Technologies with an approved simple PAS may have been assessed via the STA, MTA or HST programmes, though only STA guidance was considered within the study. HST guidance was excluded due to differences in the decision-making process versus the STA or MTA programmes, and the small number of completed HST appraisals. Though MTA guidance may face many of the same issues regarding transparency following redaction of key information, MTAs have been excluded from this analysis for several reasons. Companies are not required to submit an economic model, and so there may be cases where sensitive information is not redacted because it is simply not present within the company’s submission. In the Evidence Review Group’s (ERG’s) report, results are presented for all treatments under assessment, and so if only results for one intervention are commercially sensitive, the whole set of results may require redaction, and so it is not possible to easily determine the reason(s) why results are redacted. Furthermore, there have been many more STAs in recent NICE history versus MTAs, and so an analysis purely of STAs was considered sufficient for the purposes of our study.

After applying the aforementioned exclusion criteria, the total number of completed STAs with an approved simple PAS discount was identified. For these appraisals, the published Final Appraisal Determination (FAD) document was downloaded, which contained the evidence presented by NICE from which the final decision was made. The NICE website provides a range of documentation published throughout the time period over which a given technology is appraised, including company submissions, ERG reports. etc. However, these interim documents reflect the committee’s preferred assumptions and results at the time of publication, which may contradict those in the final published guidance. Consequently, only the cost-effectiveness results presented within the FAD may be considered to form the basis of the final decision made by NICE.

Subsequently, we searched the FAD to identify the cost-effectiveness results (i.e. the ICER and/or its component parts) from which the final decision was made. Where a definitive ICER was presented, we searched the most recent interim documentation to ascertain the nature of redaction presented to protect commercially sensitive information (e.g. the slides presented at the committee meeting, or the most recent ERG report). For appraisals where the FAD did not contain a definitive ICER, company submission documentation (including results with the PAS discount) were searched to establish the nature of redaction presented. Redaction methods were categorised according to the following:

  • Unreported: the FAD does not contain any ICER, or its component parts.

  • Range: the FAD states that the ICER was above, below or between given value(s).

  • Reported: the FAD provides a single ICER from which the final decision was based.

Next, we investigated whether the conclusion reached by NICE appeared justified in light of the evidence presented in the FAD. The conclusion reached was deemed justified if the ICER(s) presented were less than or equal to the specified λ, or if NICE stated that while the ICER was greater than λ, the treatment was recommended on the basis of other social value judgements. The specification of a decision as ‘justified’ or ‘unjustified’ is purely made in relation to the presentation of the ICER and not whether the decision was correct or incorrect.

Academically sensitive evidence may be provided by companies to aid committee decision making. However, presentation of such within the public domain may prejudice future publication of the information in a scientific publication [6]. In public committee meetings, these data are presented on the slides shown to the public, but are omitted from paper/online versions of the meeting slides and other documentation (e.g. the FAD). The agreement between the Association of the British Pharmaceutical Industry (ABPI) and NICE states that for unpublished clinical trial data, a minimum period of 12 months from publication of the clinical study report (CSR) is considered appropriate over which data are redacted [6].

While redaction of these data is of clear importance while awaiting publication, if these data are never revealed within NICE documentation (except for those in attendance of the public committee meetings), the transparency of past decision making is compromised. With this in mind, we searched company submissions for instances of academic-in-confidence redaction. It was not possible to ascertain dates of CSR publication for each included trial, so the aforementioned minimum period of 12 months could not be directly applied. Instead, we report the number of STAs which include academic-in-confidence redaction which were published at least 12 months ahead of the extraction date—thus guaranteeing a minimum of 12 months had elapsed, but more likely a minimum of 18 months (given the length of the NICE process).

For those STAs including a simple PAS discount, we attempted to back-calculate the volume of discount based on the information presented within the public domain. STA documentation was searched to ascertain the methods used to redact commercially sensitive information. If this clearly prohibited estimation of the PAS (e.g. complete removal of all cost-effectiveness results), no further attempts were made at estimating the PAS. However, if enough information was available (e.g. disaggregated total costs), back-calculation was attempted. In the interest of keeping PAS discounts confidential, we do not refer to specific STAs in our results.

Given the confidential nature of the ‘true’ volume of discount, it was not possible to verify whether the estimated volume of PAS discount was correct. However, the method of estimation was verified by another reviewer to check the approach used was logically sound. In addition, it was not possible to adopt a fully systematic approach to estimate the PAS associated with each product, due to differences in reporting, redaction, and estimation methods required. As such, the objective of this aspect of the study was to establish whether or not back-calculation of the PAS discount was possible for a non-zero number of STAs.

3 Results

A list of all published NICE technology appraisal guidance was downloaded from the NICE website on 26 October 2018. The identification of STAs relevant to this study is presented in the form of a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram (Fig. 1).

Fig. 1
figure 1

PRISMA diagram for inclusion of relevant NICE STAs. Relevant appraisals were identified via available material from the NICE website. It was noted that some documentation was contradictory (e.g. the NICE summary of decisions downloadable Excel file stated that some MTAs were STAs, and vice versa). All excluded MTAs were hand searched to ensure no STA was inadvertently excluded. HST highly specialised technology, MTA multiple technology appraisal, NICE National Institute for Health and Care Excellence, PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses, STA single technology appraisal

Full size image

Prior to exclusion of appraisals, a total of 118 products were found to have an approved commercial arrangement. The approved commercial arrangements were used to inform 171 distinct STAs, MTAs and HST appraisals. From these, a total of 110 STAs with an approved simple PAS discount were identified, of which 62 (56.36%) were conducted in cancer. Of the remaining 48, the most common disease areas for which simple PAS discounts were used were in ophthalmology (n = 10), dermatology (n = 9), and rheumatology (n = 8). A summary of the included STAs by disease area is provided in Fig. 2.

Fig. 2
figure 2

Overview of identified single technology appraisals (STAs) by therapeutic area

Full size image

For each of the relevant STAs, the FAD was searched to ascertain the transparency of the recommendation issued. In 24 STAs (21.8%), no ICER was presented within the FAD—this illustrates that a substantial proportion of recommendations made for products including a PAS discount lack the necessary transparency such that the decision made is well founded. In 66 (60.0%), an ICER range was presented (i.e. the ICER was above, below or between given value[s]). While an improvement on presenting no ICER at all, statement of an ICER range precludes the understanding of the committee-preferred assumptions or scenarios (and thus ultimate results) that factored into their decision making. In the remaining 20 STAs (18.2%), a final ICER was presented including the PAS from which the decision was made.

For STAs which reported either an ICER or ICER range (n = 86 STAs, 78.2%), the decision was considered justified by the cost-effectiveness evidence presented (and any associated social value judgments). For these STAs, the company submission was searched to ascertain the methods used to redact commercially sensitive information (i.e. the volume of discount). Most submissions redacted the component figures that contribute to the ICER (i.e. the incremental costs and QALYs). However, in a small number of cases, full results were presented including the PAS discounts, with results excluding the PAS discount fully redacted. Presentation of results in this manner offered the greatest transparency.

It was noted that even between submissions by the same company, redaction methods may differ. Two examples of this are presented in Table 1. In the two submissions by Pfizer, one showed all the component costs, QALYs and life years (LYs) as redacted whilst in the other submission, ICERs were presented. In the two submissions by Novartis, the ICERs including the PAS were presented; however, only in one of the submissions were the component QALYs and LYs provided. These examples demonstrate the presence of excessive redaction, as LYs do not form a part of the ICER calculation, yet are hidden from public view.

Table 1 Alternative methods of redaction submitted by companies
Full size table

For those STAs where the decision did not appear justified (because no ICER or ICER range was presented, n = 24, 21.8%), the date of FAD publication was used to establish whether there was a difference in the justification of the decision made over time. Of the 24 STAs considered to exhibit ‘unjustified’ decisions, 21 were published in the past 3 years.

To ascertain the extent of potentially unnecessary academic redaction, STAs published before 2018 were searched. Nearly all of these STAs (n = 68 out of 86, 79.1%) included the redaction of some academically sensitive information—usually data were redacted from the latest trial cut-off, or from the CSR (not in the public domain). As these data were redacted, it is not possible to determine whether they could also contribute to the protection of the commercially sensitive discount. However, there were some clear instances where the redaction appeared important in understanding the clinical context (for example, relating to the proportion of patients expected to remain on treatment over time).

For the STAs including a simple PAS discount (n = 110), back-calculation of the PAS discount was attempted. We were able to back-calculate the PAS discount in at least seven cases. In some cases, the methods used were relatively simple (e.g. comparing between tables to cross reference missing numbers). In other cases, the methods required partial rebuilding of a model. It is likely that a larger number of PAS discounts could have been estimated by essentially rebuilding the submitted cost-effectiveness models; however, this was beyond the scope of our study, which was to look at easily back-calculable discounts.

4 Discussion

The results of our review demonstrate that at present censoring appears to be performed on an ad hoc basis with no consistent pattern in the information censored. This applies not just between companies, but also between submissions from the same companies. The approach of censoring all (or at least the majority of) cost-effectiveness results leads to reduced transparency, with few decisions appearing justifiable in light of the information presented. However, the ad hoc approach if implemented poorly does not even necessarily keep discounts confidential (our cursory analysis was able to estimate the PAS in a non-trivial number of cases).

Of particular concern was the number of appraisals where the decision was classified as ‘unjustified’ (n = 24, where no ICER or ICER range was reported in the FAD). Within these STAs, the majority, if not all of the outcome information is censored (costs, QALYs and ICERs). This raises a number of issues and greatly undermines the legitimacy of the HTA process—in many instances even LYs (which are completely unrelated to PAS discounts) were censored. Based on the lack of information presented in cases such as these, a patient would not be able to understand the evidence presented regarding their likely outcome with treatment. Conversely, patients asked to forgo treatments would not be able to understand why a given technology is funded in preference to the one they would desire. In the latter case, we feel it is particularly important that patients may comprehend how trade-offs have been made relating to aggregated population health. Worryingly, the majority (n = 21, 87.5%) of ‘unjustified’ decisions identified by this review were published in the last 3 years, which we hope does not indicate a trend.

A secondary finding was that the volume of data censored as academic-in-confidence is extremely high. Given the length of time since decisions were published (some as much as 10 years ago), it is unlikely that the majority of academically sensitive information remains as such—indeed most should be published within a year of CSR publication (and consequently, less than a year after NICE documents are published online). Despite this, however, there exists no step (such as embargo for a year, as seen with some academic publications) by which academic-in-confidence information is uncensored in the NICE process—this ideally would be rectified when the NICE methods guide is next updated (if not sooner). An example of such data can be seen in NICE TA552, which includes large quantities of redacted clinical data, including baseline patient characteristics from the pivotal clinical study [7]. The table of characteristics with partial redaction (dated July 2018 in the NICE committee papers) was subsequently published (in full) in September 2018, while the FAD was published in November 2018 [8, 9]. Nevertheless, these data will remain redacted on the NICE website in perpetuity under the current process.

Our review has some limitations, in that we did not include MTAs, assessments where more than one treatment was subject to a PAS, or assessments where a complex PAS was applied. Whilst those excluded studies may have proven to be more difficult to back-calculate the PAS from available information, it would not affect the issues raised around transparency in decision making.

We also note that several of the findings of our review may be useful more broadly within the HTA field—for example, how redaction may inadvertently reveal commercially sensitive information if performed incorrectly. NICE highlight the importance of transparency within their methods guide, though this sentiment may not be considered as important for other markets in light of the role pricing negotiations may have on the decisions made. Ozierański et al., (2019) recently published a summary of the transparency of decisions made by the Polish Agency for Health Technology Assessment (AHTAPol) [10]. The authors of this study found that of the 332 assessments they assessed, the relationship between costs, health effects and the threshold price was redacted in 290 cases (87.3%). However, unlike our findings, the proportion of AHTAPol assessments that were redacted appears to be decreasing on a per-year basis (100.0% of those conducted in 2012, versus 76.7% in 2015).

5 Recommendations and Conclusions

Based on the results of our review, there are multiple ways in which redaction could be performed; none, however, by definition, offer both perfect transparency and the ability to maintain the confidentiality of PAS discounts. We recommend a balanced approach be undertaken (where possible) by censoring ‘without PAS’ results and not providing fully disaggregated costs for the ‘with PAS’ results, such that the final ICER (including the PAS) may be reported in the public domain. This approach constitutes an improvement on many previous appraisals for which the decision was either completely opaque or for which the PAS may be inadvertently ‘back-calculated’. However, it is noted that this may not be possible in cases where a PAS discount is introduced mid-way through the NICE process.

Given the findings of our work we would suggest that the current approach to redaction needs refining, and potentially codifying. Firstly, some level of standardisation should be applied to commercially sensitive discounts, to ensure that transparency is maintained as much as possible (with particular reference to specific approaches, per our suggestion above). Secondly, we believe the embargo period specified for academic-in-confidence data should be adhered to, in order to ensure information is released when no longer academically sensitive. A 12-month period following publication of final guidance on the NICE website over which information remains confidential would seem sufficient to ensure any material likely to be published has been (this in reality would mean the data were first generated at least 2 years before being made publicly available, and more likely 3 years).

We are aware that adoption of these recommendations will have some practical challenges. For example, a non-redacted document may be made public before a PAS is proposed. Retrospectively redacting published documents would clearly be inappropriate, as these documents may have been downloaded previously. Likewise, it is commonplace for committees to conclude that there is no ‘base case’ ICER, or even a range of plausible ICERs, if they consider the source data and/or modelling to be too uncertain for decision making. However, the inconsistencies that have been highlighted in this paper demonstrate the need for serious discussion on this topic and for clear guidance to be set out providing a system for determining which aspects of the supporting evidence for STA decisions should be considered commercially or academically sensitive; perhaps noting that a revision to the appraisal process may be necessary in order to enforce such guidance (for example, NICE could potentially redact all cost-effectiveness results from the public domain until final guidance is issued).

It may not be possible (without total censoring) to allow a PAS to be completely hidden, yet transparency is vital for the validity of HTA as an institution; thus transparency in evidence submitted by companies should be encouraged where possible. Updated NICE guidance on this topic may improve overall transparency and prevent the inadvertent disclosure of commercially sensitive information.