Introduction

The success of kidney transplantation has resulted in a steady expansion of the number of patients on the transplant waiting list. Despite recent increases, the number of deceased donors has failed to keep pace with this growing need for transplantation and there are nearly 100,000 patients on the Organ Procurement and Transplantation Network (OPTN) kidney transplant waiting list. This has resulted in long waiting times for potential transplant recipients. These extended waiting times for kidney transplantation challenge the ability of transplant centers to provide attainable therapy for a large numbers of the patients with end-stage kidney disease.

The efforts of transplant programs to provide medically appropriate transplantation to a broad range of recipients occurs in an environment of intense regulatory oversight by the Centers for Medicare and Medicaid Services (CMS) and the OPTN. Currently, routine program evaluation is based on 1-year post-transplant patient and allograft survival. This data is collected by the OPTN and analyzed by the Scientific Registry of Transplant Recipients (SRTR). The SRTR produces publicly available semiannual program-specific reports (PSRs) which provide outcome assessments based on 2.5 year cohorts of recipients. These outcomes are risk adjusted by the SRTR to reflect the expected outcomes based on the clinical characteristics of a transplant center’s recipient population and those of the donor organs transplanted. These data are reviewed by the OPTN Membership and Professional Standards Committee (MPSC) and programs falling below defined thresholds for observed to expected outcomes generally enter into structured performance improvement programs. These same data compared to a different set of criteria for expected outcomes are also utilized by CMS to determine compliance with CMS Conditions of Participation (COP). Failure to meet CMS outcome criteria can result in loss of Medicare reimbursement for all programs of a hospital.

The OPTN has a strategic goal of increasing the number of transplants in the USA and to begin to close the gap between the demand for transplantation and the limited supply of organs available for transplantation. The ability to significantly increase the number of deceased donor transplants in the USA likely lies in developing strategies to expand the effective utilization of organs from less than ideal donors and to define populations of patients that would derive benefit from these perceived higher-risk transplant procedures [1•].

Influence of the Current Monitoring System on Clinical Decision Making

Transplant centers have long argued the current regulatory environment, and the metrics used by the OPTN and CMS to assess performance have resulted in risk-averse clinical decision making that limits patients’ access to the transplant waiting list as well as constraining the choice of donor organ programs are willing to consider for transplantation. The relatively small difference between risk-adjusted expected outcomes and decrements in measured performance that can result in regulatory flagging is viewed by many as a risk to transplant program viability and patient access to transplantation. Deceased donor kidneys are allocated in part based on the estimated duration of allograft function post-transplant as measured by the kidney donor profile index (KDPI). It has been pointed out by the SRTR that the risk adjustment models used for program assessment effectively capture the risks associated with the transplantation of donor kidneys with a high KDPI, that is kidneys expected to function for a shorter period of time, and that programs transplanting higher-KDPI kidneys are not at increased risk for low performance evaluations compared to programs that transplant fewer high-KDPI kidneys [2]. Despite these assurances, others have noted that the current models explain only a limited portion of the observed graft failures as shown by the model C-statistic of about 0.65 [3]. It is felt by many that the models lack important pieces of information that impact graft survival and could be improved [4]. Analysis by CMS, using SRTR data, has shown that many low performing adult kidney programs when entered into System Improvement Agreements or approved for mitigating factors by CMS subsequently have significant improvements in their standardized mortality ratio for 1-year patient survival [5••]. Volume in these programs, however, tended to decline while national volume increased. Patients listed in those programs may temporarily have less access as programs reduce volume to improve outcomes.

The potential limitation of patient access to transplantation resulting from risk-averse program behavior has been extensively discussed in the transplant community [6, 7]. Schold et al. found that between 2007 and 2009, 46 of 205 adult kidney transplant centers found to have statistically significantly lower than expected 1-year patient or graft survival had transplant volume decline by a mean of 22 cases while other centers increased transplant numbers [8•]. More recently the same group found an increase in waitlist removals and a decreased transplant rate in centers with low performance evaluations [9•]. White and colleagues reported a 38% decline in transplant volume and a 55% decrease in the use of higher KDPI organs in programs with ongoing noncompliance with CMS COPs [10•].

It has been argued from a systems’ perspective that the impact of low performance evaluations has minimal overall impact as reduced transplant volumes at low-performing centers are compensated for by increased transplant volumes at other centers and that low-performing centers have subsequently improve performance which is the desired outcome [5••]. In contrast, it has also been argued that the regulatory pressure for constantly improving results, with risk-adjusted expected 1-year patient survival of 98% and risk-adjusted expected 1-year allograft survival of 95% for many programs has resulted in loss of appreciation and disregard for the enormous survival advantage of kidney transplantation compared to remaining on dialysis. In a retrospective cohort study of adults wait-listed for kidney transplantation in the USA from 2003 to 2010, Schold et al. investigated whether measured transplant center performance modified the survival benefit of transplantation versus dialysis [11••]. Deceased donor transplantation was independently associated with lower mortality compared to remaining on the waiting list at all levels of transplant center performance. Hazard ratios favoring survival with transplantation ranged from 0.24 at the highest performing centers to 0.40 at the lowest performing centers. This demonstrated that the survival benefit of transplantation remained highly significant even at centers with, by current metrics, low-post-transplant outcomes. The investigators concluded that policies that emphasize improved center performance should concurrently address issues of access to transplantation for a broader group of patients in an attempt to improve end-stage renal disease population outcomes.

The effective use of high-KDPI kidneys remains a challenge despite clear evidence that these organs can provide a survival advantage for appropriately selected recipients. Approximately 20% of all the kidneys procured for transplantation are never transplanted and are discarded, with organ quality being the most commonly cited reason [12]. The rate of discard rises steeply as KDPI increases beyond 70 with discard rates over 50% for the highest-KDPI kidneys. Based on KDPI data only, many of these organs would be expected to provide good functional outcomes. Massie et al. evaluated the mortality risk associated with increasing KDPI values over 70 in first time adult registrants adjusted for candidate characteristics and found that high-KDPI transplantation was associated with increased short-term but decreased long-term mortality risk [13•]. The survival benefit of transplanting high-KDPI organs was greatest in patients over 50-year old and in patients at transplant centers with a median waiting time of greater than 33 months. It has been reported that, in the USA, kidney transplant recipients over age 60 have allograft survival that exceeds patient survival and that there was a survival benefit associated with rapid transplantation with a high-KDPI kidney compared to delayed transplantation with a lower-KDPI organ [14].

Based on these findings, a variety of approaches have been proposed [15, 16, 17•, 18] which intend to decrease apparent transplant program risk-averse clinical decision making and which have a goal to increase the number of transplants in the USA through the use of a broader range of kidneys offered to appropriately selected recipients. These have largely focused on revision of metrics used by regulatory bodies and alterations in the structure of transplantation financial reimbursement. In 2009, the American Society of Transplant Surgeons published a response to the CMS COPs which it felt represented a threat to innovation because of the expected promotion of risk-averse behavior. The ASTS proposed the development of a program oversight mechanism that would allow for the analysis of predetermined high-risk transplants separated from standard risk transplants and also suggested that patients undergoing transplantation as part of experimental protocols be excluded from outcomes monitoring [16]. This has been termed a “carve out” approach to program performance metrics. The OPTN and the SRTR sponsored a consensus conference on transplant program quality and surveillance in 2012 to further address these issues [17•]. Participants felt that the current risk adjustment models did not adequately capture patient risk and that the perception of inadequately risk-adjusted assessment of outcomes results in transplant programs avoiding transplants with suitable high-risk candidates and donor organs. The Consensus Conference participants urged the collection of additional comorbidity variables and the development of more statistically robust risk adjustment models. They also proposed a carve out approach that excludes patients in approved protocols from outcomes analysis suggesting that this would protect innovation. In response to these suggestions, the OPTN sought public comment on two proposals to revise performance metrics. One proposal, developed by the MPSC, does measure the outcomes of higher risk transplants separately from a standard risk group in a carve out approach. This proposal was approved by the OPTN Board of Directors and awaits implementation. A separate approach, also sponsored by the MPSC proposed a system of graded probabilities of random programmatic review based on four tiers of 1-year outcomes assessments. This proposal was felt to be overly complicated and was withdrawn [19, 20].

Alternative approaches to promoting continuous quality improvement efforts through statistical process control techniques such as cumulative sum charts (CUSUMs) which can be applied to currently available data have been also been proposed. These use currently collected data to provide continuous, real-time assessment of clinical outcomes [21, 22]. This approach has been adopted by the SRTR and is available to transplant centers as a way to supplement program performance assessment alongside the use of 1-year outcomes.

The current system of transplant center performance metrics is also used by the third-party payers to structure transplant reimbursement contracts. This is also seen as a potential barrier to transplantation of a broader pool of recipients and transplant organs. A recently published national retrospective cohort study assessed the impact of the donor kidney and recipient characteristics on transplant center costs. Higher transplant costs were associated with increased recipient risk as measured by the Estimated Post-Transplant Survival (EPTS) score and with higher KDPI donor characteristics [23••]. Higher risk transplants are financially disincentivized by a reimbursement structure that is not adjusted based on case risk [24].

There is broad consensus in the transplant community that kidney transplants with higher expected risks and costs are disincentivized by the current regulatory policies and financial structures and that these policies and structures do not align with the best interest of the healthcare system as a whole or with the interests of transplant centers and patients with end-stage organ failure requiring transplantation. This misalignment may penalize patients and stifle the growth of transplantation [15, 24]. CMS and the Health Resources and Services Administration (HRSA) have recognized the need to address these fundamental concerns about the current system used to evaluate transplant center performance. In response, CMS has recently proposed changes to the thresholds by which low-performing programs are identified and this may address risk-averse program decision making to some degree. HRSA has also recently amended the OPTN contract to support a pilot program termed the Collaborative Innovation and Improvement Network (COIIN) which is being developed by the United Network for Organ Sharing (UNOS).

COIIN Project

The COIIN project proposes to pilot an alternative and novel strategy for transplant program evaluation based on broad score cards with diverse measures that include process, outcomes, and other quality metrics within a real-time performance monitoring framework. These measures will be deployed within a collaborative and information sharing group of transplant centers with their partner Organ Procurement Organizations (OPO). The COIIN initiative proposes a completely new approach to transplant program oversight which moves far beyond the incremental revisions to current risk-adjusted 1-year graft and patient survival metrics that have formed the basis of most proposals. This type of collaborative approach to improvement could be used in many settings, but for this initial pilot, the overall aim is to increase kidney transplantation, with a particular focus on moderate to high-KDPI kidneys, defined as a KDPI score greater than 50%. At a KDPI of 50%, the rate of kidney discards begins to rise significantly.

Collaborative Improvement Methodology

The COIIN project will utilize intervention guides for three improvement areas of focus (Organ Offer and Acceptance, Waitlist Management, and Care Coordination) developed with the expertise in the community from hospitals and partner OPOs who have demonstrated clinical success in these areas. Participating hospitals will work together in a Collaborative Framework of “all share and all teach” during each of the 90-day cycles of improvement. Hospitals and partner OPOs will be supported by trained improvement staff who will assist them throughout the project in their improvement efforts. Performance will be monitored with a collection of outcome, process, and relationship measures to ensure that project aims are met. Program specific reports will continue to be published by the SRTR. The COIIN project does not alter the current flagging criteria but does provide to participating hospitals a waiver from routine MPSC outcomes monitoring for kidney transplants done during the COIIN project. Parallel initiatives to revise the flagging criteria will continue during the COIIN project, and it was not felt to be within the scope of the COIIN project to alter flagging methodology.

The model of improvement for COIIN will require that each participant:

  • Declare the aims of improvement : For the overall goal of project and for each of the 90-day cycles of improvement.

  • Measure the improvement effort : Utilizing outcome, process, and relationship measures.

  • Implement change and evaluate : In rapid-cycle tests of change during each 90-day cycle.

  • Allow for empirical learning : Based on a plan to test changes on a small scale, then implement changes throughout the system with appropriate adaptions. Share these learnings with the other members of Collaborative in the spirit of “All Teach and All Learn”.

Collaborative Effectiveness

Utilizing a collaborative framework for improvement is not new to the transplant community. The Organ Donation Breakthrough Collaborative (2002–2004) was a quality improvement initiative designed to encourage the adoption of “best practices” for identifying potential donors and obtaining consent for deceased organ donation. Study findings suggest that the collaborative efforts led to an increase in donation rates at participating hospitals. [25•]. Similarly, CMS engaged the Institute of Healthcare Improvement (IHI) from 2002 to 2005 in an initiative to increase arteriovenous fistula (AVF) placement and use for dialysis patients in the USA. Among other approaches, this also included an increase in payment for AVFs to remove the financial incentives of placing grafts over AVFs. The use of fistulas has been shown to decrease mortality and morbidity for patients and reduce the cost of care. The use of fistulas in the ESRD Networks increased from 33% in September of 2003 to over 40% at the end of 2005 [26].

Improvement collaboratives have evolved out of several disciplines, and their structure and theoretical foundations may vary depending on the improvement effort [27•]. The most successful collaboratives have attempted to address program-specific educational needs rather than relying on a set curriculum. To support that customized approach to accelerated improvement, all COIIN participants will select interventions of improvement from a menu of potential interventions. Hospitals and OPOs will have the ability to customize their improvement approach by selecting relevant interventions that address their desired areas of improvement. Alemi and colleagues studied improvement activities in several healthcare organizations and found that results were quickly achieved when the focus was on testing changes rather than on detailed analysis of the current practice [28]. One of the key strategies of the COIIN project is to facilitate these rapid cycle tests of change within 90-day cycles of improvement.

While all changes do not lead to improvement, all improvement does require change. Throughout the project and the life cycle of the 3 distinct 90-day cycles of improvement and change, the participating hospitals and OPOs will be supported with 1:1 coaching from COIIN improvement staff and selected subject-matter experts in the community. COIIN improvement staff will work with participants directly during their cycles of improvement and teaching participants how to make and evaluate changes. The theory of diffusion suggests that ideas are most likely to be adopted from the same professional group. The thought is to encourage peer to peer spread from innovators to early adopters [29]. The Institute of Healthcare Improvement (IHI) Breakthrough Series efforts incorporating a variety of clinical topics such as the, Saving 5 Million Lives Campaign, Surviving Sepsis Campaign, and Reducing Cesarean Section Rates are all Breakthrough examples that have incorporated these same techniques [30]. Within the COIIN project, the use of subject matter experts, who have been identified based on their success in deploying the clinical effective practice, will be integrated into the collaborative efforts via Web-Learning Sessions, on-site coaching visits, and creative e-learning strategies including a COIIN Web-based “CLASS” (Collaborative Learning and Sharing Site) portal.

Social support is another key feature of the COIIN collaborative. All COIIN participants and their partner OPOs will be meeting regularly via Web and in person meetings throughout the life of the project. This type of support has often been emphasized as critical in the success of the collaborative learning community. Central support and real-time feedback/monitoring act as feedback mechanisms that help to maintain the tension for change [31].

Difference Between Research and Performance Improvement

There are distinct differences between improvement through research and improvement through the discipline of performance improvement. COIIN is designed as a performance improvement initiative. Table 1 outlines the differences in measurement for improvement as compared to measurement for research [32]. Performance improvement is about cycles of testing and not for proof of effectiveness. In essence, conducting cycles of testing to learn what is going to improve. Within COIIN, it is about continually measuring the metric of interest that participants want to improve and not coming up with just one intervention, but potentially multiple interventions, based on learning from prior cycles. These cycles allow for sustained improvement through a series of interventions that were informed by testing in the actual system needing change.

Table 1 Measurement differences in research and performance improvement
Full size table

Collaborative Relationships and Quality Outcomes of Care

It has frequently been said that transplantation is a team sport. The coordination of efforts between multiple disciplines in a discrete period of time that demand accurate and timely communication and effective problem-solving is the key requirement leading to a successful transplantation outcome. Within the COIIN project, the science of relational coordination will be introduced and evaluated in COIIN project participants in the context of the relationships within the transplant hospital team and the external team relationship with the partner OPOs.

Relational coordination science theory specifies three attributes of relationships that support the highest levels of coordination and high performance: shared goals, shared knowledge, and mutual respect. These relational dimensions are reinforced by specific dimensions of communication that support coordination and high performance, namely frequency, timeliness, accuracy, and when problems arise, coupled with a focus on problem-solving, rather than blaming [33•]. The research of Gitell began with her study of work coordination within the airline industry and quickly migrated to other areas of health care [34,35,36,37]. In these studies, significant correlation between effective work coordination and collaboration was often associated with improvement in quality of care outcomes, increased staff satisfaction, and reduction of cost of care. Relational coordination is evaluated using a validated survey instrument that contains seven questions evaluating team performance across the three relational attributes and specific dimensions of communication. Participating hospitals and partner OPOs will have their relational coordination scores determined at baseline and at the end of the project via the survey instrument. The theory is that those transplant hospitals and OPOs who have established a good sense of team and collaborative working relationships built on mutual respect will have similar high performance in their clinical outcomes.

COIIN Data Collection and Monitoring

The COIIN data collection and monitoring plan is based on a combination of outcome, process, relationship, and structural measures called a balanced scorecard. Data elements are available from the existing OPTN data repository that will support the monitoring of outcome and process measures with a limited need to obtain additional data from the pilot programs. The collected data and visualizations will be updated according to the timeframes available for each unique metric. For example, outcome data by SRTR may be updated monthly, whereas process measures submitted by pilot programs may be updated bi-weekly. The balanced scorecard will be used to display trends over time that will help describe each pilot program’s performance and activity. The COIIN staff will review outcome, process, relationship, and organization data for trends and based on this information, action revisions or consultations with the COIIN staff or faculty may be suggested. Rather than looking at outcome measures as a stand-alone indicator of success or failure, the use of additional measures has a “balancing” effect; in this way, we can measure other areas that impact performance such as processes and relationships. By including other indicators of performance, participating organizations will have a more comprehensive picture of their improvement efforts and will be able to impact change in a more “real-time” fashion. Reliance upon outcome measurement alone does not provide a complete picture of organizational improvement or enable the improvement efforts to happen quickly. UNOS proposes to cover four domains of performance evaluation. The balanced scorecard’s specific measurement domains are the following:

  • Transplant outcome measures

  • Process measures aligned with specific drivers of improvement

  • Structural outcomes/improvement capability measures

  • Relational coordination scores

The primary outcomes of interest for the COIIN project are the following: 1-year-post-transplant graft survival rates, 1-year-post-transplant patient survival rates, and organ offer acceptance rate, stratified by KDPI. Outcomes will be monitored using CUSUM reports produced by the SRTR, calculated for each COIIN program, and updated monthly. Traditional program evaluation tools use 2.5-year cohorts which have a data lag to allow for a full 1-year follow-up on all patients. By using the CUSUM methodology, COIIN staff will be able to discuss more recent performance and trends with the program to understand either continued performance trends and/or clustered events. Updates for risk-adjusted transplant rate, risk-adjusted waiting list mortality, and organ offer acceptance rate (not risk adjusted) for each kidney program will be supplied on the metric page for the COIIN participant hospitals. The transplant rate, organ acceptance rate, and waitlist mortality rate are not meant to be used for the identification of programs that need intervention, but as ways to understand the behavior of the transplant program. In conjunction with outcome measures, each pilot program will track a variety of process measures to collect and track performance related to three effective practices: organ offer and acceptance, waitlist management, and care coordination. In addition to outcome and process measures, structural and improvement capability measures will be used to evaluate current organizational structures and staff capability and capacity to participate and sustain improvement efforts and interventions. It is recognized that the potential of increased costs resulting from the utilization of high KDPI organs may also be a major driver of decision making for some transplant centers. Given the complexities of measuring cost and the lack of reliable cost data, this issue is not directly addressed by the COIIN project. Expansion of the COIIN initiative beyond this initial “pilot” phase may allow cost issues to be evaluated.

Conclusions

There continue to be opportunities to evaluate processes to increase transplantation, improve the quality, and effectiveness of care. The body of knowledge collected through research and performance improvement collaboratives like the COIIN project will continue to drive change and encourage sustainable improvement. Past collaboratives in the transplant industry, such as the Organ Donation Breakthrough Collaborative have established the groundwork for community success in these types of efforts. The COIIN project will continue to add the body of knowledge and a different perspective on alternative monitoring for the transplant industry.

The views expressed in this article are those of the authors and do not necessarily reflect the official policy of the OPTN or the federal government. This paper reports data from the OPTN system, which includes data on all donors, wait-listed candidates, and transplant recipients in the USA, as submitted by the OPTN members, with oversight provided by the Health Resources and Services Administration of the Department of Health and Human Services.