There are now approximately 13.7 million cancer survivors in the USA [1], and this number will increase due to improvements in diagnosis and treatment and the aging of the population [2, 3]. Not only are cancer survivors at risk for recurrence of their primary cancer, but also complex, multimodality treatments place them at risk for long-term and late effects such as secondary malignancies, cardiovascular disease, endocrine disorders [4], and general symptom distress [5–8]. These factors have stimulated a focus on survivorship care and long-term follow-up, where gaps in care and fragmentation have been identified as problems [9]. To guide remediation of these difficulties, the 2006 Institute of Medicine (IOM) report [10] identified four components of survivorship care as “essential” and outlined ten recommendations for care that describe a range of activities to improve outcomes. More than 6 years later, however, cancer centers and survivorship programs continue to struggle to accomplish these recommendations, and systematic evaluations have been few [11–15].

A key IOM recommendation was the provision of treatment summaries (TSs) and survivorship care plans (SCPs) for patients completing primary treatment [10] to facilitate transition to post-acute care and improve coordination of services [9, 16]. Other groups, such as the President’s Cancer Panel, the Centers of Disease Control, the American Society of Clinical Oncology, and the LIVESTRONG™ Foundation, have also called for TSs and SCPs to be provided to patients completing primary therapy [10, 17, 18], and the American College of Surgeons Commission on Cancer (CoC) has mandated that accredited institutions make these available by 2015 [19, 20]. TS and SCP documents outline disease and treatment exposures and provide education regarding long-term and late effects, resource information, and a comprehensive follow-up plan for medical and psychosocial care. Numerous TS and SCP templates are in use, including the LIVESTRONG™ Care Plan powered by Penn Medicine’s OncoLink (livestrongcareplan.org), Journey Forward (journeyforward.org), What’s Next? Life After Cancer Treatment (www.cancer.org/acs/groups/content/documents/document/acspc-025795.pdf), Prescription for Living (tiny.cc/SFA8e), American Society of Clinical Oncology (ASCO) Treatment Summaries and Survivorship Care Plans (www.asco.org), and various institutionally created models. The number and variety of these templates are likely to expand as centers begin implementing care plans in earnest, yet there has been little attention paid to evaluating the quality of these templates or the outcomes they achieve [11, 12, 21–23].

Along with mandates to provide TSs and SCPs [19, 20], there are also recommendations concerning the content to be included. The most comprehensive of these recommendations has come from the IOM [10] (Table 1) and has been adopted by the CoC [20] as a minimum standard. The IOM recommendations, however, are broadly defined and non-specific, providing a general frame for a range of treatment settings, cancer sites, treatments, follow-up plans, and long-term and late effects. This lack of specificity makes it difficult to ascertain whether recommended content is actually being addressed, compare outcomes across document templates, or develop an integrated research literature that will allow for evidence-based decisions regarding which elements are truly essential to improving outcomes.

Table 1 IOM recommendations for content of the survivorship care plan
Full size table

One approach to remedying this situation is to develop standardized metrics to quantify the information being provided across the various TSs and SCPs. The objective of such work is not to mandate what information should be in a care plan, but rather to allow clinicians and researchers a means of describing what is being presented to survivors with reference to some agreed upon set of criteria. This, in turn, should foster implementation efforts by allowing assessment of content and fidelity. Similarly, it provides a means of describing and reconciling a diverse literature in which numerous TS/SCP documents are being promulgated. The goal of the current project was to develop cost-effective metrics to assess concordance between IOM recommendations for content and the TSs and SCPs being provided to breast cancer survivors, and to demonstrate that these metrics can be scored reliably across raters as a first step in the process of construct validation. Breast cancer survivors were chosen as the initial group for metric development since they constitute the largest and most well-characterized group of cancer survivors in the USA [2] at approximately 2,971,610 in 2012.

Method

Operationalizing IOM recommendations

This project required the creation of two checklists to operationalize the IOM recommendations for TS/SCP content. The multidisciplinary study team included a medical oncologist (AD), oncology nurses and nurse practitioners (LJ, CS, AJ), and clinical health psychologists (SP, BR). As a first step, all IOM recommendations listed in Table 1 were deconstructed and mapped onto potential disease characteristics, treatment exposures, and follow-up options by study leaders (SP and CS). For example, the IOM TS item 2, “Tumor Characteristics,” was deconstructed into items reflecting the reporting of definitive breast cancer diagnosis, laterality, staging, histologic or nuclear tumor grade, estrogen receptor status, progesterone receptor status, HER2/neu status, and status of any lymph nodes dissected.

Content validity refers to the degree to which the elements of an instrument are relevant to and reflective of the content of a given construct used in a specific context [24]. This includes such aspects as presentation, scoring, ordering, and instruction. In the current project, the instrument being developed possesses content validity to the extent that the items represent the IOM recommendations concerning TS/SCP to be provided to breast cancer patients. As an initial attempt to establish content validity, remaining team members, representing a range of subject matter experts, were presented with the items for comment on completeness, clarity, and comprehensiveness with respect to IOM recommendations. These exposure, comment, and revision cycles were repeated in an iterative fashion until 100 % consensus that items represented all aspects of the IOM recommendations was reached. Following the creation of items, a similar iterative process was undertaken regarding assignment of items to overarching domains.

As a next step in establishing content validity, decision rules were established to allow for scoring. Again, exposure, comment, and revision cycles by subject matter experts were performed until consensus was reached. Items were determined to be dichotomous and reflect the presence or absence of information in the TS or SCP documentation. Items were considered present if noted in the TS/SCP or if the TS/SCP provides an explicit field to list a relevant piece of information (e.g., a field is devoted to clinical trials information), and this was noted as being non-applicable. Items were considered absent if content was not described in the TS/SCP or if an explicit field was present but not completed. Thus, non-applicable information could be scored as present so long as it was noted that it was not applicable for a given survivor. Accuracy of noted information was not taken into account. Items were weighed equally to derive a total score.

Procedure for reliability assessment

Once consensus was reached concerning item construction and scoring, and following approval by all relevant Institutional Review Boards, 13 clinical sites were recruited to generate a relevant pool of TS/SCP materials for rating. These included seven LIVESTRONG™ Foundation Survivorship Centers of Excellence, each an NCI-designated Comprehensive Cancer Center engaged in providing survivorship care and TS/SCP materials to breast cancer survivors. To ensure heterogeneity in type of setting and materials received for rating, we also recruited six community-based centers associated with the Centers of Excellence. Community and public hospitals were the most common type of community site, with the remaining sites representing a community health center, a university-based cancer treatment clinic, and a multi-specialty group practice. Further details about participating sites are reported elsewhere [11].

Materials to be rated to establish reliability consisted of the last five TSs and SCPs and accompanying resources (e.g., pamphlets, referrals, resource lists) that were provided at each of the 13 sites in the preceding calendar year (n = 65 TSs and n = 65 SCPs total). All materials were de-identified to maintain patient confidentiality. Accompanying materials (e.g., pamphlets) as well as specific TS/SCP documents were rated in order to capture all written information received by survivors. There was heterogeneity in the materials to be rated. Most sites (62 %) provided patients with institutionally developed TSs, while 23 % used the LIVESTRONG™ Care Plan, and 15 % used Journey Forward to create the TS document. Institutionally developed SCPs were used by 31 % of participating sites, and 46 % used the LIVESTRONG™ Care Plan alone or in conjunction with additional materials. The Journey Forward care plan was used by 15 % and the ASCO care plan by 8 %, also in conjunction with supplementary materials.

These TS and SCP materials were then rated by two independent raters (CS and SP) using the concordance tools described above. Inter-rater reliability was calculated across items within each TS and SCP using Cohen’s kappa. Total scores for each TS and SCP were calculated, and intraclass correlations (ICC) using a two-way mixed method were calculated as an index of reliability for absolute agreement at the document level.

Results

The scorecard instruments

The approach described above resulted in two scorecards: a 60-item tool assessing treatment summaries and a 32-item tool assessing survivorship care plans, collectively called the survivorship care plan assessment checklist. The TS concordance tool covered 13 content domains, while the SCP concordance tool covered information across 10 domains. Items are scored as positive if (1) information corresponding to a given item is present or (2) a dedicated space (e.g., text box) for such information is present but noted to be non-applicable. Tables 2 and 3 provide the scorecard instruments themselves, domains assessed, and additional scoring instructions.

Table 2 Scorecard: treatment summary items
Full size table
Table 3 Scorecard: survivorship care plan items
Full size table

Reliability

Overall, raters were in agreement concerning presence or absence of TS information 82 % of the time across the 65 sets of documents. The mean inter-rater reliability (kappa) across the 65 TS ratings was 0.76 (SD = 0.12, 95 % CI = 0.73–0.79) indicating “substantial agreement” [34] between raters. Kappas ranged from 0.42 to 0.97 within the sample TSs. Within the sample of 65 SCP documents, raters were in agreement concerning presence or absence of SCP information 90 % of the time. Mean kappa again indicated substantial agreement (M = 0.66, SD = 0.16, 95 % CI = 0.62–0.70) with a range from 0.33 to 1.00. Intraclass correlation based on total score across documents was similarly high. Within TS materials, ICC = 0.85 (CI = 0.76–0.91, p < 0.001), while within SCP materials, ICC = 0.75 (CI = 0.62–0.84, p < 0.001). These results are consistent with a “strong” (SCP) to an “almost perfect” (TS) agreement [25].

Discussion

The 2006 IOM report [10] recommended standards for the information to be included in TS and SCP documents, and the CoC has mandated that these documents be part of the basis for accreditation by 2015 [20]. Despite this, very little work has been performed to outline means of ensuring that these standards are met [11, 12]. This is unfortunate, as cancer survivors often have low confidence in their ability to navigate survivorship [26], and SCPs may prove to be one means of improving this and other relevant outcomes in post-treatment cancer survivors. Moreover, despite mandates for implementation [19], there has been little investigation concerning the efficacy or effectiveness of these interventions [21–23], and research to date has focused more on reach, uptake, and qualitative analysis of stakeholder preferences for content and delivery than on the outcomes achieved by patients [21, 22, 27–35]. Indeed, the IOM [10, p. 154] states that “such plans (have) not yet been formally evaluated. Despite the lack of evidence to support the use of (SCPs)… some elements of care simply make sense—that is, they have strong face validity….” Others [36] have noted the similarity between care plans and hospital discharge summaries, noting that these were (and are) being implemented prior to a literature demonstrating effects on outcomes accumulated.

This project developed two instruments for assessing the degree to which TS and SCP materials are in concordance with IOM recommendations. As demonstrated, ratings can be made with a high degree of inter-rater reliability and agreement. These characteristics enable use of these instruments for clinical, quality improvement, and research purposes. With practice, these instruments can be used reliably to evaluate the degree to which the materials provided to breast cancer survivors mirror the recommendations of the IOM. Our hope is that this will, in turn, foster implementation research and allow for greater ease in reconciling a diverse literature to enable systematic review and meta-analysis as the literature grows. For example, we have used these instruments to examine the degree to which LIVESTRONG™ Survivorship Centers provide TS/SCP materials that are in concordance with the IOM recommendations [11]. Our findings suggested that even in settings with dedicated staff and funding for such efforts, less than half of the recommended TS content and less than two thirds of the recommended SCP content, on average, were included in materials delivered to breast cancer survivors. Some information was commonly provided (e.g., possible toxicities and late effects). Other information (e.g., familial cancer risk and documentation of which provider is responsible for routine cancer surveillance), however, were rarely provided, and these included areas that both survivors and providers have reported to be highly desirable [27, 29, 33, 37, 38]. These results have enabled members of the LIVESTRONG™ Network to target areas for improvement, demonstrating the potential of the scorecard for use in quality improvement efforts. In addition, these findings have facilitated the development of a standardized TS and SCP for use in an ongoing study exploring outcomes of care plans delivered to breast cancer survivors.

As they currently stand, the IOM recommendations for TS/SCP content are comprehensive, broad, and likely to be over- rather than under-inclusive. Given that completion of such plans is not currently a reimbursable activity and that the resources required to complete these plans are noted as a primary barrier to implementation [11, 12, 16, 29, 39, 40], less comprehensive and detailed care plans are likely to become the norm in practice. Thus, determination of and consensus about what constitutes the “essential elements” to be included in a plan of care should become a priority. Instruments such as this can play a crucial role in describing informational content, allowing comparisons across different sets of materials, and will ultimately be essential for linking particular elements of care plans to demonstrated outcomes.

Others have used similar “scorecard” methods to describe the content of TS/SCPs relative to IOM recommendations across differing institutes and populations [12]. Although findings from these authors are quite similar to those we have presented elsewhere [11], neither were the tools used [12] specific to the populations of interest (i.e., both breast and colorectal cancer documents were rated using the same instruments) nor were psychometric data reported. In addition, rather than examining materials presented to survivors, blank templates and “de-identified or hypothetical completed” materials were examined. These limitations were overcome in the current project.

Limitations

The use of the IOM recommendations as the basis for concordance may lead some to assume that these are either empirically based or the “gold standard” against which all TS/SCPs must be evaluated. Rather, the IOM recommendations are consensus-based, but the most comprehensive recommendations currently available and the standard against which accreditation will be assessed [20]. As such, they provide an excellent starting point from which to begin the discussion of which elements of care plans are essential to improving outcomes. Another limitation concerns the assessment of reliability in the current project. All documents were rated by study team members primarily responsible for the creation of the rating instruments. This familiarity with the materials may have somewhat inflated reliabilities, and we are currently working to replicate psychometric findings using naïve but trained raters to better approximate clinical reliabilities. As well, presence or absence of information concordant with IOM recommendations is what is being scored. Accuracy or depth of content cannot be assessed using these methods, and this should certainly be examined in future work. Finally, the current study focused on reliability of the scorecard instruments themselves. Future work will need to examine the degree to which given items or domains affect patient outcomes such as health behaviors, knowledge concerning treatment and late effects, and communication between patients and providers.

Conclusion

The IOM, the CoC, and others are moving the provision of TS/SCP materials to cancer survivors forward and mandating the content to be addressed within them. Implementation and outcomes research are lagging behind these mandates, and tools are needed to facilitate progress. This project was conducted in direct response to this need and offers two easy-to-use and highly reliable tools for assessing the degree of concordance between TS/SCP materials and these content areas. We are working to develop similar metrics with analogous content domains that can be applied to other specific cancer populations. Use of tools such as these can not only foster improved implementation efforts with a high degree of fidelity, but also allow for meaningful comparisons across differing documents, samples, populations, and institutions so that decisions concerning what is essential in care and care planning can move toward being empirically based.