Keywords

1 Introduction

For women in the United States, breast cancer is the most common cause of death from cancer [1]. Older women are particularly vulnerable: nearly half of new diagnoses occur at age 65 and older and one-quarter occur after age 75 [2]. Fortunately, over 62% of diagnoses are found during stage I when the cancer remains localized and highly curable, and most patients undergo lumpectomy or breast-conserving surgery [3, 4]. While the use of adjuvant radiation therapy (RT) alongside lumpectomy has been shown to reduce the risk of local recurrence, radiation therapy does not improve overall survival for the majority of older women [5, 6]. This is due to older women often having comorbidities that predispose them to non-cancer-related death, thus minimizing radiation therapy benefits [7]. Side effects of radiation therapy, such as fatigue, breast pain, and pneumonitis, and burden of treatment (e.g., travel to undergo radiation therapy) are important considerations before receiving radiation therapy [8,9,10,11]. Indeed, national guidelines recommend that radiation therapy could be omitted for women ages 70 and older with low-risk, early-stage breast cancer. Despite these recommendations, more than two-thirds of this population undergo radiation therapy [12, 13].

The discrepancy between guidelines and actual clinical treatment for older women underscores the challenges in radiation therapy decision-making. Effective treatment plans for older women with breast cancer are highly individualized and require consideration of a multitude of factors—patient age, tumor size, patient preferences, comorbidities, functional status, and potential benefit and harm. Consequently, it may be difficult for physicians to accurately estimate an individual patient’s risk for cancer recurrence as well as the benefits from radiation therapy. As a result, patients will not be able to make informed decisions that align with their preferences and values. Patients are also prevented from participating in the shared decision-making process, despite research supporting their desire to become more involved [14]. Even women who seek a more passive role in their treatment decision want to be as informed as possible [15]. Therefore, physicians are too often left to make these determinations on their patient’s behalf.

Across the health care field, digital risk calculators and decision aids have been developed to support complex decision-making. These digital decision aids—electronic tools used to improve patient understanding, participation, and decision-making in health care—have been shown to improve patient satisfaction and reduce decisional conflict [16,17,18]. Decision aids support patients’ knowledge allow for informed decision-making, and lead to better health outcomes [19, 20], making them valuable tools for promoting informed discussion between patients and providers.

Additionally, the User-Centered Design (UCD) approach stipulates that the targeted ender user provide feedback and criticism during every phase of the design process [21, 22]. In doing so, the tool or product may better serve and match the needs of the end user. Problems may be identified earlier, and with feedback from the user, each iteration of the product will improve upon the one before it. In the context of decision aids, applying these principles ensures that the final product features functions, a display, and an interface that accounts for the particular needs of the end user.

To assist the decision process of whether or not to pursue radiation therapy for breast cancer, we sought to develop a patient-centered, risk-based digital risk calculator. By providing patients with an easily interpretable, graphic-based risk-benefit analysis of radiation therapy, individuals could better understand their options and align their goals with expected treatment outcomes.

While many decision aids use risk calculators to simplify medical decisions in oncology, to our knowledge, no risk calculator for women with breast cancer ages 65 and older exists that accounts for personal comorbidities and functional status and estimates the benefits of radiation therapy to support shared decision-making in the clinical setting. In this paper we describe how we used the User-Centered Design (UCD) approach to design, develop and evaluate the usability and acceptability of the risk calculator, Radiation for Older Women (ROW), the tool’s potential to improve decision-making, and treatment satisfaction for breast cancer patients.

2 Methods

This research study focused on the acceptability and usability of the ROW digital risk calculator amongst potential users. ROW was created to improve shared decision-making between providers and older women with breast cancer who underwent lumpectomy and were considering radiation therapy. By providing individualized risk estimates with and without radiation therapy, ROW could help patients make informed decisions about their health.

Our methods followed an iterative UCD process that emphasized relevant stakeholder feedback to assess, inform, and refine the ROW tool at each development stage [21, 22]. This approach ensured ROW’s interface and functionality would meet the needs of the patients in the clinical environment. Major stages of ROW’s development process included: building a risk calculator, developing a prototype, usability and acceptability testing, and field testing and piloting.

We used two prediction models to create a risk calculator that could estimate local cancer recurrence and all-cause mortality with and without radiation therapy. We used the Early Breast Cancer Trialist Collaboration Group (EBCTCG) model to forecast both five- and ten-year outcomes for stage I and stage II breast cancer [23]. Breast cancer specific inputs included a woman’s age, tumor size, tumor grade, surgical margin status, lymph node involvement, and estrogen receptor status, which are important predictors for recurrence. To estimate an older woman’s life expectancy, we utilized the ePrognosis model, frequently used as a survival prediction tool in geriatric assessment [24]. Specific risk inputs for the ePrognosis model included a woman’s age, sex, body mass index (BMI), smoking status, functional status, and four selected comorbidities (lung disease, congestive heart failure, diabetes, and other cancer). To account for additional health factors beyond the included 18 variables in ROW, we built in an option for clinicians to use their expertise and adjust a patient’s risk based upon other known health or tumor characteristics. We conducted 56,700 model simulations using each combination of the 18 input parameters to generate risk outcomes. Further detail regarding the statistical analysis and methodology of the prediction calculator has been previously described [25].

For the tool’s iterative design process, we first put together an advisory committee consisting of breast cancer survivors (n = 6), advocates of breast cancer care and aging (n = 7), oncology clinicians (n = 4), and researchers (n = 4). The advisory committee assessed user needs and limitations to inform the tool’s functional and technical requirements. Following requirement gathering, we developed the screen mockups of the ROW tool using Balsamiq® [\cite: https://balsamiq.com/]. We presented multiple series of prototype mockups to the advisory committee. The committee provided feedback around visual presentation, interface, functionality, text, usability, and acceptability.

The ROW digital tool is divided into three sections: (1) patient data input, (2) clinician data input, and (3) risk estimate results. First, patients enter demographical, health, and lifestyle information, including age, height, weight, smoking status, comorbidities, and functional status. These characteristics are used to calculate a patient’s mortality risk. Figure 1 below is an early black-and-white sample mockup draft, which was presented to the advisory committee for feedback and comments.

Following the patient section, the clinician is required to enter data. The clinician inputs patient-specific tumor characteristics and reviews the patient’s data input for accuracy. The clinician has the option to edit and update any patient entry errors. Following the input obtained from the patient and clinician sections, the tool then generates a graphical risk estimate of the patient’s five- and ten-year breast cancer local recurrence and mortality, with and without radiation therapy.

Fig. 1.
figure 1

Early black and white ROW mock-up using Balsamiq software.

Full size image

We recruited two groups of participants in the pilot study to evaluate the usability and acceptability of the tool. While we initially sought to include only patients with stage I-II breast cancer receiving breast-conserving surgery who were candidates for adjuvant radiation therapy, there was an insufficient number of women who fit the specific qualification criteria. Therefore, we also recruited a separate group of women volunteers who did not have breast cancer. The number of participants between both groups would provide us with a sufficient number of participants for qualitative feedback. All participants were required to be women ages 65 and older who could read and understand English and provide informed consent.

Yale IRB approved the study, and volunteers were recruited from the local New Haven, Connecticut community using IRB approved flyers posted publicly in New Haven, CT. Patients were recruited from the Smilow Cancer Center at Yale New Haven Hospital using authorized access to patient EPIC medical health records. Each participant was compensated $20 for her time. Interested volunteers complete the tool in a non-clinical setting, while patient volunteers were asked to complete ROW in an actual clinical setting prior to meeting with their oncologist to discuss their diagnosis and treatment.

During pilot testing of ROW and completion of the evaluation survey, the research assistant observed patients’ and volunteers’ use of the tool, noting any challenges or difficulties collecting all qualitative feedback, and assisting with technical obstacles. In the process of completing the decision aid, volunteers and patients entered their actual demographical and health information. For the clinician section, the research assistant entered in preset, standardized tumor and cancer characteristics for the volunteer group. For patients, their oncology provider entered in the patient’s actual tumor and cancer characteristics during the patient’s clinical appointment.

Following the entry of the participants’ health data into ROW, volunteers were shown the results of their input with the research assistant, while patients viewed their results alongside their clinician. Results compared ten-year mortality and local cancer recurrence risk with and without radiation therapy. For volunteers, the risk estimates were for example purposes only, which the research assistant explained. The risk estimates for patients reflected actual clinical prognostic outcomes. The clinician discussed the local recurrence outcomes with the patient and used these results to arrive at a shared decision of whether to pursue adjuvant radiation therapy.

We focused on evaluating the usability and acceptability of ROW, as well as other qualitative perceptions of the tool, through a paper survey that was completed by both volunteers and patients. The survey for volunteers gathered feedback from a controlled, simulated clinical experience with the ROW tool. The patients’ survey reflected use of the ROW tool in an actual clinical setting with an oncologist.

All participants completed the following survey sections:

  • “Experience Using the Tool”: evaluated usability, acceptability, and perception of the ROW tool

  • “Radiotherapy Benefits”: evaluated knowledge of breast cancer survival with and without radiation therapy treatment

  • “Health and Demographics”: collected race, ethnicity, education level, marital status, income, and other characteristics

  • “Computer Experiences”: evaluated familiarity with everyday technology

The “Experience Using the Tool” portion of the survey measured usability—or the ease in which participants interacted with ROW—through the system usability scale (SUS), a validated, ten-question Likert-scale tool that may be applied to small sample sizes [26,27,28]. Open-ended questions also provided an opportunity for participant recommendations on the tool’s design and features and to measure acceptability, or the degree to which ROW served its intended purpose. The “Radiotherapy Benefits” section surveyed participants about the impact of radiation therapy on survival and whether or not they planned to receive radiation therapy. “Health and Demographic Information” collected information on participants’ marital, race, income, and education status as well as emotional responses to ROW. These questions were intended to understand any limits, biases, or preferences in the design of ROW. The Computer Efficacy Scale (CES) was used in the “Computer Experiences” section to determine participants’ familiarity and confidence with everyday technology [29]. Additionally, the patient group was asked to respond to the “Decision-Making Process”, which used a five-point Likert scale to assess patient decisional satisfaction of whether to pursue radiation therapy. This section of the survey evaluated ROW’s ability to influence patient decision-making around treatment. All participant survey feedback was aggregated and analyzed as numerical indicators and summarized using common descriptive statistics appropriate for discrete and continuous data.

3 Results

Outcomes of our work included in-depth feedback on ROW prototypes from targeted end users and advisory committee members about ROW’s design and usability, leading to a polished and user-friendly final product. The finalized tool features a large font size, sensitive language, a streamlined process for inputting user information, and a simplified pictographic for risk estimates, enabling ROW to meet the specific needs and challenges of older women. To be mindful of older population needs, the interface and screens display prompts for one question at a time in large, clear, and simple font with minimal text and graphics. A sample screenshot displaying the final high-contrast, large text interface is shown in Fig. 2.

A text-to-speech function supports read-aloud functionality for patients with vision difficulty or low literacy. A progress bar displays how far a patient has advanced through the tool, allowing the user to anticipate the time required for completion. Core features of the tool include:

  • Clean design with patient prompts displayed clearly and individually on each screen in large font

  • Older adult friendly interface

  • Straightforward patient input features

  • One-click screen navigation

  • Automated text-to-speech translation, with headphone support

  • Design for seamless use in a clinical workflow

  • A web-based platform, providing an easily accessible platform for both patients and clinicians that can be updated rapidly and disseminated broadly

  • A visual pictograph of prognostic estimates

  • Cancer recurrence and mortality outcomes, with/without radiation therapy

Fig. 2.
figure 2

ROW patient health – patient data entry prompt.

Full size image

Following the patient’s data entry, the clinician may enter and review the data entered by the patient as well as include specific characteristics of the cancer. This review includes correcting patient responses of ‘unknown’ to questions about health conditions or diagnoses. This screen display is shown below in Fig. 3.

Fig. 3.
figure 3

ROW clinician data input screen.

Full size image

ROW was optimized as a web-based application for any internet-capable device, including a desktop, tablet, or smartphone. Figure 4 shows the final risk estimate pictograph, a colorful, side-by-side design displaying risk estimates with and without radiation therapy that are specific to the individual patient’s health and cancer characteristics. This display facilitates comparison and interpretation for both patients and clinicians.

Fig. 4.
figure 4

ROW risk calculator pictograph comparing ten-year recurrence with and without radiation therapy.

Full size image

Field testing was conducted with all participants, 22 volunteers and 6 patients, who completed ROW in the pilot study. Averages between the two groups were similar: the mean patient age was 73Footnote 1 (median 73.5, SD = 3.4) while the mean age among volunteers was 73.2Footnote 2 (median = 72, SD = 5.8). Detailed demographics of the sample populations are included below in Table 1.

Table 1. Patient and volunteer demographic characteristics.
Full size table

The mean System Usability Scale score of the ROW tool was 84.7 (SD = 15.6), classifying its usability as ‘excellent’ [26]. Volunteers (n = 22) scored the ROW tool an average of 88.4, while patients (n = 5) gave the tool a 68.5 (SD = 22.9)Footnote 3. Survey responses from one patient were omitted, as her responses of ‘1’ to every question indicated that she did not adequately understand the SUS questions. For all participants who provided their age and a SUS score (n = 24), the median age was 72.5 while the mean SUS score was 87.3. Participants who were younger than the median age scored ROW an average of 87.1, while participants older than the median age scored ROW an average of 87.5. Table 2 below lists the SUS questions.

Table 2. Patient and volunteer ROW questions.
Full size table

With the exception of one ‘no’ response, all participants (n = 26) indicated that they would recommend the ROW tool to others (one respondent did not answer this question). On a scale of 1 (meaning ‘very dissatisfied’) to 7 (meaning ‘very satisfied’), all pilot study participants rated the ROW tool a mean of 6.4 (n = 27; one respondent did not answer this question); patients rated the tool a 5.7 (n = 6) while volunteers rated the tool a 6.7 (n = 21). Fourteen participants provided verbal or written feedback that ROW would help their decision of whether to pursue radiation therapy treatment. One patient commented that she did not like the “strange robotic voice” of the audio narrator. Other suggestions included having a stand to support the iPad and better differentiation of on-screen colors.

Twenty-seven participants responded to the statement, “the chance that my breast disease will come back after treatment is the same for lumpectomy with radiation as it is for lumpectomy without radiation.” Of the five patient responses, four correctly answered “false” while one reported “don’t know.” Only one volunteer correctly reported “false”; thirteen reported “true” and eight reported “don’t know.” When asked whether they would elect to receive radiation therapy, all twenty-eight participants responded: twelve responded “no”, four responded “yes”, and twelve responded “don’t know.” Of the patient group, three responded “no”, two responded “don’t know”, and one patient indicated she would receive radiation therapy.

Only patients responded to the statement, “the chance of being alive 5 years after treatment is generally the same for lumpectomy with radiation as it is for lumpectomy without radiation.” Three patients correctly answered “yes,” and there was one response each to “no” and “don’t know” (one did not respond). The six patients also reported their plans to pursue radiation therapy: one reported plans to receive radiation therapy, three reported no plans to pursue radiation therapy, and two reported uncertainty. Of the six patients, four were satisfied with their radiation therapy decision, though one of these four individuals also reported ‘not sure’ about whether to pursue radiation therapy treatment. Of the six patients, four reported ‘strongly agree’ when asked if they felt adequately informed about the issue important to their decision; one reported ‘neutral’ and another ‘strongly disagree.’

All participants were asked about their emotional state after viewing their cancer recurrence risk estimates. Answers were rated on a scale of 1 to 4, with 1 corresponding to ‘not at all’ and 4 corresponding to ‘very much.’ Patients reported a median of 3.5 (SD = 1.7) in terms of feeling calm after completing ROW, indicating a high degree of calm. Volunteers reported a median 3.8 (SD = 1.2) when asked the same question. Patients also reported a median of 1.3 (SD = 0.5) when asked if they felt upset, and volunteers reported a median of 1.2 (SD = 0.8). All patients except one reported only a 1 or 2 when asked if they were worried. Nineteen volunteers reported a 1 when asked if they were worried; one participant answered 2 and another answered 4.

All participants completed the Computer Efficacy Scale (CES) with a mean score of 7.6 out of 10 (SD = 0.9), indicating a relatively strong degree of confidence amongst study participants.

4 Discussion

Electronic decision aids have already proven tremendously beneficial in promoting shared decision-making and patient education for breast cancer treatment, but the vast majority of these published decision aids rely on general averages that do not consider actual estimated clinical outcomes for patients. Research has demonstrated an increasing desire amongst older women with early-stage breast cancer to have input and agency in their treatment decisions, and ROW empowers these women by providing highly-individualized and straightforward estimates [30]. ROW is a significant advancement for integrating a patient-specific breast cancer risk calculator and decision aid into the clinical setting. Future research is now needed to determine ROW’s accuracy to project clinical outcomes among a larger scale study amongst older women with breast cancer.

Both patients and healthy volunteers gave ROW high marks for usability and acceptability, with participants finding the tool easy to use and well-integrated into the clinical workflow. Designing technology to fit the specific needs of older adults is multilayered and requires special cognitive, visual, language, and dexterity considerations. Participants’ overwhelming recommendation of ROW and high degree of satisfaction signaled the strength of the tool’s design, presentation, and content for this population. Incorporating an extensive user-centered design process from ROW’s inception allowed for manipulation of important design and display features to best convey the sensitive and complex information of early breast cancer treatment options for older women. While patients did score ROW lower on average than volunteers for SUS, the scored average remains above the expected SUS average score of 68. Additionally, comparisons between the groups must be made with caution: the number of cancer patients was significantly fewer than volunteers. The tremendous variability in distribution of SUS scores amongst the patient group also reflects the smaller sample size. With the sensitive and emotionally driven content of ROW, it was encouraging to see that neither patients nor volunteers found ROW to cause undue emotional distress. These responses implied that the majority of participants, both patients and volunteers, maintained calmness after completing ROW and while observing their cancer recurrence risk estimates.

ROW was not designed to be used as a stand-alone tool. There was a greater consensus among the patient group: almost all understood that while radiation therapy changed the likelihood of recurrence, the risk of mortality remained unchanged. These patients likely benefitted from the provider’s explanation of the ROW pictograph estimates. Ultimately, only one patient chose to pursue radiation therapy, and it was clear from her survey responses that she recognized how radiation therapy would impact local recurrence but not overall survival. In our study, patients likely may have needed more time to make a final decision. Choosing a breast cancer treatment is a critical medical decision for anyone and carries an enormous emotional burden. Patients in our study were asked about their treatment decision only minutes after receiving their risk estimate and prognosis. However, delaying assessment of decisional satisfaction may be more accurate.

Limitations of this study included few patient group participants. There may be other considerations amongst breast cancer patients that are only revealed with larger sample sizes. Future research should expand on mixed-method evaluation of ROW amongst a larger sample of breast cancer patients to better understand how patients may use ROW to understand their diagnosis and guide treatment options. As the tool was only offered in English, this resulted in a homogenous participant pool that was predominantly white, educated, and of higher socioeconomic status. Further, we did not test ROW’s usability and acceptability amongst clinicians to understand whether it would be well received.

That said, ROW serves as a strong example of how to successfully build a risk-based electronic decision aid to improve shared decision-making in the clinical environment.

5 Conclusion

Older women found the ROW risk calculator to be a helpful, easy-to-use decision aid with important clinical implications. By providing personalized risk estimates, ROW can deliver information to a patient population that desires knowledge and shared decision-making. The usability and acceptability of ROW among older women demonstrates the promise of using a personalized prediction model to facilitate patient-provider conversations around breast cancer treatment. ROW serves as an exemplar for future electronic, risk-based decision aids.