Background

With the number of cancer survivors in the USA nearing 17 million and growing [1], there is a public health need to increase the health professional workforce that is trained to participate in their care. Key components of cancer survivorship care include health promotion, surveillance for recurrence and new primary cancers, management of chronic conditions including physical and psychosocial effects of treatment, and care coordination among healthcare teams [2, 3]. Momentum is growing to involve primary care physicians (PCPs) in the care of cancer survivors, as many aspects of cancer survivorship care fall within the scope of primary care practice [2, 4].

However, cancer survivorship care has not been widely adopted into primary care practice. Barriers include lack of knowledge and practical resources at point of care, time constraints, limited access to the patient’s oncology treatment history, and limited experience caring for cancer survivors [5, 6]. A number of primary-care focused educational approaches have been developed to address the knowledge gap, including online and in-person courses, workshops, and seminars. As described in a recent systematic review of cancer survivorship education for PCPs, relatively few educational programs have published learner outcomes and even fewer have evaluated effects on clinical practice [7]. There is a need for educators to study the impact of their cancer survivorship courses on learners and patients.

In April 2020, our multidisciplinary team launched an online, case-based, primary care focused continuing medical education (CME) course designed specifically for generalists and primary care clinicians entitled Health After Cancer: Cancer Survivorship for Primary Care [8, 9]. This course fulfills several expert recommendations for primary care-focused cancer survivorship education: (a) tailored to PCPs, (b) evidence-based approach, (c) based on learning frameworks, (d) addresses lack of clinician time via online format, and (e) evaluates learning outcomes [7]. Here, we evaluate course effectiveness and learner outcomes during the first year of the course, with a focus on learner experience, engagement, and influence on practice change.

Methods

With the goal of shaping clinician attitudes through education, our course presented emotionally sensitive case-based vignettes that highlighted the physical and psychological impact of cancer treatment. These patient cases were augmented with practical strategies and support documentation that learners can apply to clinical practice. The integration of knowledge checks and reflective statements offers the learners reinforcement as they move through each patient case.

Course development

As described in detail previously [9], we developed the Health After Cancer course by adapting the American Society of Clinical Oncology (ASCO) core components of cancer survivorship care for a primary care audience [2]. Our course differs from other courses in that rather than having subject matter experts deliver recorded lectures, we employ a peer-to-peer instructional model using a primary care physician on our team and incorporating multimedia-enriched patient cases [9]. We applied curriculum development principles from “Understanding By Design”, a backwards design approach to planning education, to ensure that all course content followed directly from our key learning objectives [10]. The key learning objectives of the Health After Cancer online course include (1) Identify key parts of a patient’s cancer history that may impact current and future health, (2) Recognize and assess the spectrum of long-term and late effects of cancer treatment on both physical and psychological health, (3) Make risk-based recommendations for co-morbidity management and preventive care according to a patient’s cancer diagnosis and treatment, (4) Provide appropriate resources and referrals for PCPs, patients, families, and caregivers, and (5) Practice strategies for communication and collaboration with a patient’s care team. To address the commonly reported “lack of time” barrier to completing continuing education, as measured in post-course evaluations, we used a succinct, asynchronous online format with a focus on real-world applicability and practical resources. The course combines instruction adapted to a primary care audience with case-based vignettes to engage clinicians and improve preparedness to incorporate aspects of survivorship care into their practice. Four animated patient cases showcase physical and psychological effects of cancer treatment while a primary care narrator offers practical resources, visit note templates, and clinical tips. In addition to case-based learning, the course employs learning reinforcement strategies such as supplemental tools for use in practice, knowledge checks, and reflective statements.

Recruitment strategy

The Health After Cancer course is currently available on two online education platforms. The first platform, called LearnMed, is a Stanford Medicine public-access platform that houses medical education modules. The second platform, American Medical Association Ed Hub, is an online learning platform powered by the American Medical Association. We recruited learners to participate in the course via marketing efforts by the Stanford CME office and the American Medical Association. Because the course is publicly accessible, we did not set eligibility requirements for learners; however, learners who claimed CME credits were required to provide additional demographic data such as degree, city, state, and country.

Post-course evaluation plan

We collected learner outcome data with an immediate post-course evaluation survey (Supplemental File 1) and a delayed follow-up survey (Supplemental File 2). The post-course evaluation survey was completed by learners who claimed CME credit for their participation in the course. This survey was designed to assess a learner’s participation, satisfaction, self-reported knowledge transfer and competence, based upon Moore’s 7 Levels of CME Outcomes Measurement, an established outcomes framework for continuing education providers [11]. The levels of outcomes measurement relevant to the Health After Cancer online course include participation, measured by registration numbers; satisfaction, measured by post-course evaluation questions; learning, measured by embedded knowledge checks; competence, measured by commitment to change survey questions; and performance, measured by follow-up assessment on practice change. Post-course evaluation survey questions included measures of satisfaction with the course material: (1) covered content useful in practice, (2) engaging and interactive, and (3) value of the topic. For each statement, learners were asked to rate their level of agreement on a Likert scale (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree). A free text question asked learners about intent to change their practice after taking the course (Supplemental File 1).

90-day follow-up evaluation plan

A follow-up survey was disseminated to learners at least approximately 90 days after they completed the course and claimed CME credit. This follow-up survey aimed to measure learner application of the cancer survivorship course to clinical practice. Follow-up survey questions included implemented practice changes, barriers to course completion and resource utilization; changes to clinical practice were evaluated with one quantitative survey item and one qualitative item (Supplemental File 2). We monitored the number of CME completions weekly and expanded our marketing efforts over time to increase course participation and broaden our educational reach.

Learner metrics

We analyzed learner metrics and interaction with the course using data that were automatically generated by the course learning management system. Data included the number of unique learners who registered for the course and the number of times each module was viewed. Because each module was composed of one video, we were able to determine the amount of time spent viewing each video and calculate the percentage of the video that was viewed in order to estimate a module completion rate for each of the four patient cases. A view was defined as any viewing session that lasted for more than 5 seconds. We calculated the proportion of views that covered at least 50%, 80%, and 100% of each case based upon the session view time and total video play time for that case. We considered 100% completion to be any view that lasted at least the full duration of the video play time or up until 30 s before the end of the video. Additional data that were summarized descriptively included the number of downloads for each of the course resource files: Clinical Considerations Guide, Communications Guide, visit note template, succinct case summaries for each of the 4 patient cases, and 3 bonus cases.

Data analysis

The primary goal of our evaluation plan was to ascertain the impact of our online course, including the presentation of patient cases as well as the provision of supplemental materials, on learners’ clinical practice. The analysis of our quantitative and qualitative data was key to understanding the connections between learners’ participation and engagement in the Health After Cancer course with their commitment to apply their learnings to practice. Therefore, we analyzed quantitative survey data descriptively using frequency tables and means for Likert-scaled items pertaining to course satisfaction, engagement, and interactivity. The frequency tables and means allowed us to visualize the magnitude of corresponding responses and average for each data set. We analyzed qualitative survey data using thematic analysis. First, we cleaned the qualitative data by removing nonsensical responses and responses with unrecognizable characters (Supplemental File 3). To determine our learners’ commitment to change their practice as a result of participating in the course, we coded all responses in Excel and generated latent themes aligned with the course learning objectives. In this process, we identified and grouped responses that had similar subject matter. Thus, the thematic analysis resulted in three latent themes: improve communication with patients, utilize course materials for developing cancer survivorship care plan, enhance collaboration with multidisciplinary team (Table 2).

Results

As of June 1, 2021, 742 learners have registered for the Health After Cancer course. Among them, 288 learners claimed continuing medical education credits and responded to the post-course evaluation survey. The Health After Cancer course was geared toward clinicians in primary care and oncology; however, learners with a variety of medical backgrounds participated in this course. Physicians with MD, DO, or MBBS degrees comprise more than a third of the learners (n= 87, 35%), and individuals with PhD, PharmD, MPH, RN, MSN, NP, BSN, and MSc degrees comprise the remainder (Fig. 1a). The course reached a global audience, with 48.6% of learners (n=140) from the USA and 49.3% (n=148) from 40 countries across 6 continents. The follow-up survey that was disseminated 90 days after course completion yielded 47 responses. Approximately one quarter of follow-up survey respondents (n=12) were physicians with MD, DO, or MBBS degrees, with the remaining respondents comprising of pharmacists, nurse practitioners, nurses, and medical students (Fig. 1a). It is important to note that our demographic data is limited to information collected in the course registration form for learners claiming CME, and therefore only includes degree, profession, city, state, and country.

Fig. 1
figure 1

Demographics of learners who completed the post-course evaluation. 1a: Learners by professional degree. 1b Learners by medical specialty. 1c Learners by geographic location

Full size image

Learner-case interaction

In the Health After Cancer course, learners interact and engage with content from four cases. In Case 1, Amelia is a survivor of childhood cancer whose cancer treatments and experience have left her and her husband with questions about what to expect as they prepare for a future and a family. In Case 2, Bob is a survivor of oropharyngeal cancer that was positive for Human Papillomavirus, or HPV. His treatment has changed his daily life and he looks to his physician for guidance with symptoms and continued health. In Case 3, Seema is a post-menopausal 60-year-old woman and a survivor of hormone receptor positive breast cancer. She, her husband, and her family have questions and concerns about her care, medications, and new normal. In Case 4, Richard is an 8-year survivor of prostate cancer. He relies on his primary care physician to help him maintain life at his functional age and guide him as he ventures to keep up his favorite activities. In addition to these cases, supplemental materials and bonus cases were provided to augment the learning experience.

Although 742 learners registered for the course, the total number of times each patient case was viewed ranged from 279 to 418, distributed as follows: Case 1, n= 418; Case 2, n= 342; Case 3, n= 313; and Case 4, n= 279. Each case had slight drop-offs in viewing over the course of the video playtime; however, later cases (Case 3 and 4) had fewer drop-offs than earlier cases (Fig. 2). While Case 4 had the fewest total views, it also had the highest proportion of views (82%) covering at least 80% of the content (vs. 60% for Case 1) and 100% of the content (77% of views for case 4 vs. 55% of views for Case 1) (Table 1). Optional course resources, such as the supplemental materials and bonus cases, that were available for download were variably used, with 105 total resource downloads.

Fig. 2
figure 2

Number of views per patient case throughout the case run time

Full size image
Table 1 Proportion of learners viewing each patient case
Full size table

Learner satisfaction

We evaluated course satisfaction in our post-course evaluation, which was completed by 288 learners who earned CME credit (Fig. 1a, b, c). Ninety-four percent of learners thought the course covered content that was useful in their practice (strongly agree, 60.8%, n= 175; agree, 33.3%, n=96; neither agree nor disagree, 5.9%, n=17; none disagreed or strongly disagreed). A similar proportion thought the course was engaging and interactive (strongly agree, 66.7%, n=192; agree, 28.5%, n= 80; neither agree nor disagree, 4.9%, n=14, none disagreed or strongly disagreed). Most learners thought the course presented valuable topics (strongly agree, 70.5%, n= 203; agree, 22.9%, n= 66; neither agree nor disagree, 5.9%, n=17; disagree, 0.7%, n= 2; none strongly disagreed).

Intent to change practice

We also assessed learners’ intent to change practice in our post-course evaluation using a free text question. After cleaning the data, 164 responses were analyzed that generated three latent themes (Table 2): Improve communication with patients (n= 67), Utilize course materials for developing cancer survivorship care plan (n= 14), Enhance collaboration with multidisciplinary team (n = 7).

Table 2 Intent to change practice: examples of learners’ qualitative free-text responses categorized by latent themes
Full size table

Learners who completed the 90-day follow-up survey (n=47) reported making several changes in their clinical practice after completing the course. The most reported change to practice was improved patient education such as sharing course resources with patients (22.2%, n= 28), followed by communication with patients (20.6%, n=26), communication with other healthcare professionals (17.5%, n= 22), diagnosis and screening (12.7%, n= 16), treatment (11.9%, n= 15), and visit organization such as using the survivorship visit note template (7.9%, n= 10). Three respondents (2.4%) indicated they did not make any changes to their practice based on the materials presented in the course. The thematic analysis for clinical practice changes reported in the follow-up survey aligned with two latent themes generated from the immediate post-course evaluation survey: improve communication with patients (n= 10) and Enhance collaboration with multidisciplinary teams (n = 3).

Discussion

We developed the Health After Cancer course as an innovative educational solution to fill a training gap for primary care physicians using an emotionally sensitive and clinically relevant, asynchronous online course. Our course is evidence-based (adapted from the ASCO core curriculum for cancer survivorship), built upon a learning framework (“Understanding by Design”), and tailored to a PCP audience—meeting several key recommendations for survivorship education programs [7]. Similar to other cancer survivorship courses with published learner outcome data [7], our learners self-reported satisfaction with the course, specifically indicating they found the content useful to their practice and the topics to be of value. As was recently reviewed [7], very few cancer survivorship courses have published data assessing higher level learning outcomes. Our follow-up survey, disseminated several months after course completion, allowed us to measure the degree to which learners applied what they learned from the course to change their practice [11]. An analysis of this data generated insights into learners’ self-reported implementation of practice changes based on the course material. These results suggest that our course increased clinicians’ motivation and commitment to change their practice related to patient-clinician communication, clinician engagement in survivorship care, and patient education regarding survivorship topics.

Learner engagement

We were able to track learner engagement with the Health After Cancer course using the learning management system in which the course resides. In reviewing aggregate learner data on interactivity with each patient case, we found that the total number of views for each case decreased from the first to the final (fourth) case. Although each case had slight drop-offs in views throughout the video playtime, there was less drop-off in the later cases. If learners tended to complete the cases sequentially, this may indicate that those who advanced through all cases were those who were more likely to remain engaged throughout the duration of each case video. Because the modules could also be viewed a la carte rather than sequentially, it is also possible that there was greater interest in the first two cases compared to the last two cases, resulting in more total views of the former. We attempted to understand this further through our follow-up survey, which asked respondents to indicate any barriers to completing the modules or the course; however, we did not receive any responses to this item. Based on anecdotal information, possible explanations for the drop-off in case interactivity and completion include the following: (1) physical learning environment not conducive to case/course completion, (2) lack of interest in the content, (3) lack of relevance to clinical practice, (4) lack of time, (5) life circumstances, (6) inability to meet educational needs, (7) course was not engaging, (8) course was too long, (8) lack of computer or Internet connectivity issues, (9) difficulty accessing the course, (10) resource availability, or (11) excessive download times.

We estimate a learner may take up to two hours to complete this course and thus can claim up to 2.0 CME credits; however, each of the four case videos is between 6 and 10 min long. The shortest video is Case 4, lasting just over 6 min, and this is also the video that had the highest proportion of views completing 100% of the video, suggesting there may be an advantage to shorter case lessons. In future iterations of this course, we may consider separating the cases to stand alone as microlearning lessons. Developing microlearning modules may result in increased viewership while delivering bite-sized, accessible cancer survivorship education to learners [12]. By applying microlearning design principles to our education, we may also address a few of the anecdotal barriers that are within our control, such as lack of time, course was too long, difficulty accessing the course, and excessive download times due to breaking down, so that it is more manageable to participate in while decreasing the bandwidth needed to access the content.

Approximately 95% of the respondents to our post-course evaluation survey indicated that the course was engaging and interactive, perhaps reflecting our use of the “Understanding by Design” principles to guide course development [10]. This also confirms that our use of animation to depict emotionally sensitive patient cases as well as the integration of interactivity components (i.e., knowledge checks) after each case appealed to most of our learners. When we expand our cancer survivorship curriculum, we will continue to incorporate opportunities for our learners to interact with the content as a learning reinforcement strategy.

Influence on practice change

A thematic analysis of qualitative evaluation data showed the importance of emotionally sensitive patient stories in shaping clinician attitudes. We found that the most common response from course learners regarding intent to change based on Health After Cancer was through “enhancements to communications with patients”. Several respondents touched on the psychosocial aspects presented in the cases, stating that they will, “Address/initiate the conversation of sensitive topics, that the patient may be reluctant to talk about” and “Be more open and empathic about patient history and to try to manage their symptoms according to prior cancer treatments.” Respondents also noted that “sometimes being an empathic listener to the patient can go a long way, as cancer survivors are prone to depression.” Though more studies must be conducted using the “Understanding By Design” learning framework, the qualitative data supports the notion that this innovative approach toward emotionally sensitive case-based learning is effective and may contribute to closing the training gap, improving clinician engagement with their patients’ cancer histories, and preparing clinicians to deliver survivorship-informed care to their patients.

Another latent theme generated through the qualitative analysis was “utilization of course materials and resources.” In our analysis, resource utilization was not as prominent as the first theme and was absent in the follow-up survey results. Similarly, the reports from our learning management system revealed that in relation to the 742 total learners registered for this course, there were only 105 downloads for the nine documents. We had expected to see greater resource utilization, especially with 94% of learners reporting that the course content was relevant to their practice. As previously indicated, barriers to resource utilization may be availability (i.e., videos not playing) and download times; therefore, we may consider providing the resources as supplemental materials post-course completion via e-mail to learners. We speculate that supplying learners with these handouts in their email inboxes may increase utilization in practice, though we would need to investigate this behavior in a future study. Another potential explanation is that learners may have been satisfied with the core course content (as indicated by the high rates of course satisfaction) and did not have a need for the supplemental materials that we provided.

The final latent theme that emerged in our data analysis was “enhanced collaboration with multidisciplinary teams.” This theme also resurfaced in the follow-up survey. A very small number of responses were linked to this theme; however, considering this course was developed for primary care clinicians to bridge care coordination after cancer, we felt it important to address. According to our demographic data on learner specialties (Fig. 1b), of the 87 physicians, the percentages of primary care physicians (26.4%, n= 23) and oncologists (33.33%, n = 29) were similar. In order to reach our target audience of primary care clinicians, we directed our advertising and identified marketing channels aimed at this population. However, despite our marketing efforts and strategies to appeal to primary care clinicians, such as including a PCP in course planning, content creation, and course narration, our attempts fell short in reaching our target audience. That said, the proportion of primary care learners in our course was higher than a prior, lengthier (estimated 10 h to complete) asynchronous online course on cancer survivorship, the American Cancer Society and George Washington University Cancer Survivorship E-Learning Series for Primary Care Providers, which was also targeted to primary care yet had only 11% of learners from a primary care background, with nearly 75% from oncology [13]. Despite reaching a relatively low proportion of primary care clinicians, our qualitative data on improved collaboration within healthcare teams, such as “cancer survivorship requires a team-based approach” and “working with the oncology team to provide primary care for my patients inclusive of their specific needs for follow up after cancer treatment”, is encouraging. We expect to expand our marketing efforts to target primary care clinicians and have already started to do so by co-locating Health After Cancer on the American Medical Association Ed Hub website in hopes that more primary care physician access and participate in the course.

Limitations

Demographic learner data reveals that the course appeals to a global audience (Fig. 1c) and diverse group of healthcare professionals and researchers at various levels of training; however, this broad participant base may have caused participant bias during data collection. We saw high rates of satisfaction with the course content on our immediate post-course evaluation survey; however, this survey was only completed by the 288 learners who claimed CME credit, which may have been a more motivated, engaged group of learners. Similarly, while all learners were invited to complete the 90-day follow-up survey, those who agreed to do so may have been more likely to rate the course in a certain way. We noticed several nonsensical responses while cleaning the qualitative data for analysis; for example, “nil,” “já fiz,” “x”; which prevented us from capturing more information related to learner behavior and implementation of changes in practice.

Another limitation of this study is the absence of longitudinal learner data. We are required to use separate platforms to deliver the content, award CME credit, and disseminate the post-course evaluation surveys, which prevents us from tracking individual learner behavior over time. Moreover, we are not able to evaluate whether a learner who completed the whole course is more likely to apply knowledge from the course in clinical practice, compared to a learner who completed only one case. Additionally, our measurement of outcomes is limited to the learner’s subjective reflections. Therefore, we are not able to evaluate Health After Cancer’s impact on health outcomes of cancer survivors at this time.

Future implications and conclusions

One year after launching Health After Cancer, a case-based, emotionally sensitive online course that aims to improve primary care clinicians’ interest in and preparedness to incorporate cancer survivorship care into practice, our early course evaluation highlights several strengths of this approach as well as areas for improvement. Satisfaction with the course was high, with 95% of learners claiming CME credit reporting that the content was relevant and valuable. The course was also successful at engaging learners, highlighting the value of using patient-centered stories to illustrate medical principles, in contrast to more traditional, less engaging lecture formats. While learners expressed an intent to modify practice based on participating in the course, and the subset who completed the follow-up survey reflected on actual effects on their practice, we acknowledge these are intermediate outcomes. The ultimate goal of survivorship education for primary care clinicians is to improve the care and experiences of cancer survivors. Future efforts should evaluate the effect of clinician education – such as the Health After Cancer course – on cancer survivors’ primary care experiences and long-term health outcomes. Demonstrating an improvement in patient experience and health outcomes may encourage the incorporation of survivorship-focused education into general medical training and primary care practice.