Introduction

Over the past decade, monitoring of electronic patient-reported outcomes (ePRO) has increasingly been considered a promising approach to enhance surveillance and care of cancer patients (Putora 2020). Treatment modalities in outpatient care vary from well-known chemotherapy regimen to sometimes very recently approved targeted therapies. All of them bear the risk of severe, potentially life-threatening adverse events, and timely detection is essential. However, research suggests that physicians often underestimate patients’ burden of symptoms and that patient-reported outcomes are superior to clinicians’ assessment reflecting severity of symptoms and side effects (Basch et al. 2009; Laugsand et al. 2010; Gilbert et al. 2015; Atkinson et al. 2016).

In the context of the current COVID-19 pandemic, ePROs have gained even more attention (Abelson 2020). Cancer patients represent a risk group for severe courses of disease (Liang et al. 2020; Cook et al. 2020; Tian et al. 2020), and every physical contact with health care facilities puts additional risk to acquire a COVID-19 infection (Gosain et al. 2020; Al-Shamsi et al. 2020). However, most cancer patients depend on continuous antineoplastic treatment to prevent progression of disease. Treatment surveillance via ePRO might help solve this dilemma by providing the opportunity of continuous monitoring while minimizing the frequency of clinic visits (Abelson 2020).

Several studies have shown impressive benefits of ePRO, such as enhanced quality of life (Basch et al. 2016), reduction of hospital admissions (Basch et al. 2016), earlier detection of adverse events (Denis et al. 2014), and even prolonged survival (Basch et al. 2017; Denis et al. 2019). In a randomized controlled trial by Basch et al., median survival among patients treated for advanced solid cancer was 31.2 months when being monitored by an ePRO-tool, compared to 26 months with standard care (Basch et al. 2017). Another randomized controlled trial by Denis et al. among lung cancer patients proves earlier relapse detection and improved survival with ePRO (22.5 versus 14.9 months) (Denis et al. 2019).

Nevertheless, evidence on successful implementation of ePROs in a real life setting remains limited, and the process of establishing them as a standard of care is moving at a slow pace (Kotronoulas et al. 2014; Anatchkova et al. 2018; Scheibe et al. 2020). In-depth practical knowledge about potential barriers is essential to avoid failure and frustration, and to enable health care facilities to implement ePRO into their daily routine successfully.

To address this demand, we conducted a feasibility study within a cohort of myeloma patients. Our main focus was to identify barriers met during the process and to investigate adherence and perceptions of patients as well as involved health care personnel. Subsequently, we contextualized our result with evidence and recommendations of a semi-structured literature review.

Methods

Study site and recruitment

During the period of September 2019 until June 2020, we conducted an observational study accompanying the implementation of a commercial ePRO system in an outpatient clinic adjacent to one of the largest hospitals in Berlin. Two oncology nurses and five doctors were involved in the project. No additional staff was employed; however, nurses were specifically trained in introducing and supervising patients using the ePRO system.

The ePRO system was offered to patients diagnosed with multiple myeloma under current treatment. Once the patient consented, one of the nurses would set up an account and give instructions on how to use the application. Patients were then included into the observational study if they provided written informed consent, were over 18 years of age, had access to an electronic mobile device or a private computer and were literate in German or English language.

ePRO system

Patients used a web-based ePRO-monitoring tool on their own mobile device or private computer. The tool included symptom questionnaires based on CTCEA (clinician-based common terminology criteria for adverse events) with graphic display and side effect alert. By a messenger service patients were able to send requests and information to the clinic. On this basis, once a week patient received a symptom questionnaire and a subsequent symptom report was sent to the clinic. The questionnaire collected information on the patient’s general well-being, disease symptoms, and treatment side effects. In addition, the application provides the option to record symptoms in the form of a diary and to send additional reports or contact the clinic directly. The responsible physicians as well as the patients themselves have access to a graphic summary of all reported symptoms when logging on to their account.

Whenever a patient reports a symptom classified as severe by the application, an adverse event message (AEM) is sent to the health care team and to the patient, respectively. Depending on the severity of the symptom, the automated response to the patient would either instruct them to contact the clinic during working hours, or to seek immediate medical help via the emergency department. At the outpatient clinic, the nurses involved are responsible to check the incoming ePRO reports and AEM on a daily basis. In case of an AEM, the nurses are supposed to contact the patient and to inform the treating physician if considered necessary.

Data collection

Patients’ data was pseudonymized and entered into an electronic database using Microsoft Excel. Baseline parameters included age, gender, duration of disease, treatment protocol (first-line versus treatment of recurrence), and ECOG status.

In the further course, patients were followed up for 6 months. Number and causes of AEM were recorded into the database at monthly intervals. For each AEM, information on the subsequent reaction of the clinic (i.e., text message, phone call, unplanned visit at the outpatient clinic or hospital admission) was obtained. In case of several reactions to one AEM (i.e., phone call followed by a visit at the outpatient clinic), all of them were documented. In cases of reactions referring to more than one AEM (i.e., symptoms connected to each other such as fever and weakness), the respective AEM were summarized.

Patient adherence was defined as the percentage of questionnaires submitted by the patient in relation to the number of questionnaires sent to the patient. For each patient, adherence to the ePRO system was determined both on a monthly interval as well as overall adherence regarding the whole study period.

At the end of the study period, perceptions of both patients and the involved medical staff were assessed in the form of a structured interview. The interview was conducted face-to-face or via telephone following a questionnaire (see Table 2). In addition, patients and staff were asked to describe their impressions in their own words. A descriptive analysis of all obtained data was carried out via Microsoft Excel. Percentages are presented rounded without decimal numbers.

A semi-structured literature search was conducted using MEDLINE via PubMed. The chosen search terms were “epro oncology” combined with the terms “implementation,” “adherence,” “benefits,” “barriers,” and “feasibility,” respectively. References of the identified papers were also considered if relevant (“snow balling”). Randomized-controlled trials and implementation studies focused on ePRO implementation in oncology facilities were taken into account and systematically checked for benefits, barriers, feasibility, and perceptions of patients and staff.

Ethical considerations

The study was approved by the ethics committee of the medical association of Berlin (Ethikkommission der Ärztekammer Berlin).

Results

Baseline data

Overall, 36 patients diagnosed with multiple myeloma were identified at the outpatient clinic. Out of these, 11 were enrolled into the observational study. Reasons for not participating included language barrier (11%), not being under active treatment (22%), patient denial (8%) and lack of a mobile electronic device or private computer (6%; see Fig. 1).

Fig. 1
figure 1

Recruitment procedure

Full size image

When asked to participate in the structured interview at the end of the study, 10/11 patients accepted and 1/11 denied due to poor health condition. The median baseline parameters are shown in Table 1.

Table 1 Patients baseline data and adherence
Full size table

Adherence

Of the 11 patients, three patients stopped using the ePRO system in the course of the study period after a mean time of 1.7 months. Reasons were concerns about data security in two cases, although one of these two patients additionally mentioned dissatisfaction with the ePRO system as an additional reason. One patient stopped using the ePRO system after being referred to a different treatment site.

Patients received a median of 4.5 questionnaires per month and a median of 26 questionnaires per patient in the course of the study period. Overall adherence was found at a median level of 95% (per protocol analysis), the median monthly adherence remained on a constant high level, showing no tendency to decline within 6 months (see Table 1 and Fig. 2).

Fig. 2
figure 2

Adherence of 8 patients continuing use of ePRO tool until study end

Full size image

Adverse event messages

Overall, the medical team received 59 AEM; 64% of patients received at least one AEM. The mean number of AEM per month and patient was 0.92, with a range of 0–5.33 alerts per month.

When investigating the causes of AEM, we found pain to be the most common cause, with almost 50%, followed by cardiopulmonary symptoms and gastrointestinal problems (see Fig. 3).

Fig. 3
figure 3

a Causes of adverse event messages in percentages (%) b Reactions to adverse event messages in percentages (%)

Full size image

Following a received AEM, the involved medical staff would in most cases decide to either send a text message or to call the patient (see Fig. 3b). Only about 15% of AEM resulted in an unplanned visit at the outpatient clinic and 10% led to hospital admission.

Patients’ point of view

Of the patients, 60% judged the required time investment as adequate, although the number of questions was perceived as too high by 60% of patients. However, 80% of patients were content with usability of the app and 70% reported no or few technical issues. Assistance in using the ePRO system was needed by 30% of patients (see Table 2).

Table 2 Results of the questionnaire evaluation for patients and staff in percentages (%) and absolute numbers (n/N)
Full size table

All patients evaluated the automated message as not helpful. Five patients remembered to have been contacted by the nurses or doctors, out of whom three patients thought this was helpful (60%).

In general, patients did not have the impression that using the ePRO tool had a positive impact on their oncological treatment (90%), nor did they feel that their symptoms received more attention by the treating physician (80%). Nevertheless, most patients (60%) stated that they would recommend the ePRO tool to other patients.

When asked for open feedback, most patients gave a positive overall statement. Some mentioned the diary function as helpful, and thought that the ePRO tool gave them an overview of their symptoms and provided them with a sense of security. The option to contact the outpatient clinic via text message was also appreciated.

On the other hand, several patients criticized the redundancy of questions and felt that the answering options did not represent their symptoms accurately. When asked about their overall impressions, two patients answered that it was “a good idea in theory, but has not yet delivered in practice”.

Perceptions of the medical team

Judgement of the medical team was overall negative and differed from patients’ perception in several aspects (see Table 2). Usability was rated unsatisfactory by all staff members, and 57.1% (4/7) reported frequent technical problems. In the opinion of 86%, the system did not display patients’ symptoms in a clear structure and did not provide them with an overview; 86% (6/7) complained about an increase of workload.

AEM were evaluated as mostly inappropriate and unreasonable by 86% (6/7), and no positive impact on treatment was perceived (86%). None of the involved staff members agreed that they would recommend the ePRO system to colleagues.

Nevertheless, the medical team still expressed their belief that ePRO systems in general could offer benefits to the oncological care of outpatients (4/7; 57%). In an open question, the following possible benefits were mentioned: easement of communication (4/7), gaining a better overview on patients’ complaints (4/7), earlier detection of severe events (2/7) and prevention of hospital admissions (1/7). In addition, one of the physicians suggested benefits in regard to pandemic situations.

Literature review

Seventeen relevant publications were identified. The respective content of these publications was analyzed for statements regarding our study questions as mentioned above (see Table 3).

Table 3 Results of the literature review
Full size table

Benefits of ePRO were discussed in 6/17 publications. Measureable clinical outcome benefits included enhanced health-related quality of life (Basch et al. 2016), earlier detection of relapse (Denis et al. 2014), decrease in hospital admission (Basch et al. 2016), and prolonged survival (Basch et al. 2016; Denis et al. 2019). Three publications reported positive impacts on communication, such as facilitation of focused discussion (Zhang et al. 2019) and identification of topics that might otherwise be missed or downplayed (Wu et al. 2016; Rotenstein et al. 2017).

Barriers of ePRO implementation were subject to discussion in 6/17 publications. Five out of these six publications found reluctance of physicians to be an important barrier of successful implementation (Wu et al. 2016; Rotenstein et al. 2017; Nordan et al. 2018; Zhang et al. 2019; Taarnhøj et al. 2020). Other frequently mentioned barriers included disruption of workflow (Zhang et al. 2019) and technical issues (Rotenstein et al. 2017; Nordan et al. 2018; Taarnhøj et al. 2020), such as failure to integrate ePRO into the preexisting electronic health files (EHR) (Rotenstein et al. 2017), causing additional effort of time (Harle et al. 2016). Confusing graphical display was additionally found as a barrier (Wu et al. 2016).

Adherence was assessed by 8/17 publications, all of which found high patient adherence (Judson et al. 2013; Denis et al. 2014; Benze et al. 2019; Friis et al. 2020; Taarnhøj et al. 2020). Adherence rates of physicians was measured in only two studies, both of which found roughly about a third of ePRO questionnaires actually being looked at by the doctors (Rotenstein et al. 2017; Taarnhøj et al. 2020). Rotenstein et al. reported that only 34% of providers within their departement would review ePRO results routinely after one year (Rotenstein et al. 2017). Similar results were presented by Taarnhøj et al., who found that only 35% of PRO questionnaires were reviewed by the physician at first consultation after treatment initiation, with physician compliance remaining on a low level (0–52%) throughout the course of treatment (Taarnhøj et al. 2020).

Patients’ perspective was taken into account in 9/17 publications, revealing mostly positive feedback. Most commonly, patients expressed satisfaction with usability and reported a feeling of reassurance (McCann et al. 2009; Denis et al. 2014). Some papers also pointed out drawbacks, one of them being frustration among patients when they noticed that PRO-results were not being reviewed by their physicians (Rotenstein et al. 2017). Also, Friis et al. found some of their participants (17%) to feel more worried about their cancer (Friis et al. 2020). Van Eenbergen et al. received positive overall feedback; however, patients criticized redundancy of questions and limitation of topics (van Eenbergen et al. 2019).

Nurses’ and doctors’ point of view was investigated by 7/17 publications. Conclusions were diverse, with 4/7 giving mainly positive feedback (Wu et al. 2016; Rotenstein et al. 2017; Benze et al. 2019; Friis et al. 2020), 1/7 giving mixed feedback (Maguire et al. 2008), and 2/7 reporting overall negative feedback (Nordan et al. 2018; Zhang et al. 2019).

Positive feedback included perceived relevance of PRO data and AEM (Rotenstein et al. 2017; Friis et al. 2020), as well as an improvement of symptom detection and management (Maguire et al. 2008). Physicians’ impression that PRO data would draw more attention to issues that might have been downplayed otherwise was also mentioned as beneficial (Wu et al. 2016; Rotenstein et al. 2017). In contrast to this finding, Zhang et al. found doctors expressing that PRO assessment would disrupt communication with patients (Zhang et al. 2019). The most important point of criticism was disruption of workflow as well as an increase of time effort and workload (Maguire et al. 2008; Wu et al. 2016; Nordan et al. 2018; Zhang et al. 2019). Out of the five publications reporting mainly positive or mixed feedback, three still mentioned concerns about workload (Maguire et al. 2008; Wu et al. 2016; Friis et al. 2020), and one described “little delay” in workflow (Rotenstein et al. 2017).

Discussion

The results of our study provide insight into some of the challenges met in the process of implementing an ePRO-tool into clinical practice.

Uptake and adherence

General attitude of patients regarding the ePRO technology is a key factor to successful implementation. In our study, we found the majority of patients agreed to participate in the ePRO procedures. Among patients fulfilling all legibility criteria, only three patients denied participation (3/14, 21.4%), which is consistent with refusal rates found in other ePRO trials (Judson et al. 2013; Wu et al. 2016; Taarnhøj et al. 2020).

However, language barrier caused 11.1% of our patients to be excluded. This topic is rarely addressed in literature, even though it is an often stated exclusion criteria (McCann et al. 2009; Judson et al. 2013; Wu et al. 2016; Basch et al. 2016; Sztankay et al. 2019). If mentioned, language barrier accounts for very few patients to be excluded (Wintner et al. 2015; Klagholz et al. 2018). One possible explanation to this might be disparities in the population structure, with our study site located in a very multicultural area in Berlin. Offering an ePRO system translated to a wider range of languages might be necessary in places with higher cultural diversity, though admittedly this presents a challenge.

Among the eight patients continuing until the end of study we found a high-level adherence without any tendency to decline, but three patients dropped out for reasons related to the ePRO tool. High adherence is congruent with previous studies measuring adherence of cancer patients using ePRO tools (Benze et al. 2019; Friis et al. 2020). In some studies, adherence tended to improve with longer intervals of symptom questionnaires (Judson et al. 2013; Denis et al. 2014). However, the “ideal” intervals might depend on the respective setting. For example, Denis et al. found higher monthly adherence among lung-cancer patients in a follow-up situation, using ePRO as a tool for earlier detection of relapse. In this context, close monitoring of severe toxicities is no longer necessary, and patients might prefer not to be reminded of their cancer more than necessary.

Overall, both recruitment and adherence of patients seemed feasible at our facility, but we also identified several barriers that limited general use (e.g., language and lack of IT). Moreover, patients expressed satisfaction with usability of the application, appreciated having a better overview of their symptoms and felt reassured. Again, these findings are similar to previous research, receiving almost entirely positive feedback from patients (McCann et al. 2009; Denis et al. 2014; Wintner et al. 2015; Wu et al. 2016; van Eenbergen et al. 2019). In conclusion, these findings lead to the impression that barriers of ePRO implementation are not primarily to be found on the patients’ side, but lack of perceived benefit certainly jeopardizes patient commitment needed for this tool.

Barriers in a real-life setting

As opposed to that, doctors and nurses at our facility expressed remarkable frustration. In their opinion, ePRO monitoring caused an increase of workload without providing any benefits. Previous evidence on perceptions of clinic staff proves to be diverse, with perceived benefits often compromised by workflow disruption (Maguire et al. 2008; Wallwiener et al. 2017; Benze et al. 2019; Friis et al. 2020). Reluctance of physicians is often mentioned as an important barrier (Wu et al. 2016; Harle et al. 2016; Rotenstein et al. 2017; Nordan et al. 2018; Taarnhøj et al. 2020).

Clear structures and guidelines in how to integrate ePRO into clinical routine might enhance chances of success. However, finding guidelines to integrate ePRO in preexisting work environments proves to be challenging. Each facility has built up their specific individual workflow, and various ePRO systems offer a heterogeneous assortment of functions. It remains unclear which of these functions are actually responsible for the improvement of clinical outcomes. For instance, Denis et al. (2014) proved that AEM sent to clinicians when patients developed signs of disease relapse are very effective in earlier detection of relapse. However, AEM might not be the most important feature in other settings. Basch et al. divided patients into groups according to whether they were computer-experienced or not. Computer-experienced patients reported remotely from home, while computer-inexperienced patients filled out ePRO questionnaires only at clinic visits. In the end, the results did not differ much, and some benefits were even more pronounced in the subgroup of computer-inexperienced patients (Basch et al. 2016). In this context, drawing attention to patients’ subjective burden of disease or gradual deterioration might be the key feature leading to the improvement of clinical outcomes.

Obviously, adapting an ePRO system to an individual setting requires knowledge and effort. Even if it is sometimes suggested that ePRO systems might help save time (Bennett et al. 2012), none of the feasibility studies we assessed confirmed this. On the contrary, additional time and work effort was one of the most frequently reported barriers (Maguire et al. 2008; Harle et al. 2016; Hans et al. 2018; Zhang et al. 2019), even though one scoping review reported no relevant effect on clinical encounters (Howell et al. 2015). Another implementation study found the need for explaining and supporting ePRO completion to be the most time consuming (Nordhausen et al. 2022). In our opinion, it is reasonable to assume that monitoring patients at home, dealing with AEM, and discussing PRO results at consultation is desirable and potentially improves health care, but potential gain in effectiveness is counterbalanced by extra effort needed. Considering the fact that preexisting conditions in health care facilities are already often compromised by shortage of personnel, additional staff might be required at least during the process of implementation. In the long run, optimized procedures and habituation might mitigate initial teething troubles and make continuation possible without additional efforts. However, in the pursuit of this objective, initial commitment and time investment is essential.

Still, the impressive benefits offered by ePRO systems justify putting effort in their implementation. Any novel drug or treatment regimen leading to similar survival benefits would be approved as standard of care almost regardless of additional expenses. It is difficult to argue why a non-pharmaceutical intervention this beneficial to patients’ survival does not deserve the same commitment that any pharmaceutical option would clearly receive.

Limitations

Our study represents a limited in time and scope (single center and patients) intervention and thus results cannot be generalized. Nonetheless, barriers found are minimally addressed in literature and scientific discussions. Also the semi-structured literature search only covered one database and included limited search terms; more detailed literature reviews are available (Howell et al. 2015; van Egdom et al. 2019; Graupner et al. 2021). The described high adherence to the ePRO tool has been found only for those continuing to use it until the end of the study, but we had drop-outs directly related to the tool, reducing factual adherence.

Regardless of the limitations, our study provides needed data about implementation barriers for ePRO, as until today real-life application outside of study settings is scarce, most likely because of the limitations described.

Conclusions

Overall, there is little doubt about the substantial benefits of ePRO monitoring to the care of oncological patients. Our research has shown that the implementation of an ePRO tool is well received by patients, who are willing to participate, presented good compliance, and expressed subjective benefits.

However, difficulties regarding the integration into preexisting workflow routines and an increase of workload leads to frustration on the part of the clinic members. As opposed to the expected effect of saving time, additional commitment is required to overcome initial challenges. Personnel support may be needed to customize the ePRO tool to the respective facility and maintain procedures.