Keywords

1 Introduction

Chronic kidney disease (CKD) carries a substantial global health burden with high associated economic costs to health systems and substantial impact on quality-of-life (QoL) [1,2,3]. Treatment options and care trajectories for CKD patients are often complex and can vary widely between patients as well as over time. Haemodialysis treatment (HD) in particular places patients under a high treatment burden [1,2,3], with some studies highlighting that patients experience confusion, anxiety, frustration or dissatisfaction with their personal experiences of the disease and care trajectories as well as important fluctuations in their physical health [4].

Furthermore, the intense schedule of HD can also have substantial impact on families and social relationships [5]. One of the key decisions that is required is the choice of how HD is delivered – vascular access (VA). Most importantly, this key decision must be made at a time of illness, in a pressurized situation, potentially with limited time for professional input. Currently this is often delivered in an environment with time-limited consultations, paper-based generic information, and unstructured internet information. This is time-consuming, inefficient and can confuse patients. In addition, there is no routine mechanism to collect real-time patient experiences or outcomes.

Given the widely varying and unpredictable patient experience, it is not surprising that getting a reliable method of assessing and collecting patient-related outcome measures (PROMs) has proven difficult. PROMs focusing on Quality of Life (QoL) are clinically important as many of the key decisions regarding care are subjective and rely on patient input. Any additional support for patients to help them manage their care would be a major advance [4, 5]. Improving the integration of QoL into clinical care presents several hurdles. A fundamental requirement includes a mechanism for data collection that can be reliably used by patients during dialysis. There is little data in the HCI literature on how app-based technology for patient data collection compares to traditional paper, nor on how this can be evaluated in clinical situations where it is most required. In particular, patients on dialysis have particular accessibility limitations due to the impact of their underlying condition and intensive ongoing treatment (e.g. diabetes and reduced visual acuity) [6, 7]. Furthermore, collection during dialysis introduces situational impairments as patient movement is constrained. As such, multi-disciplinary research is required between haemodialysis clinicians and HCI designers.

Handheld tablet computers are promising data collection tools as they are lightweight and, in comparison to paper, support direct entry rather than requiring transcription hence have the potential for more reliable input. However, the design of a tablet solution for this population and environment is not clear and there are concerns with the introduction of shared devices such as transmission of infections amongst a vulnerable cohort. This design also poses an interesting case study as standard co-design methods are challenging when patients have considerable health issues and their treatment already entails a considerable lifestyle burden.

This paper reports on use of a tablet-based tool to support longitudinal developmental and validation studies of a novel QoL measure (the Vascular Access Specific Quality-of-Life, VASQoL) [8]. A set of design recommendations was developed from medical and HCI domain experts and a proof-of-concept portal proposed [9]. Previous literature and related work have emphasized that patient portal systems suffer if they do not meet the expectations and needs of stakeholders [10,11,12]. A multi-stakeholder co-design approach was performed including patients in design and evaluation, and aimed to answer the following research questions:

  1. 1.

    What are the benefits and disadvantages of digital tablet-based data collection over paper for the in-hospital HD population?

  2. 2.

    Is a two-stage approach in which the system is designed with multi-stakeholder co-design before testing with patients suitable?

  3. 3.

    What situational, accessibility and usability issues need to be considered for development of in-hospital systems for HD patients?

Previous research developed a set of design recommendations from multidisciplinary domain experts (comprised of medical, informatics and human computer interaction (HCI) academic experts) [9]. This paper reports the iterative development of a portal with a multidisciplinary group of experts followed by an in-hospital study with patients during dialysis. Qualitative feedback from patients and investigators’ observations were collected and analyzed along with assessment of the usability of the system using the System Usability Scale (SUS) [13]. Given the lack of previous literature of usability studies in clinical settings with hemodialysis patients, it is currently not clear what the benefits and barriers to tablet-based data entry for patients are and how patients would respond to tablet-based entry compared to paper.

This paper presents the first case study of design and development of a portal conducted with a challenging user group in a hospital setting with the direct aim of supporting their clinical journey. The research contributions to the medical and HCI communities include:

  • a novel technology in heterogeneous population, which warranted wider and deeper analysis;

  • co-design approach produces a system which met needs and expectations, securing stakeholder interest;

  • demonstrating engagement and activation of patient users;

  • confirmed the benefits of digital data collection and considerations for alternatives where appropriate;

  • identification of situational impairment and population accessibility issues.

2 Background

Haemodialysis (HD) is an intense, intermittent procedure typically performed three times a week, with sessions lasting four to five hours. A key limitation to HD is the mechanism to access the blood, their vascular access (VA). VA must allow regular cannulation with the insertion of needles to draw and return large volumes of blood via the dialysis machine. The three most common methods of VA are a fistula (surgical connection in the arm between the vein and artery, to be used for cannulation), a graft (plastic tube joining an artery and vein in the arm or leg, also used for cannulation), and a catheter or “line” (long plastic tube in the chest or groin, that hangs outside the skin which is connected to the machine). Each varies widely in the ease of insertion or creation, the practical use, the implications on regular lifestyle, and the frequency and severity of complications. Complications from VA are the leading cause of hospital admission and the leading modifiable cost of providing care for patients with kidney failure.

HD treatment is life-prolonging, but it is recognized it places a great burden upon the patient alongside their chronic condition. HD has been likened to an airplane flight, with busy periods at start and end of the activity with safety procedures, checks and actions to be completed [14], and a long period in between of restricted movement and activity. Outside of treatment, patients must manage their health, endure restrictions in diet, fluid intake, activities and monitor their vascular access for any irregularities or complications, alongside everyday life. Treatment can vary greatly based on patient characteristics and the options available, leading to dissatisfaction and frustration [4, 15].

To better understand and improve the patient experience and outcomes, routine collection of patient-related outcome measures (PROMs) via a patient portal was proposed in prior work [9]. Patient portal technologies previously deployed in clinical settings are discussed in the following section.

3 Related Work

There has been increasing recognition within the HCI (Human-Computer Interaction) literature that hospitalized patients are poorly served with supportive mechanisms such as facilitating patient-provider communication or accessing and managing health information. For example, one report highlights hospital patients wish to be engaged in tracking their health collaboratively with healthcare providers but lack the appropriate tools to do so [16]. Patient portals have been tried but have also occasionally failed to secure engagement from both patients and providers [10,11,12]. Research into patient engagement has shown patient characteristics (such as age, ethnicity, health literacy, etc.) strongly influence their interest and engagement [17], while the technology of patient portals itself presents barriers to engagement. Technology-related barriers in a frail population include a lack of experience or feeling uncomfortable using technology, difficulty accessing technology and a lack of trust in technology [12, 18]. It has been previously suggested that patients’ preference for in-person communication and the lack of perceived need for patient portal use currently present the most important and unquantified barriers [18]. The patient-provider relationship is often discussed in the literature, as patients value this relationship and are often concerned systems that facilitate communication will replace or impact on it [11, 12, 16]. This is often met with the recommendation that the technology should always support the relationship and not replace it. It has been suggested that technology-based interventions can support dialysis patients, but there should also be availability of patient peer and provider support where technology is inappropriate for the individual [19]. Perhaps most importantly, some studies have suggested that patients do not necessarily perceive a need for patient portals [10, 11]. The failure of data input to influence clinical decision making can lead to disengagement from the technology. Conversely, when the features of the patient portal align with stakeholders’ needs and functionalities, then engagement and endorsement can be sustained [11]. Without an objective assessment of these needs, there is a risk that systems’ design will fail to meet user expectations [10].There are few design guidelines in this sphere in the literature, and thus any new work detailing functional and non-functional design requirements for patient portal systems is important both from an HCI and Medical Informatics perspective [9, 10].

Co-design is an effective methodology to gather design requirements. There are two common approaches – top-down and bottom-up. The typical top-down approach of adapting existing systems designed for healthcare providers results in systems inappropriate for inpatient use, a flaw that may be addressed by employing a bottom-up approach instead, where patients are the primary stakeholders [17]. Other studies demonstrate the benefit of capturing differences in goals and expectations between stakeholder groups [20] and producing a more widely accepted and person-centered system [21], with engagement from various stakeholders increasing over time. One extension of this is to utilize a multidisciplinary team of stakeholders, producing a system that requires fewer future redesigns and wide acceptability amongst stakeholders [21]. However, this theoretical approach has not been evaluated in a real-world setting. Co-design methodologies can also produce more person-centered systems, rather than an inappropriate “one size fits all” approach [19]. This was demonstrated in dialysis patients where the differences between individual patients meant no single solution or approach was sufficient. This is similar to other subpopulations of patients (i.e. elderly, low-income or those situationally impaired during treatment) who encounter accessibility issues in inappropriately designed systems [12, 19], which can often be remedied with design considerations and provision of alternatives e.g. audio output alongside text.

4 Methodology

This study followed a case study design [22], detailing the patient portal development and deployment within the context of a HD patient population. Case studies constitute an established research methodology within psychology and sociology but have been appropriated by other disciplines such as law, medicine, and political science. They are recognized as a qualitative approach where researchers explore one or more bounded systems (i.e. the case or context) over time [22]. A case study design was selected as the complexity of the case (i.e. a patient portal deployment with HD patients during treatment) warranted a deeper understanding and investigation, and the collection of various data from multiple sources allows for much richer design requirements and considerations for the system in question and others.

This study was completed in three parts. The first consisted in the iterative development of a patient portal with domain experts [9], collecting qualitative feedback to elucidate a refined set of design requirements. The second sought to evaluate the patient portal, through qualitative feedback from patients alongside a usability evaluation. Finally, the third part enlisted study coordinators to provide qualitative feedback based on their observations during the study.

These three parts are explained in detail in the following sections on participant recruitment, data collection and data analysis.

4.1 Recruitment of Participants

Expert Consultations.

As part of this study, a multidisciplinary steering group (MSG) was convened, consisting of medical professionals and senior academics, with expertise in nephrology, vascular and transplant surgery, Medical Informatics and HCI. Seven domain experts provided feedback and further design requirements for the patient portal during thirty-three (n = 33) regular meetings between February 2019 and November 2020. Five experts were medical professionals, while the remaining two were senior academics with expertise in Medical Informatics and HCI. Medical experts were able to advise on what was required in practice and how to integrate the patient portal into routine care with patients. The academic experts provided expertise on system design, development, and implementation. The details of participants’ expertise are provided in Table 1.

Table 1. Domain expert professions, expertise and sex
Full size table

Patient Participants.

Ethical approval for this study was provided by the University of Strathclyde Computer and Information Sciences departmental ethics committee (ID 1061) and the Greater Glasgow and Clyde (NHS GGC) health board (GN19RE634). Informed consent was obtained from all patients prior to participation.

A patient portal [9] was used to collect data for a validation study of a vascular access specific quality-of-life measure (VASQoL) [8] for patients requiring HD. This provided an opportunity to evaluate the system with patients in a clinical setting.

A quota sampling technique was employed for the recruitment of patients to complete digital questionnaires and cognitive interviews, with the intent to recruit a diverse population in terms of age, primary renal disease, vascular access history and mix of vascular access modalities. Inclusion criteria for the VASQoL study included (i) patients with chronic kidney disease and (ii) undergoing or about to undergo regular HD treatment.

Participants who met these inclusion criteria were approached and recruited from five regular dialysis units in the NHS GGC health board to participate in the study over a 6-week period. Patient participant numbers and characteristics are described below for both forms of evaluation.

Patient Usability Evaluation.

A total of 26 out of 101 patients (25%) using the Patient Portal for quality-of-life data collection provided an SUS evaluation [13]. 35% were male (9/26) and patient ages ranged from 28 – 85, with 58% under 65 years of age (15/26). Half of patients (13/26) were in their first year receiving haemodialysis (HD) treatment, with two pre-HD and the remainder having received HD for over a year. There were 2 occurrences of incomplete data where patients did not provide their occupation, but otherwise the collected data was complete (Table 2).

Patient Comments and Semi-structured Interviews.

Of the 26 participants in the SUS evaluation, just over half provided written feedback via a comment on the SUS form (14/26). An additional nineteen patients (n=19) were interviewed as part of the validation study and provided feedback on their experience using the Patient Portal as part of the validation study.

Researcher Interview.

The clinical research fellow who conducted the study was also interviewed. The research fellow had ten years’ experience of working with HD patients with some prior experience collecting data with traditional paper forms but no prior experience of working with a tablet device.

Table 2. SUS evaluation participant characteristics
Full size table

4.2 Data Collection

MSG Group Sessions.

The MSG met monthly in-person from February 2019, pausing in March 2020 as result of the COVID-19 pandemic, before continuing meetings fortnightly via Zoom from April 2020. A total of 33 meetings occurred in this period and meetings lasted between 60 to 90 min. During these meetings, the patient portal prototype was discussed and demonstrated, with participants able to view the system in-person or via a web app version prior to meetings. These sessions were audio recorded and transcribed, alongside contemporaneous notes which were summarized and distributed to all members of the group after each session. The continuous refinement of the Patient Portal ensured the system met the expectations of clinical expert stakeholders and aligned with clinical practices.

Patient Feedback and Evaluations.

The Patient Portal was used to complete QoL measures regularly during their regular dialysis treatment. This required patients to access the Patient Portal via an Android application on one of two dedicated Samsung Galaxy Tab A tablets. The Android development environment was decided early in MSG meetings due to ease of app development and deployment, and the 10.1-inch screen size compromised screen size for viewing the interface and ease for patients holding the tablet with one hand. The clinical researchers delivered the devices to the patient during HD treatment and supported patients if required. Patients were required to complete the following three tasks:

  1. (1)

    update their vascular access modality and dialyzing status.

  2. (2)

    complete the QoL data collection.

  3. (3)

    log out and leave feedback if appropriate.

The three questionnaires (Short Form 36-Item Health Survey [23], EQ-5D-5L [24] and the VASQoL measure [8] under validation) were accessed via three separate buttons from the main menu, with only the relevant questionnaire accessible according to the scheduling of reporting. Other non-relevant questionnaires were made inaccessible until required (e.g. the SF-36 was not available if the latest submission was completed within 25 days of the current date, as the questionnaire is designed for monthly use). The data from the QoL questionnaires was not analyzed as part of this work.

Patients were asked to participate in the SUS evaluation upon completion of their final VASQoL study visit and final use of the patient portal, having used the portal up to four times over six weeks for QoL data collection. The SUS was used to measure system usability [13, 25] and the original questionnaire and questions were not modified. Paper questionnaires were chosen over digital ones to reduce the burden of participation for patients. The clinical researchers distributed the SUS to patients and aided with comprehension or acted as a scribe for participants where appropriate (e.g. writing arm being used for cannulation during dialysis, impaired vision, etc.).

Patients were also encouraged to record any comments or feedback they felt was important about the Patient Portal in a blank space below the SUS questions on the paper questionnaire.

Qualitative feedback was gathered from a separate cohort of patients as part of the VASQoL study. Data was collected by clinical researchers (two medical professionals with extensive experience of HD) thus avoiding patient contact with additional individuals outside those providing their treatment. Social distancing guidelines were adhered to throughout including limited access to hospital facilities during national restrictions in response to the COVID-19 pandemic. This also limited the number of patients they were able to recruit for SUS completion during HD sessions. However, it is widely accepted that the SUS measure is valid with smaller sample sizes (recommendations for at least twelve participants) [26].

Researcher Interview.

The questions sought to elicit their experience working with patients and collecting patient data in paper and digital formats, alongside their views of the Patient Portal and their observations of patients’ interactions with the Patient Portal. The 41-min interview was conducted remotely over Zoom, was audio recorded and subsequently transcribed.

4.3 Data Analysis

Thematic Analysis of Qualitative Data.

Transcripts and notes from MSG meetings and patient and researcher interviews were analyzed using the health information systems quality assessment framework reported in [27], which is derived from DeLone and McLean’s model of quality in information systems [28]. The model consists of six dimensions for ensuring information quality in health information systems, with potential issues, solutions and benefits provided for each: (1) eHealth information system quality, (2) information quality, (3) information usage, (4) user satisfaction, (5) individual impact and (6) organizational impact.

For example, the first dimension, eHealth information system quality is defined as the performance of information processing. Potential issues include a mismatch between system functionalities and clinical work processes or ambiguity of coding standards and errors or variability in assignment of codes. The proposed solutions to these issues are co-design of systems with stakeholders to closely match clinical practices (i.e. regular MSG meetings prior to deployment) and automated validity checks. The latter is a theme discussed in the following results section, highlighted by the clinical research fellow. Previous work within this setting [9] used this relevant framework in thematic analysis, and it was thus used to allow for consistency and comparison. The thematic analysis was completed in separate steps at each phase of the study. Transcriptions of meetings and interviews were indexed and coded before charting of codes in respect to the six dimensions detailed by the framework [27]. Finally, themes were synthesized from the charted codes, providing insight into the impact of the Patient Portal on treatment and patients and new or refined design requirements.

System Usability Scale (SUS) Quantitative Data.

The SUS questionnaire [13] data was used to calculate an overall average usability score and averages for individual questions as well, to allow for insight into the different aspects of the SUS questionnaire and how patients responded to these in respect to the Patient Portal. For example, the second question “I found the system unnecessarily complex” is of relevance to a system that does not wish to impose further burden upon a high-treatment burden population such as HD patients.

5 Results

The results of this work are described as follows: (1) SUS scoring, (2) thematic analysis of patient and researcher interviews and (3) refined set of design requirements for the Patient Portal.

5.1 System Usability Scale (SUS) Scores

The overall average usability score was 86.9 (range 72.5 and 100 / 100) which can be considered as a “good” score [29]. Figure 1 shows average response score by question and is important to note for Fig. 1 that odd numbered questions (Q1, 3, 5, 7, 9) are scored low to high, with 5 being the highest score possible and 1 the lowest. The opposite is then true for even numbered questions (Q2, 4, 6, 8, 10). For example, Q3 has a very high average score of 4.8 and Q4 a low average score of 1.2 but this indicates that patients found the system easy to use and did not think they required support from a technical person (Q3 and Q4 respectively).

Fig. 1.
figure 1

Average score by question

Full size image
Fig. 2.
figure 2

SUS score by participant

Full size image

Figure 2 shows SUS score by patient, with the minimum score of 72.5 placing the system in the “high” acceptability range as proposed by Bangor et al. [29]. While a small number of patients found the system presented a challenge and was considered “unnecessarily complex” (2/26 agreed or strongly agreed with statement Q2) or required prior learning before use (2/26 strongly agreed with Q10), the tablet-based system performed well and was of an acceptable standard to most patients, suggesting the co-design process was successful in producing a system which met the needs and expectations of stakeholders.

5.2 Thematic Analysis of Requirements for Patient Portal

Following the health information systems framework derived from DeLone and McLean [27, 28], design requirements were elicited from three sources: expert feedback in early iterative design phases, patient feedback via comments and interviews following study implementation and researcher observations during implementation. Findings are described according to the six dimensions of the framework.

T1: eHealth Information System Quality: Digital vs Paper.

When asked during interviews if they would prefer paper alternatives to the digital Patient Portal, most patients preferred the tablet-hosted questionnaires (11/19) or had no preference (5/19), while three would have preferred paper. Both the researcher and patients noted the completion of the tablet-based questionnaires was easier and more feasible than using pen-and-paper during dialysis sessions. This is an important and previously unidentified observation as dialyzing with a fistula − particularly if in the dominant hand − makes writing difficult whilst receiving dialysis treatment but it did not limit the use of the Patient Portal. However, it was clear from both the patient and researcher interviews that traditional alternatives should be provided for those who may be inexperienced or unwilling to use technology. This suggests that while there are benefits to digital PROM data collection for this population, there is a need to provide traditional alternatives when appropriate [19, 30, 31].

“We had some trouble at the beginning but actually its quite good, a good thing to use it. I really liked it, I liked to work with the app or with the tablet.” – Researcher.

Overall, the interviews revealed patients found the Patient Portal to be usable. This reflects the results of the SUS evaluation, where scores indicated the Patient Portal was “easy to use” and patients did not think the support of a technical person was required to use the system. There was discussion amongst patients that it may be easier for younger and more experienced patients, but some inexperienced patients also praised the ease of use of the Patient Portal, as did the clinical researcher.

“I think it’s easier to place a tablet on your legs and use a pen or stylus, even with your non-dominant hand, you can do that…So I think it’s much more convenient to use a tablet, especially for the one-handed patients.” – Researcher.

“I personally would prefer to do it on the app [Patient Portal]. And for people who, if you are going to do the questionnaire for people who are on dialysis it is actually quite hard to write. Some people have their fistula in their dominant hand, I don’t fortunately, but even just writing can be awkward but some people are a bit funny about computers. So I don’t know, you maybe have to do a bit of both.” – Interview P8.

T2: Information Quality.

The researcher interviews revealed further benefits of the digital system over traditional data methods. Firstly, the validation processes of the Patient Portal reassured the researcher that any completed questionnaires were complete and automatically stored securely, mitigating the risk of missing or incomplete data from human error (i.e. incomplete questions only detected after participant has completed study or transcribing paper responses to digital formats).

“And the other thing is the feedback, if you miss a question, it doesn’t store…for paper forms, I won’t realize until they missed a question or something…” – Researcher.

Secondly, there was also a common theme of independence amongst the patient comments provided, praising the ability to complete the quality-of-life questionnaires independently and provide honest responses. These comments suggest patients can be uncomfortable discussing their health and quality-of-life with others or feel unable to provide honest answers. Thus, the ability to self-complete the quality-of-life measures via the Patient Portal provided a “safe space”, with no pressure from other individuals to respond in a certain manner. This positive feedback suggests that provision of systems like the Patient Portal encourage patient activation and engagement in their care, which would otherwise be difficult to achieve through purely direct communication with their healthcare team.

“I really enjoyed using the tablet system. I also preferred being on my own to do it so I could put honest answers.” – SUS P3.

“I like being left to complete it. I feel I can be more honest than if I am asked a question directly.” – SUS P6.

This important aspect of patient feeling better equipped to disclose sensitive information to a “computer” has also previously been highlighted in other work on computer-mediated patients’ medical questionnaires [32,33,34,35,36].

T3: Information Usage: Perceived Value of Engagement.

The theme of communication between patient and healthcare provider was identified in patient comments, with patients indicating they wished for staff to review their responses. However, while there was potential for the Patient Portal to support patient-provider communication, it was of little value to patients if their responses were not reviewed. These findings reflect those of Absolom et al. [30], where the perceived value of an intervention and collection of patient-reported outcome measure (PROM) data was doubted by patients when data was not referred to during clinical counters.

“Useful for nightshift or twilight shift to communicate with doctors - no use if nobody looks at it.” – SUS P15.

“I would like the VA [vascular access] team to know my answers.” – SUS P16.

While both this study and the VASQoL validation study did not utilize PROM data clinically, a clear sentiment was reported by patients that they only found benefit in reporting data through the Patient Portal where it is viewed and utilized by healthcare providers. This utilization of data will need to be visible in future implementations, through referral in discussions or other means to retain engagement from patients.

There were opposing comments, notably one patient felt “perfectly able” when communicating with healthcare providers and were the only participant to respond they strongly disagreed that they would like to use the system frequently. This suggests for patients who are confident in their ability to communicate and discuss their healthcare, interventions such as the Patient Portal are seen unnecessary and as a possible hinderance to their patient-provider relationship and communication.

“I feel I am perfectly able to communicate with nurses/doctors when I need to. I am also quite able to understand what is being said to me when discussing my health.” – SUS P4.

T4: User Satisfaction: Physical Accessibility.

Both patients and the researcher enjoyed using the system during the study. There were accessibility obstacles to overcome early on during the study, notably concerning patients’ ability to utilize touchscreen input.

“What made a difference, a huge difference, is using like a pen [stylus]. They are not that precise without a pen. They sometimes miss a field.” – Researcher.

“Awkward because in dominant hand but much easier than writing - difficult to add written comment with non-dominant hand - a voice recognition function could help with things.” – SUS P14.

Considerations were made for these accessibility issues during the initial phase, as clinical experts provided this insight. Observations by the study coordinators highlighted the scale of the issue of touch input and HD patients. Decreased sensitivity or sense of pressure in patients’ fingers, credited to carpal tunnel syndrome symptoms or neuropathy [6], appeared to result in incorrect gestures being registered and the system providing an incorrect response to the intended input (i.e. patients press on elements such as buttons for a longer length of time and the touch gesture is read as a “long press” instead of a click event). This caused frustration amongst patients and prevented them from completing the tasks required of them without difficulty. Immediate action was taken to remedy this by providing styluses alongside the tablet devices, which improved the touch input and accuracy of patients input.

Another common barrier was the impaired vision of patients, with the clinical research fellow required to support those unable to view the tablet and user interface clearly. Impaired vision can be common in this population, especially in diabetic or elderly patients receiving haemodialysis long-term [7].

“I totally underestimated, there are a lot of visually impaired patients.” - Researcher.

Therefore, the addition of alternative output and input methods (e.g. text-to-speech and speech-to-text) should be considered and may also be well-received by other users i.e. those who experience issues with touch input.

T5: Individual Impact: Activating Patients.

Patients highlighted how the Patient Portal and the quality-of-life questionnaires caused them to consider their healthcare and their role. There was a request for the addition of further information on how to leave comments following questionnaire completion and inclusion of a question to elicit patient preferences.

“Would like to be able to expand on other aspects of care or problems. Instructions of how to leave comments at the end.” – SUS P14.

“I think adding…asking a question that sticks in your head what is the preference of the dialysis patient. I mean, at the end of the day it doesn’t fall into the preference because this is your lifeline. If this one fails you need to end up with this one.” – Interview P3.

The earlier theme of providing honest responses also supports this activation of patients, as they feel they can provide honest answers and engage with their health independently. There was a request for better explanation of some questions, which should be considered carefully in order to continue facilitating the independent completion of the questionnaires. This also connects back to the usability of the system, where the need for explanation of a question or instruction suggests the support of a technical individual is required and reduces system usability.

“I liked being able to fill it in and then have people ask me about it. I don’t like bringing things up myself. I don’t talk about it much.” – SUS P8.

“Most relevant to me are the health questions. Fill in the vascular access one if I have problems (haven't had with this line).” – SUS P21.

T6: Organizational Impact: Facilitating PROM Collection.

The Patient Portal proved to be an effective and usable method for collecting patient-reported outcome measure (PROM) data from patients, praised by both patients and researcher. There were benefits over paper data collection (e.g. accessibility for dialyzing patients, validation of data and reducing risk of human error) but considerations should be made for those who may not wish to engage with digital methods or are unable to. This population is typically older [37], and while there is an expectation that the prevalence and familiarity with technology will grow with time, this subpopulation of users should be supported, either through the accessibility of the system or by providing alternatives [19, 30, 31] e.g. pen-and-paper if requested or providing support through scribing.

“Like I said, there are some patients who just can’t do it by themselves. They just have no experience.” – Researcher.

5.3 Formal Design Requirements

A set of formal design requirements was collated from all three sources: (1) iterative review and feedback from experts, (2) patient usability evaluations and interviews, and (3) researcher observations of the system implementation. They are classified as functional and non-functional, the former describing what a system will do and the latter how it does this [38]. These can also be understood as what makes the system useful and what makes it usable.

Expert feedback is described in four distinct phases which occurred during iterative development and feedback with the MSG: (1) Identification of Core Functionalities, (2) First Refinement, (3) Second Refinement and (4) Final Refinement. The first confirmed the essential functionalities and purpose of the Patient Portal. The second phase and third phases built upon this by incorporating elements to improve the user experience and improve performance of the system while the fourth the phase consisted of final refinements to ensure the system was ready for implementation ahead of the VASQoL study.

While most of the design requirements gathered from iterative development with experts were implemented, some functionalities were not included in the version of the Patient Portal used for hosting the QoL questionnaires for the VASQoL study and usability evaluation. The priority in development was to ensure core functionalities that were identified early were implemented robustly before the addition of later functionalities. For the VASQoL study, this required the quality-of-life questionnaire data collection and the accurate capture of clinical events and changes in vascular access as required of by the study. Others were not implemented due to feasibility and time-constraints, namely multiple language availability.

After the commencement of the VASQoL study, it became clear some emerging design requirements were of high priority and resolving these were critical to the patients’ effective and continued use of the system. Early observations reported that dialysis patients struggled with touch gestures using the tablet devices, with a reduced sense of pressure or sensitivity in their fingers impacting their ability to tap buttons onscreen (i.e. too much pressure indicated a long-press gesture, highlighting the text of the button rather than registering a click event as intended).

Fig. 3.
figure 3

Screenshots of EQ5D5L layout change on orientation, highlighting offscreen positioning of navigation buttons

Full size image

To avoid interrupting the study and increasing frustration for patients, rubber-tipped styluses were acquired and provided alongside the tablets for the remainder of the study. Other modifications to reduce system complexity and patient frustration included the disabling of the user feedback functionality (which prompted patients during logging out to leave feedback) and modifying the size of the EQ5D5L user interface elements so all content was available onscreen regardless of, device screen orientation. In this case, patients were disorientated when navigation buttons were not visible in landscape orientation without scrolling (see Fig. 3). This may seem easily resolved by rotating the screen to the portrait orientation, but for a patient dialyzing with a fistula or graft, they are unable to both hold the tablet and touch the screen with one hand and rely on tablet being positioned upright to interact with it.

Otherwise, the systems key functionalities and user interface elements remained unchanged for the duration of the study (Table 3).

Table 3. Formal design requirements and sources, in chronological order of identification. (Type: F = Functional, NF = Non-Functional).
Full size table

6 Discussion

We sought to produce a haemodialysis patient portal with the involvement of domain experts, building on a previous determined design [9, 17], and tested this with patients in a hospital setting. This study sought to determine if a two-phase multidisciplinary approach could produce a system appropriate for implementation into a real-world setting i.e. during HD treatment in a hospital environment. This was achieved by following a case study methodology, consisting of iterative developments closely supported by experts until the Patient Portal was robust enough to collect patient-reported outcomes and implement into clinical practice with patients. The system was evaluated with patients, achieving an above average SUS score and gathered rich design requirements from both patient feedback and investigator observations. The in-depth thematic analysis of qualitative data supplemented the quantitative SUS scores and the framework utilized in previous work with domain experts [9] proved to be suitable in this work.

The delivery of the QoL measures digitally via the Patient Portal benefitted most patients, overcoming situational impairment where traditional paper-and-pen questionnaires would have been difficult to complete. The researcher also noted the validation of the digital data collection reduced human error and streamlined the process. However, observations also confirmed that younger patients were often more comfortable and adept at using the tablets than their older peers, with patients also aware others may simply chose not to engage with the technology due to personal preferences. While the growing prevalence of technology is often cited as an eventual solution to this issue [31, 39, 40], conventional alternatives should be provided alongside the digital options to prevent patients from becoming excluded from healthcare [19, 31].

A highly usable Patient Portal resulted in the engagement and activation of patients, promoting a sense of independence, and providing a private space for reporting their health and satisfaction with treatment. Interestingly patients reported that they felt they could be more honest via the tablet app than in a face-to-face conversation and felt it was a way to initiate discussion, confirming previous findings in the sphere of computer-mediated patient medical questionnaires research [32,33,34,35,36]. The perceived value of the system indicates it met the needs and expectations of patients [10] and was also a motivator for engagement for both patients (as demonstrated by their feedback) and clinicians [11], which has been difficult to secure with similar systems as noted in the section on related work.

While this work did not utilize patient data or influence treatment in any manner, it was clear from the patient feedback that the system will need to demonstrate this value or risk losing patient engagement, as made clear by patients’ feedback. Systems such as the Patient Portal need to acknowledge patient input and demonstrate engagement from the other side, such as read receipts of submitted data, where an action in response may be delayed e.g. follow-up appointment with consultant. Implementation of functionalities such as this may reassure patients their input matters and prevent perceived value and engagement deteriorating.

The positive reception of the Patient Portal through implementation with HD patients showed clear support for future work in this field. This case study of the Patient Portal evaluation with HD patients during treatment identified unique accessibility issues within this user population. This included an example of situational impairment, already highlighted by Mishra et al. [16], which in turn can lead to patients preferring horizontal orientation for the tablet devices and identifying issues with the patient portal user interface. Considerations were made for such issues in selection of a suitable device and the design and layout of the user interface but still required refinement to improve the accessibility of the system with HD patients, such as adaptive layouts with orientation changes and use of a stylus to overcome touch sensitivity difficulties. While some actions were taken during the study to remedy this (e.g. the introduction of styluses), the Patient Portal will need to take these issues into consideration in future iterations, such as accounting for longer presses to achieve a click event or ensuring the shift in screen orientation does not result in additional actions to complete tasks (i.e. scrolling down to view offscreen buttons). Other condition-specific accessibility issues were also captured, including vision impairment which is common within this population. These findings will hopefully inform future work with this population and demonstrate the benefits of the in-depth analysis and description this case study has produced.

While we believe our methodology was appropriate and sufficient, this study has some limitations. The study was conducted under lockdown and other Covid-19 restrictions during the global pandemic in 2020 and great care had to be taken for patient safety as chronic kidney disease patients are classed as vulnerable [41]. This prevented a non-medical researcher attending the medical facilities, so data collection was reliant on healthcare professionals already working in the hospital. A single usability measure was employed as clinicians felt additional measures would have placed an excessive burden upon patients and the researchers during an already difficult period. The case study was conducted in only one setting, replication studies are planned as part of future development cycles.

Considerations for future work include further refinement of the existing system following this evaluation and implementation into routine practice, potentially at national and international levels. Most importantly, piloting this within a routine clinical setting such as monthly haemodialysis clinic reviews will be important as where there is lack of perceived value, the intervention is less likely to become normalized into routine practice [42, 43].

Further work with HD patients to address and resolve barriers to engagement and use of the Patient Portal is also required, notably those arising from situational impairment and condition-specific challenges, such as vision impairments and touch input difficulties. The design requirements elicited in this work will provide direction for further refinements of both this system and similar technologies.

Overall, the usability evaluation of the Patient Portal produced results indicating the system is usable and of “good” quality, with an average SUS score of 86.9. This score is supplemented with positive feedback from both patients and a clinical researcher familiar with the domain.

To our knowledge this work is novel and demonstrates the successful deployment and evaluation of a co-designed patient portal with a patient cohort marked by treatment and disease burden, comorbidity and age, within a clinical setting. This case study with HD patients using the Patient Portal during their regular dialysis treatment also provided an effective evaluation and yielded rich and important design requirements to consider, with data gathered from multiple sources. These insights and considerations are required to produce a system fit for purpose and accessible by its target end-users [44].

7 Conclusion

Our multi-stakeholder co-development method led to a functional application that facilitated completion of a digital PROM study that was usable by a comorbid population of patients, as evidenced by above-average SUS scores despite the challenging use environment. Researcher observation and patient interviews highlighted areas for review such as need for a stylus due to physical limitations with touch screen for this population and specific design issues such as their difficulty in rotating a tablet during hemodialysis. Patients overwhelmingly preferred tablet input over paper, primarily because of ease of entry and increased privacy. However, we identified a small group of patients who had a strong preference for paper. The study also highlighted the need for clinical apps to reassure users by demonstrating feedback and clinical responses to their input to maintain the perceived value of using the system. Our future work will include developing approaches for such feedback as well as addressing the accessibility issues raised and applying our lessons in development of care support apps for in-hospital patients.