Keywords

1 Background

With the rise of medical information and technological advancements, digital health technology offers a variety of benefits to support healthcare providers and patients, improving compliance with standards of health quality, cost, and practice [1]. Therefore, accelerating the acceptance and engagement of digital health technology has been recognized as a national policy priority [2]. In 2009, the Health Information Technology (IT) for Economic and Clinical Health (HITECH) Act under the American Recovery and Reinvestment Act was initiated to expedite the coordination and delivery of American healthcare through health IT, including the adoption of digital health technologies.

Although a growing body of literature suggests that higher adoption of digital health systems is associated with safer and higher-quality care, [3] several pieces of evidence report unintended adverse effects of these technologies on clinical workflows due to poor usability issues [4]. The design of digital health technology is a complex process because of the inherently complex nature of clinical procedures that are mostly characterized by dynamic, non-linear, interactive, and interdependent collaborative activities, with uncertainty in outcomes [5]. Dealing with this complexity demands following an extensive set of design and usability requirements, constraints, and safety measures. Poor usability of digital health technology may result in substantial increases in medical error and associated costs, decreased efficiency, and unsatisfied users [6]. Recent reports have highlighted the importance of using cognitive engineering and human factors approaches for the design and empirical assessment of technology used in clinical settings [7].

Focus group and interview techniques have been widely used as common qualitative approaches to capture users’ feedback and to obtain in-depth insights on the usability of digital health technologies [8]. Focus group and semi-structured interview discussions are carefully planned and designed to obtain the perceptions of the individuals on a defined area of interest, facilitated by a moderator to keep the focus of the discussion. In these methods, participants are invited to discussion sessions to communicate their comments voluntarily in a safe and supported manner. This approach may lead to uncovered issues that researchers might have been unable to plan in advance. Moreover, focus group methodology is a cost-efficient way of evaluating user experience, as several subjects can be interviewed at the same time.

2 Current Challenges

Focus group methods have many advantages over other usability evaluation methods, however, as with any research methodologies, there are limitations. Some of these limitations can be overcome by well-structured planning and moderation, but other issues are inevitable and unique to this approach. Despite the widespread use of these qualitative methods, few guidelines exist for analyzing the data gathered from participants. Compared to structured questionnaires and quantitative experimental approaches, data collected during usability focus groups and interview sessions are often difficult to assemble and analyze. Annotating and coding qualitative data can be time-consuming and complicated for most digital health technologies given their inherent complexities. Although extensive previous literature provides frameworks and guidance on designing and conducting usability focus groups and interview sessions [9], scarce literature exists on extracting and coding usability data to effectively inform system design and development improvements based on a human-centered approach.

3 Proposed Coding Framework

Few studies evaluating the usability of digital health technology have attempted to combine qualitative coding with usability theories and principles [10]. Although these studies provide insights on the categorization of usability issues, there is still no standard methodology on how to integrate usability principles in coding this type of qualitative data. In this study, we aimed to develop a coding framework for the systematic evaluation of user feedback comments generated during focus group sessions, underpinned by fundamental usability principles and design components. Integrating usability principles and design components in coding and analyzing data generated during focus group sessions may help to systematically improve the degree of shared understanding between users with different roles, identify the extent of overlap in their comments, and the communication content generated during the sessions. Developing a shared understanding of the usability components of digital health technology can better ground the system on effective communication, leading to improved outcomes related to patient care coordination, teamwork, and care continuity.

4 Codebook Development

In usability studies, there are several well-established principles that can be utilized to categorize issues and comments collected during focus groups. Most studies attempting to evaluate usability via focus groups suffer from poor reproducibility of evaluations due to variation and subjectivity in codes, and a lack of standard reporting [10]. To tackle this challenge, we developed a codebook system that systematically evaluates each comment against usability principles for digital health technology. Moreover, since usability focus group sessions are primarily focused on different parts of a system interface and its specific features, this codebook incorporates categories related to design functions and elements. Furthermore, we included a coding category to capture the user’s clinical role associated with each specific comment, offering a better understanding of role-specific challenges and perspectives, as well as the level of shared understanding across the multiple clinical end-users.

The codebook allows researchers to code each users’ feedback based on three main categories guided by prompt questions: a) Usability Components: Which usability principle is targeted by the feedback? b) Design Components: Which design function and element are targeted by the feedback? c) Clinical Role: Which clinical role provided the feedback? The following sections provide details on each of these categories, helping researchers to categorize comments extracted during focus groups targeting digital health technology usability.

4.1 Usability Principles: Which Usability Principle is Targeted by the Feedback?

The main goal of usability focus group sessions for qualitative studies is extracting information about system issues and gathering insights on a prototype and/or system. Based on the human-centered design perspective, the Ten Heuristic Principles developed by Jakob Nielsen [11] have been identified as standard principles to identify and categorize usability issues, including 1) Visibility, 2) Match, 3) Control, 4) Consistency, 5) Error, 6) Recognition, 7) Flexibility, 8) Aesthetic, 9) Recovery, and 10) Help.

4.2 Design Components: Which Design Function and Element Are Targeted by the Feedback?

Usability principles alone are not sufficient to accurately categorize the issues associated with the usability of digital health technology, and other significant factors related to system design need to be integrated into the coding system. Qualitative usability studies aim to collect insights about how users interact with the product or service. This interaction, as described by scholars in the design community, happens at the interface level or the product ‘front-end’ which enables user interaction through communication and conversation. Therefore, this codebook incorporates usability principles to categories related to design functions and design elements [12].

Design Functions.

Most of the comments expressed during usability focus group sessions target a particular user interface [12]. Although these interfaces come in various forms such as buttons, pop-ups, radio buttons, panels, etc., they can generally be associated with a higher-level category of components. To date, though there are some available taxonomies, there is no well-accepted framework to categorize user interfaces [13]. For example, in previous research, Kamaruddin et al. proposed that the types of interface design consist of four main categories with separate features: presentation interface, conversation interface, navigation interface, and explanation interface [14]. According to these schemas, we considered the following categories for coding comments expressed during usability focus group sessions: input controls components, navigational components, informational components, and container components. These categories have been used frequently in the design community in order to describe the support provided to users by each individual interface during specific tasks. Each category encompasses user interfaces with common functions in the system (Table 1). This categorization can be helpful for analyzing the results from focus groups and interview sessions where the research team will gain better insights into function-wise usability issues in particular. Moreover, evaluation of function-wise usability issues across various iterations can be helpful to better compare versions of designs in various interactions throughout the prototype life cycle.

Table 1. Design functions of user interfaces.
Full size table

Design Elements.

We have also incorporated design elements into the codebook to have a more elaborated view of each feedback from a design perspective. Previous studies have established various design elements based on different perspectives and research frameworks. Most of these elements focused on the fundamental design components of the interface, which target a wide range of visual, audio, and content aspects. Based on these design foundations, a list of principles and elements are categorized into a shorter list of design elements [13]: color, imagery, typography, language, location, and audio. In line with these studies, the healthcare usability literature also suggests that color, imagery, position, and text style are the main design elements in digital health technology, which contribute to the ability of the user to accurately interpret and use the interface [15]. Incorporating these elements in the coding process of data generated during focus group sessions can help researchers to have a more accurate and detailed evaluation of users’ feedback. Each comment can be evaluated based on these six basic design elements (Table 2).

Table 2. Design elements of user interfaces.
Full size table

4.3 Clinical Role: Which Clinical Role Provided the Feedback?

In each focus group, participants with different clinical roles and backgrounds were invited to the sessions. Getting their unique perspective and integrating it with data collected from other participants can be useful for creating a shared understanding of usability issues across different parts of the design. The clinical role was considered as a code category for the person in that role providing feedback.

The codebook described in the previous sections allows researchers to code each participant’s comment based on three main categories guided by interviewers’ prompts. Adhering to this systematic coding framework allows researchers to assess saturation of usability issues in general, across design components, and roles in particular. Furthermore, since usability focus group data is analyzed one group at a time, following a systematic coding approach enables researchers to aggregate the findings across multiple focus groups, informing design modifications. It also allows the quantification and analysis of system issues by specific usability principles and/or roles, which may identify areas of design and development to focus on in future iterations.

4.4 Data-Driven Design Modifications

After categorizing the user comments based on usability principles, design components, and clinical roles, researchers and design and development teams can frame a list of potential changes to address the raised issues. Due to limitations in time and costs, implementing all the changes is often not feasible. The prioritization of which system modifications should be made is one of the most important challenges in designing and/or re-designing a digital health technology. As a part of the proposed framework, we created a voting system to facilitate this process and quantitatively inform system modifications, as well as their level of prioritization (Table 3).

Fig. 1.
figure 1

Main categories and components of the proposed coding framework

Full size image

This voting system evaluates each suggested modification based on three criteria (usability importance, clinical relevance, and technical feasibility), using a 5-point Likert scale. An advantage of this voting system is that it can incorporate multidisciplinary aspects from various experts, including not only the technical design and development team but also clinicians and human factors analysts.

Table 3. Decision criteria and rating for prioritization.
Full size table

5 Use Case

We applied the coding framework to evaluate an electronic cognitive aid (Smart Checklist) that was developed to guide cardiac surgery teams during common cardiac procedures in the operating room (OR) [16]. The Smart Checklist uses a carefully elicited and domain expert-validated process model to monitor the progress of an ongoing surgical procedure, determining the expected next tasks for each of the team members, thereby providing a context- and patient-specific perspective on each team role’s task management. This study was approved by the Mass General Brigham (MGB) Institutional Review Board (IRB) and all research subjects completed an informed consent procedure.

5.1 Data Collection

Four multidisciplinary focus group sessions were conducted with OR team members, representing key cardiac surgery roles: cardiac surgeons, cardiac anesthesiologists, perfusionists, and OR nurses. Two interviewers (LKM and H.C) conducted focus group sessions via videoconferencing while demonstrating the Smart Checklist. HC presented a detailed demonstration of the Smart Checklist to convey its primary informational features, where they were located, and the various ways in which users could interact with the interface (e.g., clicking buttons, entering notes). Pre-determined prompt questions were interspersed by the researchers to gather targeted feedback from participants. User’s comments were transcribed independently post hoc by both interviewers after review of the focus groups recordings. For example, Fig. 2 shows specific parts of the Smart Checklist in which participants were asked to discuss their preference over two alternative views: separated view vs. merged view. The merged view shows the next tasks for all teams in a single column and the separated view has different columns for each team or role.

Fig. 2.
figure 2

Different views of the surgical steps within the Smart Checklist

Full size image

5.2 Feedback Coding

Two independent coders (L.KM and H.C) used the coding framework to analyze the qualitative data generated in the focus groups. The coding of the 118 transcribed sentences was conducted via Dedoose, a web application for mixed methods research. First, the coders were instructed on coding components by providing examples of various types of usability principles and design components. Discrepancies between the coders were discussed during sessions until a consensus was reached. Then, the coders were asked to code each sentence based on the codebook (Fig. 1).

5.3 Results

A total of 12 subjects participated in four focus group sessions. Group 1: surgeon, anesthesiologist, perfusionist, nurse; Group 2: anesthesiologist, perfusionist, nurse; Group 3: surgeon, perfusionist, nurse; Group 4: anesthesiologist, perfusionist. To assess the inter-rater reliability (IRR) of the coding framework, we calculated Kappa coefficients. Analysis across all coding showed a moderate IRR with a Kappa coefficient of 0.53, p-value < 0.001, and an overall percentage of agreement between coders of 82.3%.

A total of 18 modifications were suggested by the focus groups. The priority voting system was completed by 3 human factor experts, 4 OR clinicians (1 attending cardiac surgeon, 1 attending cardiac anesthesiologist, 1 perfusionist, and 1 scrub nurse), and 3 technical designers/developers. Table 4 shows the suggested modifications with the respective scores (median) across all three criteria.

Table 4. Priority scores by human factors (HF), clinicians (CL), and technical experts (TE).
Full size table

6 Limitations and Future Directions

Even though the coding scheme was established based on well-established usability principles and design components and was tested to code usability data of a digital health system, it cannot be guaranteed that it will aid in the coding of all possible usability issues. Future studies are needed to further validate this codebook and voting system in the design and development process of other digital health technologies in additional clinical settings. Moreover, the coding system is specifically created to code data generated during focus group sessions, and future studies should evaluate the applicability of this framework on data gathered through other usability methods, such as verbal protocol and structured questionnaires.

7 Conclusion

In this study, we report the development of a comprehensive coding framework for usability evaluation of digital health technologies. In addition to incorporating relevant domains, such as usability principles, design components, and clinical roles, we have also provided a structured voting system to inform the prioritization of system modifications. The use case involving an electronic cognitive aid in the cardiac OR showed that this framework is feasible and useful not only to better understand distinct usability aspects that may be suboptimal, but also to recommend relevant modifications in the system from various perspectives, including the clinical team. The method and framework described herein may be adopted and applied to other highly complex clinical settings, in which digital health systems may play an important role in improving patient care and enhancing patient safety.