Keywords

1 Introduction

Recent publications have identified usability and safety concerns with electronic health records (EHRs) [1, 2]. Based on reports of possible patient harm, the American Medical Informatics Association (AMIA) convened a Task Force on Usability that developed ten recommendations for EHRs [3].

The purpose of this usability test was to establish a baseline of usability and user satisfaction for the current Health Information Technology (HIT) system used in Veterans Health Administration (VHA) for use as a reference for evaluation of future EHRs in the VHA.

Areas of interest for this study, as defined in ISO 9241-11 [4] included:

  • Effectiveness—the accuracy and completeness with which specified users can achieve specified goals in particular environments;

  • Efficiency—the resources expended in relation to the accuracy and completeness of goals achieved;

  • User Satisfaction—the comfort and acceptability of the work system to its users and other people affected by its use.

This paper will focus on findings associated with the process of conducting a usability assessment of an EHR rather than on the usability results. There were challenges associated with developing user tasks, selecting participants, and responding to errors that are unique to the health care arena. The lessons learned from this study may be useful in the design of future usability testing in health care.

2 The Usability Assessment

2.1 Method

For this study, 30 participants individually completed the entire test scenario, consisting of six tasks. All participants were practicing VHA physicians, physician assistants, or nurse practitioners who had experience with electronic prescribing, clinical information reconciliation, and clinical decision support. Some of the physicians were completing a residency in quality and safety, a one-year program that follows a traditional residency.

Years of clinical experience in the VHA served as a marker for experience with the EHR. Less experienced clinicians were those who had four or fewer years of experience in the VHA, while more experienced clinicians were those with five or more years of experience (Fig. 1).

Fig. 1
figure 1

Participants’ clinical experience in the VHA

Full size image

The testing occurred remotely via Microsoft Lync®. The test facilitator ran a national version of the VHA’s EHR on his computer. Participants used their own computers and phone lines and connected to the facilitator’s computer. Participants signed an audiotape and screen capture release form. They also completed a questionnaire on demographic information.

The six usability tasks and the test patient characteristics were derived with the assistance of subject-matter experts with consideration for the ONC Safety-enhanced Design 2014 Edition criteria on electronic prescribing [§170.314(b)(3)], clinical information reconciliation [§170.314(b)(4)], and clinical decision support [§170.314(a)(8)] [5]. These criteria were used because they describe common and important clinical tasks. The scenario and tasks were identical for all study participants. Clinical staff reviewed the task descriptions to ensure that the content, format, and presentation were representative of real use and substantially evaluated the required functionality. The tasks, which represented common and relatively complex functions, were as follows:

  1. 1.

    Review Lab Results: Please review the lab results from September for the patient. Tell me which results are not in the normal range.

  2. 2.

    Locate Clinical Guidelines: Assume you want to find clinical guidelines for cellulitis. Please show me how you would do this. State verbally the source and publication date of the information.

  3. 3.

    Order Medication (Alert): Next, we will ask you to order a medication. You decide to order Bactrim for this patient’s cellulitis. Please place this order using CPRS as you would normally with regard to dosage, route, etc. [Note: In this task, the provider receives an alert that the patient has a previous adverse reaction to Bactrim and that Bactrim interacts with another medication the patient is taking. The expected response is for the provider to refuse to order the medication.]

  4. 4.

    Order Medication (No Alert): Because of this patient’s allergies, you decide to order Clindamycin. You have reason to believe this antibiotic is appropriate in this situation. Please place the order using CPRS.

  5. 5.

    Locate Patient Education Material: Please provide the patient with education material related to warfarin. State verbally the source and publication date of the material.

  6. 6.

    Locate Patient Alerts: The patient’s daughter expresses frustration that her dad could do more to take care of his health and says she and her brother are going to help stay on top of things. She wants to know if he is due for any preventative testing. Please locate patient alerts and reminders and tell me if any clinical actions need to be taken.

In addition, each participant was asked to describe how s/he performs medication and problem list reconciliation, what works well about this method, and what does not work well. Each participant answered these questions for several different scenarios, including for a patient who is only seen at the local medical center, for a patient who is seen at different VA medical centers, and for a patient who is seen at a clinic outside the VA system.

2.2 Test Metrics

After the participant completed each task and each set of interview questions, s/he was asked to answer the rating questions listed in Table 1. For each task, the metrics described in Table 2 were collected.

Table 1 Task rating questions and scales
Full size table
Table 2 Usability metrics collected for each task
Full size table

2.3 IRB

Because this research effort was considered Quality Improvement, IRB approval was not required. The project adhered to the following ethical guidelines:

  • The performance of any test participant must not be individually attributable. The individual participant’s name should not be used in reference outside the testing session.

  • Information about the participant’s performance should not be reported to his or her manager.

3 Lessons Learned

3.1 Test Plan

Initially, the intention was to conduct a summative usability evaluation of the VHA’s EHR. Due to technical limitations with the test environment, however, it was not possible for participants to access all of the programs that providers normally use. Therefore, formative usability evaluation techniques were used for the “medication reconciliation” and “problem list reconciliation” tasks. Rather than performing the information reconciliation tasks, the participants answered a series of questions about how they would normally perform these tasks.

Using formative evaluation techniques when it was not possible to perform a full summative evaluation allowed the investigators to learn about how clinicians perform these tasks and what difficulties they encounter, but there may have been additional findings had they actually observed clinicians performing the tasks.

3.2 Scenario Development

Scenario development began by identifying relevant tasks from the ONC Safety-enhanced Design 2014 Edition criteria [5]. The initial draft of the tasks was composed with the help of a clinician. Multiple iterations of the scenario with clinicians with different areas of specialty were completed to ensure the tasks were written in such a way that clinicians agreed on the correct response.

Lack of realism is often a concern with simulated task environments. In cases where the correct answer is to halt a task, there is always a concern that a participant will proceed simply because the moderator has given a direction. For the medication ordering task with the alerts, one participant entered “CPRS Test” as the reason for the alert override, indicating she was not acting as she would in a normal clinical setting. After this occurrence, the moderator added an instruction to participants to complete tasks as they would normally, and questioned any participants to determine if they thought they had acted as they would with an actual patient.

3.3 Participant Recruitment

Participants were recruited from a variety of clinical areas, with a result that some participants did not commonly perform all the tasks in the test scenario. Several participants who worked in specialty areas noted that they did not commonly prescribe antibiotics, and they had to spend additional time locating information on the correct dose. For future studies, it may be best to limit participants to one practice area or create different scenarios for each clinical specialty. In addition, it may be best to consider within-subject differences in task times for the same task performed using different EHRs rather than drawing conclusions based on aggregate task times on a single EHR, because a participant who does not often perform a particular task may artificially inflate the total.

Investigators recruited participants with a variety of experience, but it was difficult to locate true novice users for this study. Residents in health care settings generally begin their residency during July, and testing was conducted in the late fall and winter. Therefore, all participants had been using the EHR for at least several months. In addition, many physicians are initially exposed to an EHR during medical school. Future studies may wish to consider this timeline and conduct testing during the summer in order to target new users of an EHR.

One goal for this usability assessment was to obtain a geographically distributed test sample to reflect the diversity present in the VHA, which has over 1500 sites of care located around the country [6]. Therefore, all tests were conducted remotely. In addition, there was no eye tracking software available. Due to this, it was impossible for investigators to know where participants were looking as they performed the tasks. During a medication ordering task, one participant noted that he was using an app on his cell phone to look up dosage guidelines. It is impossible to tell if any other participants did this as well. One solution could be to ask participants explicitly at the end of each task if they used any additional reference material during the task.

3.4 Ethical Considerations

It is generally possible to protect participants in a usability test by consolidating the data. There may be a problem if the study identifies in some way a participant who has made an error. This identification could occur if the authors are too specific about a participant’s job title, role, and/or location.

3.5 Unexpected Findings

One participant made an unanticipated error during the medication prescribing task. During the debrief period following the test session, the moderator explained the error and the correct procedure to the participant. After discussion with team members and patient safety experts, it was determined that there was no patient safety concern associated with this error. Therefore, the team took no additional action until all testing was complete in order to avoid altering the outcome of future test sessions. The results of the testing will be incorporated into future training to correct this error.

4 Discussion

A key finding from this usability assessment is that flexibility is essential when evaluating a complex tool in a changing environment, such as an existing EHR in an active health care setting. Evaluators may wish to consider employing a mixture of usability evaluation techniques if technical issues prevent a purely summative test. Valuable findings can come from methods such as structured interviews, even if these do not yield direct quantitative measurements.

It is important to call upon the experience of multiple clinicians with diverse backgrounds when developing a usability scenario, because there can be significant differences in practice between professionals depending on area of clinical expertise and typical practice environment.

Having participants with varying amounts of experience can be useful when attempting to identify usability issues. Often, novices will commit more errors, and it may be possible to learn more about workarounds with super-users. In addition, users who have trained others may have anecdotal accounts of usability issues they observed with trainees.

When conducting a usability assessment of a system that is currently in use in a health care setting, it is important to consider how to proceed when there are unexpected findings. Ideally, the investigators will be able to avoid any actions that would compromise findings from the remaining participants.

In most usability tests, protection of the participant is the ethical imperative. However, with health care, it is always necessary to consider patient safety as well. If there is a clear and imminent patient safety concern, one must consider how to address that, but confidentiality is still necessary, both to protect the participant and to promote future patient safety. Consider the case of New Zealand’s Independent Safety Assurance Team (ISAT), which established a confidential aviation safety information sharing program in 1988. After an analyst knowingly released the name of a reporter, industry lost confidence in the system and ISAT subsequently failed [7]. Without protections for reporters and usability assessment participants, safety will be compromised.