Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Beyond the OSCE: Evaluation Clinical Skills Using Unannounced Standardized Patients

Although the use of standardized patients (SPs) is widely used in assessment, the vast majority of SP exercises/exams involve announced encounters in which the learners know that they are interacting with a simulated patient. Objective structured clinical examinations (OSCEs) are considered a gold standard for assessing clinical skills, but their overtly contrived nature may limit their ability to capture the true behavior of medical professionals (Ozuah and Reznik 2007). The use of unannounced standardized patients (USPs) is a relatively new but increasingly used method for evaluating the competence of medical professionals (Glassman et al. 2000; Rethans et al. 2007). USP encounters do not have the artificial time constraints of OSCEs, and USPs can evaluate subjects in a real clinic setting.

USPs have been used to assess trainees as well as practitioners across the health professions, including nursing (Carney and Ward 1998), optometry (Shah et al. 2007), and a range of medical specialties, from primary care (Culver et al. 2009) to emergency medicine (Zabar et al. 2009). The published literature describes using USPs to assess clinical skills (e.g., [in residents], Ozuah and Reznik 2008a, b) or the efficacy of educational interventions. The performance of clinicians visited by USPs may be compared with that of a group who did not receive an intervention (control group) or to their own performance in announced standardized patient encounters. Intervention studies (e.g., Casebeer et al. 1999) determine the success of an educational initiative, generally by comparing performance with USPs before and after the intervention. Other studies explore the feasibility and validity of USPs compared to chart reviews and clinical vignettes (Peabody et al. 2000).

USPs can also be used to evaluate the clinical microsystem. A clinic’s commitment to becoming a patient-centered medical home is as important to health outcomes as the skills of its physicians. USPs can rate the performance of medical assistants, the ease of navigating the clinic, and the functioning of the clinic care team (Peabody et al. 2004; Zabar et al. 2009). USPs easily documents adherence to national patient safety standards, such as hand washing and patient identification. USPs can also observe a clinic’s level of patient centeredness (Epstein et al. 2005). Because they undergo every level of the patient experience, USPs are versatile judges of health centers.

Health professional schools that conduct OSCEs are poised to incorporate a USP program into their curriculum, as much of the infrastructure required to perform the two assessments is similar, such as SP/rater training and case development. Since USPs are integrated into an already established clinic setting, costs are limited to compensation for the actors. Hourly rates for standardized patients range from $15 to $25 per hour, and most USP visits last 2–4 h. Additional USP requirements include close collaboration with administrators in the clinical setting.

USP programs are not without potential difficulties. There is a risk that USPs will be detected, which can undermine the effectiveness of the program. If a medical professional realizes (s)he is interacting with a USP, he or she may not behave in natural manner. Consequently, the data collected still may not reflect a clinician’s true skills. Matching USPs with the intended clinician can be complicated in some health centers, particularly those that do not assign patients to specific providers. Constant communication with scheduling coordinators is necessary to ensure that USPs interact with the correct clinicians. There is also the concern that USP visits hinder productivity by taking up trainee patient time. However the few number of visits needed to evaluate the clinical system and provider performance is usually seen as worth the investment by hospital and educational leadership.

USP Staffing Needs

USP projects can be a major undertaking, and as with most other educational projects, collaboration within and across specialties, even across disciplines can only enrich the process. While it is necessary to have strong leaders who believe in the benefits of such comprehensive assessment programs, many other individuals are needed for adequate planning, preparation, and implementation. Table 4.1 details the additional roles that USP projects typically require beyond those detailed in for OSCE administration Table 2.1.

Table 4.1 USP staffing needs (see also: Table 2.1)
Full size table

Implementing a USP Project

Specific tasks involved in planning to integrate USP visits in a clinical setting are detailed in Table 4.2, a modification of the worksheet for assigning OSCE responsibilities and creating timelines (Fig. 2.3, Appendix C).

Table 4.2 Breakdown of USP responsibilities
Full size table

Cases

USP and OSCE case development follow the same basic principles (see Chap. 2, Step 4, “Develop Cases and Stations”). Many OSCE cases can easily be adopted to be used in USP visits. To prevent detection, it is crucial to make sure that USP cases are representative of the patient population served by the providers one plans to evaluate. A sample USP case and corresponding checklist, designed for an urban community clinic, are included at the end of this book as Appendices N and O.

Recruitment

The number of USPs required depends on the number of cases in the program, the number of clinicians involved, and the duration of the program. Medical schools are the best places to recruit USPs, since they work with actors who already have experience as standardized patients. The most qualified standardized patients will possess acting talent, punctuality, communication skills, and the ability to adapt to unpredictable situations.

Training

USP training sessions are similar to OSCE training exercises. Trainings can be divided into three sessions. During the first session, the USP program coordinator explains the purpose and logistics of the program to USPs. USPs should then read the case instructions aloud with the USP coordinator. After it is clear that the USPs fully understand their role, they practice the case, taking on the patient role while the coordinator assumes the role of physician. The second training focuses on teaching USPs to complete the evaluation forms. The coordinator shows a presentation about the correct way to observe, categorize, and document clinicians’ behavior. To practice completing the evaluation, the USPs should watch OSCE encounters and evaluate learners’ skills. During the final training session, the USPs can role-play the case with an attending physician or chief resident to learn the pacing of a medical interview. The USP coordinator can discreetly bring the USPs to the clinic before their first visits to prepare USPs for navigating the area.

Clinic Location and Visits

Before any visits are planned, program leaders must get permission from clinic administrators to conduct the program. They should speak to members of the finance office to learn how to prevent USP visits from being billed as real visits and appearing in clinic audits.

The USP coordinator should visit the clinic during a busy day to observe its layout. He or she should note the location of the registration desk, exam rooms, and other relevant areas (finance desk, pharmacy, etc.) The coordinator must observe where patients must go to check in, pay, encounter doctors, and get prescriptions and referrals. The coordinator will be better prepared to train USPs to navigate the clinic if he or she is aware of the path real patients travel.

The program team then identifies the unique characteristics of the clinic that will receive USPs. In some clinics, it is possible to schedule appointments with a specific doctor; in others, patients are assigned doctors in a first come, first serve basis. The USP coordinator needs to work with the clinic’s patient coordinator to develop a system that will ensure USPs are sent to the correct physicians. The patient coordinator should also be responsible for entering USPs’ demographic information into the clinic’s computer system. The USP coordinator can develop a process and deadline for sending the demographic information for each visit.

The USP coordinator collaborates with the patient coordinator to develop the USP visit schedule. The USP coordinator chooses dates and times for USP visits and sends them to the patient coordinator for approval. The patient coordinator approves the requests if the appointments are available and can suggest edits to the USP coordinator’s selections if there are scheduling conflicts.

After the schedule is finalized, the USP coordinator asks USPs to sign up for visits. On the day of a visit, the USP coordinator should meet with the USP before the visit begins to give him or her an audio recorder. USP visits should be recorded in order to validate the checklist data. After the USP gets the recorder, he or she enters the clinic.

The USP should invent an excuse to avoid getting labs ordered by a physician. For example, he or she can say they have to go back to work or just ate. The USP should hold onto any paperwork he or she is given (prescriptions, referrals, etc.) and return it to the USP coordinator after the visit.

Post-visit

After the visit is complete, the USP meets with the coordinator to complete the evaluation. The coordinator reviews the evaluation for missing data and inconsistencies, then performs a debriefing session, where qualitative data about the visit is discussed. Topics raised during the debriefing include the atmosphere of the clinic, the conduct of the resident and/or medical assistants, and the degree of difficulty in navigating the clinic. Debriefing sessions should also explore facilitators and barriers to patient care. When the visit is complete, the USP signs an invoice.

Budgeting

The USP coordinator can keep track of the program costs in an Excel spreadsheet. All training and visit costs for each USP should be documented and updated frequently to ensure the program stays within the budget. The spreadsheet should include the name and contact information of each USP, list every date each USP worked, and include the amount USPs were paid for each visit or training session. Excel can calculate the total program costs and the average costs per visit. For example, the NYU School of Medicine USP program costs about $120 per visit.

Learner, Microsystem, and Programmatic Evaluation

USP visits provide a wealth of information. The program leaders can disseminate a summary report for clinicians on their overall performance across all USP cases. The clinic administrators should receive a summary report of the health care team’s performance for patient safety, patient centeredness, screening assessments, and team skills.

USP scores can also be compared to OSCE performance to see how an individual performs in a testing situation versus the “real world” of the clinical environment. The sample evaluation in Fig. 4.1 shows individual and mean cohort primary care resident communication skills as measured in a ten-station OSCE and across multiple USP visits (% checklist items “well done”). As can be seen in the sample report, this particular resident (“Dr. K”) actually shows a trend of performing better in USP visits as compared with OSCE encounters.

Fig. 4.1
figure 00041

Sample learner feedback report: OSCE versus USP communication performance. Communication sub-competency scores reported: information gathering, relationship development, patient education and counseling, and organization and time management

Full size image

In our program, USPs were asked to evaluate clinical microsystem as well as clinician performance in 50 visits to primary care providers at an urban community clinic. During each visit, USPs recorded whether the medical assistant greeted the patient within a reasonable time frame; introduced his or herself; wore a visible name tag; washed hands before touching me; measured my height; took my blood pressure; weighed me; and screened for depression. USPs also assessed their general experience with clinic: how easy it was to navigate the system; team functioning; and overall staff professionalism.

Clinic administrators then conducted an educational intervention with the medical assistants to improve performance. In preliminary subsequent USP visits post-intervention marked improvements were noted. Clinical microsystems data such as these serve to inform medical directors of critical gaps in patient safety measures, patient satisfaction, and patient centeredness. With specific data on the patient experience, administrators can implement appropriate improvement measures.