Introduction

Psychiatry education is an important element in the health professions (Lipari et al., 2013; Patel et al., 2016). Simulation-based education includes interaction with real or virtual objects, devices, or persons, and represents a promising learning tool in psychiatry. In a recent systematic review and meta-analysis, Piot et al. (2020) demonstrated the effectiveness of simulation-based psychiatry education for developing knowledge, skills, and attitudes of medical students, postgraduate medical trainees, and qualified doctors. Furthermore, narrative reviews have suggested that simulation-based psychiatry education can increase students’ level of knowledge, communication skills, empathy, and engagement (Abdool et al., 2017; Brown, 2008; Kunst et al., 2017; McNaughton et al., 2008; Øgård-Repål et al., 2018; Vandyk et al., 2018). However, differences in assessment and simulation methodologies made direct comparison difficult. In addition, these studies predominantly focused on face-to-face simulation, including role-play, mannequins, and using standardized or simulated patients. Such methodologies are costly, resource-intensive for staff, challenging to schedule when used in larger student cohorts, and the number of learners accommodated at any given time is limited (Triola et al., 2006). Moreover, simulation experiences may not be standardized and there is often little opportunity for repetitive practice (Andreatta et al., 2010).

Virtual patient simulations may ameliorate some of these shortcomings (Peddle et al., 2019). In this review, we define virtual patients as interactive, screen-based, and dynamic patient cases that simulate real-life clinical scenarios. Virtual patient simulations can be delivered to a large number of students and provide access to situations where actual clinical encounters are difficult to facilitate (Kononowicz et al., 2019). In online and distance learning environments, virtual patients can also be adapted to the needs of individual learners and teachers.

While virtual patients have been researched and used in medical education as a whole (Cook & Triola, 2009; Cook et al., 2010; Kononowicz et al., 2015, 2019), they are a relatively new approach in psychiatry education (Brown, 2008; Guise et al., 2012). To the best of our knowledge, there has been no systematic review of the use of virtual patients in health disciplines offering undergraduate psychiatry education. In this systematic review, we evaluate empirical literature on virtual patients used as educational interventions in undergraduate psychiatry education to develop students’ knowledge, skills, and attitudes. The aims of the current review were to:

  1. 1.

    Provide an overview of the most used approaches to date and describe their effectiveness.

  2. 2.

    Examine and thematically compare learning outcomes across different undergraduate programs.

Methods

Design

The study was preregistered at Prospero (CRD42020196046) as a systematic review and was reported in adherence with the Preferred Reporting Items for Systematic Reviews and Meta-analyses guideline (Moher et al., 2009). The study was exempt from ethical approval as it was a literature study that did not directly involve human subjects.

Eligibility

We considered all studies on educational interventions using interactive, screen-based, and dynamic virtual patients in the context of undergraduate psychiatry education. Inclusion criteria also included studies reporting on outcomes related to the students’ knowledge, skills, and/or attitudes. We excluded studies that did not meet these criteria, e.g., studies examining virtual patient interventions with static multimedia representations or interventions not developed with an educational purpose. Other exclusion criteria were studies reporting on outcomes with no relevance to the educational effect of the intervention, e.g., outcomes related to satisfaction, and studies that were published in languages different from English, not peer-reviewed, or strictly descriptive studies (see full inclusion and exclusion criteria in the Online Resource 1).

Search methods for identification of studies

PubMed, PsycInfo, CINAHL, and Scopus databases were searched in January 2021. The search terms included Medical Subject Headings and free-text words that referred to (1) virtual patients, (2) undergraduate education, and (3) mental illness, separated by Boolean operators (see Online Resource 2). Free-text words were limited to titles and abstracts, and the search was limited to publications from the Year 2000 onward.

Study selection

Search results were retrieved and imported into the Endnote (version 9). The first and last authors independently reviewed and screened titles and abstracts, and eligible studies were included for full-text screening using Covidence systematic review software (Innovation, 2020). The same reviewers independently screened the studies for eligibility and final inclusion. Disagreements were resolved by discussion and inclusion of the second co-author.

Data items for extraction

The data extraction template was developed through iterative testing and revision. Studies were classified as using quantitative, mixed, or qualitative methods. The quantitative and mixed-methods studies were further classified as no-intervention controlled (single-group pretest–posttest comparison, or comparison with another group receiving no intervention), non-media-comparative (comparison with a group receiving a non-media-based educational intervention, e.g., a face-to-face simulation), or media-assisted learning comparative (comparison with a group receiving a media-based educational technology, including an alternate virtual patient). For studies with more than one comparison condition, we classified the study according to the most active control. We abstracted information on:

  1. 1.

    The characteristics of study participants (field of study; year of study; country where the study was conducted and its World Bank income category).

  2. 2.

    Type of learning outcome (knowledge, skills, and/or attitudes).

  3. 3.

    Methods used to measure the learning outcomes (subjective/objective, validated/non-validated).

  4. 4.

    The type of virtual patient was synthesized in a narrative description. We coded the type of case progression as linear (the virtual patient followed the same course regardless of decisions), branched (the virtual patient evolved with participants’ decisions affecting subsequent events), or unclear. We coded the virtual patient’s use as self-directed or teacher assisted, and whether the students carried out the intervention as an individual or group assignment. We also abstracted information on the virtual patient clinical topic, the number of cases, and the clinical variation.

Finally, we abstracted information on the presence of features of effective simulations identified in a previous review of virtual patient simulation (Cook & Triola, 2009):

  • Feedback provided to the learner (coded as low, moderate, or high).

  • Opportunity for repetitive practice (present or absent).

  • Curriculum integration (the virtual patient was an integrated part of the curriculum/course [present] or an optional activity [absent]).

Data synthesis and analysis

The data extraction structure allowed the succinct organization of the data to compare and incorporate findings from a variety of research methods (Stubbings et al., 2012). A narrative synthesis was performed to synthesize the findings systematically. We decided a priori to forego meta-analyses because of the research questions and because we expected variety in study populations, interventions, and educational outcomes.

Data related to the type of virtual patient was categorized following the framework for virtual patient classifications originally proposed by Talbot et al. (2012) and modified by Kononowicz et al. (2015).

Data related to the type of learning outcomes were compared using thematic analysis (Braun & Clarke, 2006). Emerging themes related to each of the main categories were identified and confirmed with primary sources.

Quality assessment

To assess the quality of the selected studies, we employed two tools: The Medical Education Research Study Quality Instrument (MERSQI) (Cook & Reed, 2015) for quantitative and mixed-methods studies and the QualSyst standard assessment criteria (Kmet, 2004) for qualitative studies.

The MERSQI is a validated tool designed to measure the quality of medical education research with varied quantitative methodologies, including ten items in six domains (study design, sampling, type of data, validation of evaluation instrument, data analysis, outcomes measured). MERSQI scores range from 5 (indicating lowest quality) to 18 (indicating highest quality). The first author served as the gold standard rater and assessed the quality of all included quantitative studies, and the last author assessed 24% of these as a quality check, with intraclass correlation coefficient (ICC) = 0.81.

The QualSyst assessment covers a variety of qualitative study designs and consists of a checklist for qualitative studies with ten criteria with scores. The total score ranges from 0 (lowest quality) to 1 (highest quality) and rests on the ratio of the total score earned to the total possible score. The first and last authors assessed the quality of all included qualitative and mixed-method studies, and disagreements were resolved in discussion, ICC = 0.69.

Results

Trial flow

A total of 7856 records were identified. After removal of duplicates and exclusion of irrelevant references, 240 studies remained for full-text screening. Of these, 46 studies fulfilled the inclusion criteria (see flowchart in Fig. 1).

Fig. 1
figure 1

PRISMA 2020 flow diagram for new systematic reviews applied in a systematic review on virtual patients in undergraduate psychiatry education

Full size image

Quality assessment

The mean MERSQI score of the quantitative studies was 11.94 (range 7 – 14.5), which is comparable to other research adopting the MERSQI (Cook & Reed, 2015). The QualSyst score of the qualitative studies was 0.91, which can be interpreted as indicative of high quality based on the criteria required for scores of over 0.80 in the literature (Balaguer et al. 2020; Speyer et al., 2018). The mixed-methods studies received scores of 11.78 on the MERSQI and 0.61 on the QualSyst. See Online Resource 3.

Study and participant characteristics

We identified five studies that used rigorous qualitative methods and 41 studies that used quantitative or mixed-methods comparative designs (see Table 1).

Table 1 Study characteristics, virtual patient design, and outcomes of 46 studies included in a systematic review on virtual patients in undergraduate psychiatry education
Full size table

Of the 41 quantitative and mixed-methods comparative studies, 17 (41%) studies reported comparison with no intervention including two RCTs, 11 (27%) studies reported on non-media-comparison including three RCTs, and 13 (32%) studies reported results on a media-assisted learning comparative including eight RCTs. One study was conducted in a low-middle-income country, India (Nongmeikapam et al., 2019), while the remaining 45 studies were conducted in high-income countries. Sample sizes of the studies varied from n = 6 (Washburn et al., 2016) to n = 532 (Kelly et al., 2020). A total of 5563 students participated in the studies, including 2546 medical students, 1873 nursing students, 254 psychology students, 28 social work students, 87 students who were enrolled in a psychology course or a social work course, 244 pharmacy students, and 531 students enrolled in a healthcare-related education (see Online Resource 3).

Virtual patient characteristics and educational characteristics

The reviewed studies included different descriptors for the type of intervention, e.g., “virtual patient”, “video-based teaching”, “computerized clinical simulation”. Following Talbot et al. (2012), we categorized the virtual patient in each intervention as being either a case-based presentation of a virtual patient (n = 17), an interactive virtual patient scenarios (n = 14), a standardized virtual patients (n = 10), and a virtual patient videogames (n = 5). This is shown in the Online Resource 4 together data on how the studies used different means to portray the patient, i.e., actors, avatars and animations, or real patients.

Geriatric psychiatry was the the most widely represented topic (Buijs-Spanjers et al., 2018, 2019, 2020; Chao et al., 2012; Goldman et al., 2008; Kelly et al., 2020; Matsumura et al., 2018; Robles et al., 2019; Rosen et al., 2013). Other topics included managing boundary-seeking patients (Kunaparaju et al., 2018; Taverner et al. 2000), assessing the risk of interpersonal violence (Verkuyl et al., 2017), combating stigma (Kerby et al., 2008; Wei Liu, 2021; Nguyen et al., 2012; Winkler et al., 2017), screening for suicidal ideation (Foster et al., 2015; Kullberg et al., 2020) and substance use disorders (Bremner et al., 2020; Burmester et al., 2019; Koetting & Freed, 2017; Lee et al., 2008; Tanner et al., 2012), assessment (Washburn et al., 2016, 2020), mental state examination (Fog-Petersen et al., 2020; Hansen et al., 2020; Martin, et al. 2020; ; Williams et al., 2001), diagnostic skills (Gutiérrez-Maldonado et al., 2015), treatment of mental illness (Hayes-Roth, 2004; Kitay et al., 2020; Mastroleo et al., 2020; Smith et al., 2020; Warnell et al., 2005), patient-centered skills (Chen et al., 2018; Choi et al., 2020; Foster et al., 2016; Pedersen et al., 2018, 2019; Sunnqvist et al., 2016), behavioral medicine (Berman et al., 2017), and perinatal mental health (Dubovi, 2018). One study did not clearly describe the topic focus of the virtual patient intervention (Nongmeikapam et al., 2019).

Virtual patient teaching and learning activities were divided between students working as individuals (n = 33) and in groups (n = 12). Data were unclear for one study (Warnell et al., 2005). Fifteen studies used spaced instruction, where the instruction is spread out and repeated over time (Berman et al., 2017; Bremner et al., 2020; Fog-Petersen et al., 2020; Hansen et al., 2020; Hayes-Roth, 2004; Lehmann et al., 2017; Wei Liu, 2021; Mastroleo et al., 2020; Matsumura et al., 2018; Pedersen et al., 2019; Rosen et al., 2013; Smith et al., 2020; Tanner et al., 2012; Washburn et al., 2016; Washburn et al., 2020).

Qualitative outcomes: common themes across qualitative and mixed-methods studies

Results of the abstracted themes are summarized in the Online Resource 5. The five qualitative studies used semi-structured interviews to explore students’ learning experiences with virtual patients, and were coded using thematic analysis (Braun & Clarke, 2006). Although the research questions and analytical method in the mixed-methods studies varied from the qualitative studies, we identified four common themes across the study types: First, a safe learning environment that offered opportunities to repeat practice sessions was important for developing knowledge, self-efficacy, and confidence in the students. Second, authenticity was important for developing students’ empathy and understanding of the patient’s perspective. Virtual patient interventions using actors or real patients were perceived as being authentic (Pedersen et al., 2018, 2019; Verkuyl et al., 2017), while virtual patient interventions using avatars (Washburn et al., 2020) or a videotape of a simulation between an actor and a manikin (Kelly et al., 2020) were perceived as being less authentic. The two latter studies revealed that lack of authenticity negatively influenced students’ abilities to empathize with the patient. Third, students perceived the virtual patient interventions as valuable pedagogical models to provide a scaffolding of the mental health professional’s role in the clinical encounter with psychiatric patients. Fourth, integrated feedback was considered important for identifying knowledge gaps. In some of the reviewed qualitative and mixed-methods studies, students requested further feedback (Berman et al., 2017; Choi et al., 2020; Fog-Petersen et al., 2020; Kelly et al., 2020; Smith et al., 2020; Sunnqvist et al., 2016; Verkuyl et al., 2017).

Quantitative outcomes: knowledge, skills, and attitudes

All 17 no-intervention controlled studies reported significant and positive outcomes related to knowledge (n = 8), skills (n = 7), and attitudes (n = 10).

In the 11 non-media-comparative controlled studies, the results for outcomes related to knowledge (n = 4), skills (n = 3), and attitudes (n = 5) were mixed. In the studies that compared a virtual patient intervention with a live simulation, the results favored the live simulation (Nguyen et al., 2012; Robles et al., 2019; Warnell et al., 2005; Winkler et al., 2017). However, when compared to teaching as usual or a self-paced text-based intervention (where the patient case is described in text only), results favored the virtual patient intervention (Hansen et al., 2020; Hayes-Roth, 2004; Wei Liu, 2021; Mastroleo et al., 2020; Matsumura et al., 2018; Nongmeikapam et al., 2019; Pedersen et al., 2019).

In the 13 studies evaluating differences in outcomes between virtual patients and other educational technologies, seven studies showed no significant differences in outcomes related to knowledge (n = 2), skills (n = 5), or attitudes (n = 3). Two studies favored interactive patient scenarios over case presentations in terms of knowledge (Choi et al., 2020; Lee et al., 2008), skills (Choi et al., 2020; Lee et al., 2008), and attitudes (Choi et al., 2020). One study showed that dynamic screen-based virtual patients in comparison with static screen-based virtual patients enhanced skills significantly more (Foster et al., 2016). One study demonstrated that the virtual patient could be reliably used to discriminate novice learners from experienced learners (Martin, et al. 2020).

The thematic analysis of the primary outcomes with themes and subthemes related to knowledge, skills, and attitudes are summarized in the Online Resource 6. In Table 2 we present an example of our analysis.

Table 2 Example of analysis: category, data extract, codes applied, and themes
Full size table

Discussion

This review supports the finding that virtual patients can provide important and effective teaching to diverse healthcare students. Based on the findings from the qualitative and mixed-methods studies, we hypothesize that two main mechanisms are involved in learning from virtual patients; learning in a safe environment and the authenticity of the virtual patient.

While all interventions with virtual patients provide a safe environment in the sense that the student is not at risk of misdiagnosing, mistreating etc. a real patient, not all interventions are perceived as authentic by learners. By “authenticity” we refer to the ability of educational technologies to produce and render scenarios, experiences, and processes that closely resemble real life (Shaffer & Resnik, 1999). In the few studies that evaluated authenticity, the virtual patients based on videos of actors or actual patients were perceived as authentic. In contrast, students perceived virtual patients based on avatars as less authentic. However, the studies did not provide detailed descriptions of what aspects of the virtual patient case authenticity was related to, and no studies used a comparative design. Guise et al. (2012) suggest that observing natural human facial and bodily expressions and focusing on non-verbal communication may be particularly important to learning from virtual patients in psychiatry education (Guise et al., 2012). Nevertheless, perceived authenticity is not only a matter of human representation (Fredholm et al., 2019), as it can also refer to the interface, the patient story, or the learners’ tasks. Shaffer and Resnick (1999) introduced the concept of “thick” authenticity to account for these different kinds of authenticity. They also stressed that in addition to authenticity-related aspects of virtual patients, it is important to consider their relation to the world outside education.

The quantitative studies demonstrated that virtual patients, in comparison with no intervention, teaching as usual, and text-based interventions, have been consistently associated with better learning outcomes. However, they did not indicate any superiority of virtual patients over non-technological patient simulation. These results are in line with previous reviews examining the effectiveness of virtual patients in medical education (Abdool et al., 2017; Cook et al. 2010b; Cook & Triola, 2009; Kononowicz et al., 2019; Peddle et al., 2019; Piot et al., 2020). Nevertheless, there was uncertainty regarding the quality of the evidence as the majority of the quantitative studies included in the present review were of low-to-medium quality.

We extended our findings by systematically summarizing data related to the learning outcomes using thematic content analysis. We turn to a discussion of the themes we identified below.

Virtual patients and knowledge related outcomes

Based on the seventeen articles that evaluated outcomes related to students’ development of knowledge, we identified two major themes: Knowledge of symptomatology and knowledge of psychopathology.

Knowledge of symptomatology included which symptoms characterized specific disorders, which screening tools were relevant to include in the diagnostic interview, and which treatment options were available for specific disorders. Although the results of the review suggested that virtual patients were successful in facilitating the learning of such knowledge, there were no strong indications that they were significantly more effective than other methods of teaching.

Virtual patients may offer more in terms of learning about psychopathology as virtual patients, with their ability to represent a rich picture of the patient and the clinical encounter, can help students to develop knowledge that is more person-oriented and holistic than that afforded by other modalities.

Virtual patients and outcomes related to skills

Assessment of interpersonal and clinical skills was the focus of 20 studies. The thematic analysis showed that while interpersonal skills, such as communicative and patient-centered skills, were assessed across different healthcare education programs, assessment of specific clinical skills was more contingent on clinical roles. For example, diagnostic accuracy was the focus for medical students, while skills in advising care were the focus for students in nursing education.

We found that students had mostly been assessed based on interviews with virtual patients. While this demonstrates that virtual patients in psychiatry education have been used both as a training tool and a performance-based assessment tool, it did not show how learning worked, or what the generalizability was, including whether learning transferred to the clinical setting. For example, one study suggested that interaction with virtual patients enhanced students’ skills in diagnosing virtual patients but not simulated patients (Washburn et al., 2020).

Virtual patients and outcomes related to attitudes

Based on our analysis of the 19 studies that evaluated outcomes related to attitudes, we found two major themes: attitudes towards oneself, e.g., self-efficacy, and attitudes towards others, e.g., stigma.

Self-efficacy refers to students’ judgment of their capabilities to perform a particular task successfully. Competent functioning in a particular situation requires the necessary knowledge and skills as well as personal beliefs of efficacy to meet the demands of a specific situation (Bandura, 1977). By interacting with virtual patients, students’ self-efficacy improved after taking a patient’s history, conducting screening and assessment, and providing holistic and patient-centered care in a virtual patient activity. Following self-efficacy theory, the student’s conceptions of skills, based on their interaction with virtual patients, could serve as a guide for developing competencies and an internal standard for improving them.

Goffman (1990) defined stigma as an “attribute that is deeply discrediting” and that reduces the bearer “from a whole and usual person to a tainted, discounted one” (Goffman, 1990, p. 3). Individuals with mental illness have reported feeling devalued, dismissed, and dehumanized by health professionals (Hamilton et al., 2016) such that mental illness-related stigma can be a barrier to patient access to treatment and recovery (Abbey et al., 2011; Knaak et al., 2017; Thornicroft et al., 2007). Virtual patients, who portray the psychiatric setting and population more realistically, can help dispel and address stigma and attitudes before encountering a real patient.

Research gaps and directions for the future

The majority of the reviewed studies were carried out in the field of medical education. However, psychiatry is a specialty that requires collaboration, and interprofessional education is seen as a means of improving cooperative competencies and practices. Previous studies have reported significant challenges to interprofessional education, such as logistical problems and organizational barriers to planning sessions for students from different programs or universities (Priest et al., 2008). Future research should consider the role of virtual patients to improve interprofessional education, where students from different healthcare education programs could learn about, from, and with each other. For example, students from different programs could be allowed to work together on virtual patient scenarios in online platforms, or the virtual patient case could be designed in ways that allow the students to emulate the role of different healthcare professionals (e.g. nurse, doctor, social-worker).

We found that virtual patients have been used in especially geriatric psychiatry. This finding is surprising given that a recent scoping review on simulation-based education in healthcare suggests a dearth of research on the elderly population (Williams et al., 2017). We noted a lack of focus on pediatric patients and young adults with mental illness. Virtual patients can be specifically suited to address these patient populations because of their non-obtrusive interactions with an otherwise vulnerable patient population. We also noted a gap regarding transcultural undergraduate psychiatry education, which, given the increasing number of psychiatric patients with diverse ethnic backgrounds, is a health priority (Pantziaras et al., 2015).

Finally, the majority of the reviewed studies only focused on learners’ immediate performance following the educational intervention. The assumption seemed to be that any increase in levels of knowledge and skills or changes in attitudes evidenced immediately after the intervention represents the amount of learning that occurs from the intervention itself (Shariff et al., 2020). However, research suggests that peak performance immediately following training often overestimates the amount learned (Bjork & Bjork, 2011; Soderstrom & Bjork, 2015; Stefanidis et al., 2005). Thus, in addition to baseline and immediate post-intervention measures, future studies could include follow-up assessments to evaluate performance over time and focus on ongoing learning and extended retention of knowledge, skills, and attitudes (Kononowicz et al., 2020).

The strengths of this review was the use of thematic analysis to synthesize learning outcomes associated with the use of virtual patients, our comprehensive search strategies, and rigorous quality appraisals of the studies included in the review. By using a definition of virtual patients that was broader than that used in one previous review on virtual patients in medical education (Cook & Triola, 2009), we allowed for inclusion of interventions using different virtual patient formats. This made the findings more generalizable than if we had chosen a more narrowly focused set of literature.

Several limitations should be noted. First, articles published in languages other than English were excluded from this review, and data may have been missed. Second, the majority of the studies have been conducted in high-income countries. Although a previous review on virtual patients in medical education has collected positive evidence of effectiveness from both high-income and low-income countries (Kononowicz et al., 2019), we caution against simplistic conclusions about the cross-cultural effectiveness of virtual patients. Third, the greatest limitation across the included studies was the lack of or poor reporting of the validity of the evaluation instruments, indirectly providing the evidence base for study findings. Nevertheless, we did not exclude studies based on their quality due to our aim of providing an overview of all relevant research on virtual patients in psychiatry education during the past two decades. Still, in addition to the quality assessment, an assessment of risk of bias would have helped to establish transparency of the evidence synthesis. Fourth, the thematic analyses of learning outcomes revealed that the included studies lack the patient outcome data as described in Kirkpatrick’s four-level model for effective evaluation of educational training programs (Kirkpatrick, 1996). Such translation research serves an important role, and prudent pursuit of patient outcomes could advance the field.

In summary, in this systematic review, we identified 46 studies addressing the use of virtual patient interventions in undergraduate psychiatry education to develop students’ knowledge, skills, and attitudes, published between 2000 and January 2021. We investigated the most used approaches, examined the different learning outcomes, and described their effectiveness.

This review shows that virtual patients can afford valuable learning opportunities for students from different disciplines engaging in psychiatry education. Virtual patients, by providing access to safe, repeated, and consistent practice, can help students develop their knowledge about symptomatology and psychopathology and their interpersonal skills and clinical skills. Furthermore, virtual patients can help to increase students’ self-efficacy and confidence concerning their knowledge and skills. Finally, virtual patients can be used to develop more positive and less stigmatizing attitudes toward individuals with mental illness. However, we note that the predefined outcome measures in the studies included in our review did not fully cover the educational potential of virtual patients (Edelbring et al., 2011). More research is needed into how students use virtual patients and what benefits they afford.