Abstract
Graduate medical education is moving toward a competency-based paradigm, predicated upon multiple real-time assessments to verify clinical and technical proficiency (i.e., readiness for entrustment of residents). This requires not only assessment of technical skills and medical knowledge but also critical clinical decision-making skills in preoperative, intraoperative, and postoperative settings. However, most medical education programs have adopted reductionist approaches, reducing assessment of readiness for entrustment to only assessing technical skill performance. As such, there is a growing need for tools that can provide more comprehensive and objective evaluations of the proficiency of residents to perform medical procedures. This chapter presents ENTRUST, our serious game-based online platform to assess trainees’ decision-making competence across various Entrustable Professional Activity (EPA) domains. Specifically, we discuss (1) the design of ENTRUST; (2) insights identified and lessons learned throughout the development process that can aid collaboration between serious game developers and subject matter experts; and (3) results from a pilot study of ENTRUST—demonstrating the tool’s capability to discriminate between levels of surgical expertise and providing initial validity evidence for its use as an objective assessment for clinical decision-making.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Keywords
References
ABS E-News – Spring 2018. In: News from the American Board of Surgery. The American Board of Surgery (2018). http://www.absurgery.org/quicklink/absnews/absupdate0518.html#epa. Accessed 12 Jan 2021
Ardito C., Buono P., Costabile M.F., Lanzilotti R., Piccinno A.: End users as co-designers of their own tools and products. J. Visual Lang. Comput. 23(2), 78–90 (2012)
Barnard-Brak, L., Paton, V.O., Lan, W.Y.: Profiles in self-regulated learning in the online learning environment. Int. Rev. Res. Open Distrib. Learn. 11(1), 61–80 (2010)
Berman, N.B., Durning, S.J., Fischer, M.R., Huwendiek, S., Triola, M.M.: The role for virtual patients in the future of medical education. Acad. Med. 91(9), 1217–1222 (2016)
Bird, M., McGillion, M., Chambers, E.M., Dix, J., Fajardo, C.J., Gilmour, M., Levesque, K., Lim, A., Mierdel, S., Ouellette, C., Polanski, A.N., Reaume, S.V., Whitmore, C., Carter, N.: A generative co-design framework for healthcare innovation: development and application of an end-user engagement framework. Res. Involv. Engagem. 7(1), 1–12 (2021)
Bohnen, J.D., George, B.C., Williams, R.G., Schuller, M.C., DaRosa, D.A., Torbeck, L., Mullen, J.T., Meyerson, S.L., Auyang, E.D., Chipman, J.G., Choi, J.N.: The feasibility of real-time intraoperative performance assessment with SIMPL (system for improving and measuring procedural learning): early experience from a multi-institutional trial. J. Surg. Educ. 73(6), e118–e130 (2016)
Brasel, K.J., Klingensmith, M.E., Englander, R., Grambau, M., Buyske, J., Sarosi, G., Minter, R.: Entrustable professional activities in general surgery: development and implementation. J. Surg. Educ. 76(5), 1174–1186 (2019)
Burkett, I.: An Introduction to Co-design, vol. 12. Knode, Sydney (2012)
Charlier, N.: Game-based assessment of first aid and resuscitation skills. Resuscitation 82(4), 442–446 (2011)
Chon, S.H., Timmermann, F., Dratsch, T., Schuelper, N., Plum, P., Berlth, F., Datta, R.R., Schramm, C., Haneder, S., Späth, M.R., Dübbers, M., Kleinert, J., Raupach, T., Bruns, C., Kleinert, R.: Serious games in surgical medical education: a virtual emergency department as a tool for teaching clinical reasoning to medical students. JMIR Serious Games 7(1), 1–11 (2019)
Cianciolo, A.T., Kegg, J.A.: Behavioral specification of the entrustment process. J. Grad. Med. Educ. 5(1), 10–12 (2013)
Cook, D.A., Zendejas, B., Hamstra, S.J., Hatala, R., Brydges, R.: What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv. Health Sci. Educ. 19(2), 233–250 (2014)
de Klerk, S., Kato, P.M.: The future value of serious games for assessment: Where do we go now?. J. Appl. Testing Technol. 18(S1), 32–37 (2017)
Domecq, J.P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., Brito, J.P., Boehmer, K., Hasan, R., Firwana, B., Erwin, P., Eton, D., Sloan, J., Montori, V., Asi, N., Dabrh, A.M.A., Murad, M.H.: Patient engagement in research: a systematic review. BMC Health Serv. Res. 14(1), 1–9 (2014)
Eaton, M., Scully, R., Schuller, M., Yang, A., Smink, D., Williams, R.G., Bohnen, J.D., George, B.C., Fryer, J.P., Meyerson, S.L.: Value and barriers to use of the SIMPL tool for resident feedback. J. Surg. Educ. 76(3), 620–627 (2019)
Ferreira-Brito, F., Fialho, M., Virgolino, A., Neves, I., Miranda, A.C., Sousa-Santos, N., Caneiras, C., Carrico, L., Verdelho, A., Santos, O.: Game-based interventions for neuropsychological assessment, training and rehabilitation: which game-elements to use? A systematic review. J. Biomed. Inform. 98, 103287 (2019)
George, B.C., Bohnen, J.D., Williams, R.G., Meyerson, S.L., Schuller, M.C., Clark, M.J., Meier, A.H., Torbeck, L., Mandell, S.P., Mullen, J.T., Smink, D.S.: Readiness of US general surgery residents for independent practice. Ann. Surg. 266(4), 582–594 (2017)
George, B.C., Bohnen, J.D., Schuller, M.C., Fryer, J.P.: Using smartphones for trainee performance assessment: a SIMPL case study. Surgery 167(6), 903–906 (2020)
Guise, J.M., O’Haire, C., McPheeters, M., Most, C., LaBrant, L., Lee, K., Cottrell, E.K.B., Graham, E.: A practice-based tool for engaging stakeholders in future research: a synthesis of current practices. J. Clin. Epidemiol. 66(6), 666–674 (2013)
Hwang, G.J., Chang, C.Y. Facilitating decision-making performances in nursing treatments: a contextual digital game-based flipped learning approach. Interactive Learn. Environ. 31(1), 1–16 (2020)
Johnsen, H.M., Fossum, M., Vivekananda-Schmidt, P., Fruhling, A., Slettebø, Å.: Teaching clinical reasoning and decision-making skills to nursing students: design, development, and usability evaluation of a serious game. Int. J. Med. Inform. 94, 39–48 (2016)
Kalyuga, S., Plass, J.L.: Evaluating and managing cognitive load in games. In: Handbook of Research on Effective Electronic Gaming in Education, pp. 719–737. IGI Global, Pennsylvania (2009)
Lagro, J., van de Pol, M.H., Laan, A., Huijbregts-Verheyden, F.J., Fluit, L.C., Rikkert, M.G.O.: A randomized controlled trial on teaching geriatric medical decision making and cost consciousness with the serious game GeriatriX. J. Am. Med. Direct. Assoc. 15(12), e1–957.e6 (2014)
Liebert, C.A., Mazer, L., Merrell, S.B., Lin, D.T., Lau, J.N.: Student perceptions of a simulation-based flipped classroom for the surgery clerkship: a mixed-methods study. Surgery 160(3), 591–598 (2016)
Liebert, C.A., Melcer, E.F., Keehl, O., Eddington, H., Trickey, A.W., Lee, M., Tsai, J., Camacho, F., Merrell, S.B., Korndorffer, Jr. J.R., Lin, D.T.: Validity evidence for ENTRUST as an assessment of surgical decision-making for the inguinal hernia entrustable professional activity (EPA). J. Surg. Educ. 79(6), e202–e212 (2022)
Lin, D.T., Park, J., Liebert, C.A., Lau, J.N.: Validity evidence for surgical improvement of clinical knowledge ops: a novel gaming platform to assess surgical decision making. Am. J. Surg. 209(1), 79–85 (2015)
Mavridis, A., Tsiatsos, T.: Game-based assessment: investigating the impact on test anxiety and exam performance. J. Comput. Assist. Learn. 33(2), 137–150 (2017)
Messick, S.: Standards of validity and the validity of standards in performance asessment. Educ. Measur. Issues Pract. 14(4), 5–8 (1995)
Middeke, A., Anders, S., Schuelper, M., Raupach, T., Schuelper, N.: Training of clinical reasoning with a serious game versus small-group problem-based learning: a prospective study. PLoS One 13(9), e0203851 (2018)
Nemirovsky, D.R., Garcia, A.J., Gupta, P., Shoen, E., Walia, N.: Evaluation of surgical improvement of clinical knowledge ops (SICKO), an interactive training platform. J. Digit. Imag. 34(4), 1067–1071 (2021)
New model of surgical resident autonomy coming in 2023. In: ACS Clinical Congress News (2021). Published October 23, 2021. Accessed 21 Jan 2022. https://www.acsccnews.org/new-model-of-surgical-resident-autonomy-coming-in-2023/
Nikolian, V.C., Sutzko, D.C., Georgoff, P.E., Matusko, N., Boniakowski, A., Prabhu, K., Church, J.T., Thompson-Burdine, J., Minter, R.M., Sandhu, G.: Improving the feasibility and utility of OpTrust–a tool assessing intraoperative entrustment. Am. J. Surg. 216(1), 13–18 (2018)
Oestreich, J.H., Guy, J.W.: Game-based learning in pharmacy education. Pharmacy 10(1), 11 (2022)
Plass, J.L., Moreno, R., Brünken, R.: Cognitive Load Theory. Cambridge University Press, Cambridge (2010)
R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna (2013). https://www.R-Project.org/
Salsberg, J., Parry, D., Pluye, P., Macridis, S., Herbert, C.P., Macaulay, A.C.: Successful strategies to engage research partners for translating evidence into action in community health: a critical review. J. Environ. Pub. Health 2015, 1–15 (2015)
Sánchez, R., Rodríguez, O., Rosciano, J., Vegas, L., Bond ,V., Rojas, A., Sanchez-Ismayel, A.: Robotic surgery training: construct validity of Global Evaluative Assessment of Robotic Skills (GEARS). J. Robot. Surg. 10(3), 227–231 (2016)
Sandhu, G., Nikolian, V.C., Magas, C.P., Stansfield, R.B., Sutzko, D.C., Prabhu, K., Matusko, N., Minter, R.M.: OpTrust: validity of a tool assessing intraoperative entrustment behaviors. Ann. Surg. 267(4), 670–676 (2018)
Schuler, D., Namioka, A.: Participatory Design: Principles and Practices. CRC Press, Boca Raton (1993)
Seelow, D.: The art of assessment: using game based assessments to disrupt, innovate, reform and transform testing. J. Appl. Testing Technol. 20(S1), 1–16 (2019)
Slattery, P., Saeri, A.K., Bragge, P.: Research co-design in health: a rapid overview of reviews. Health Res. Policy Syst. 18(1), 1–13 (2020)
Steen, M.: Co-design as a process of joint inquiry and imagination. Des. Issues 29(2), 16–28 (2013)
Ten Cate, O., Scheele, F.: Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad. Med. 82(6), 542–547 (2007)
Ten Cate, O., Chen, H.C., Hoff, R.G., Peters, H., Bok, H., van der Schaaf, M.: Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE guide no. 99. Med. Teach. 37(11), 983–1002 (2015)
Ten Cate, O., Carraccio, C., Damodaran, A., Gofton, W., Hamstra, S.J., Hart, D.E., Richardson, D., Ross, S., Schultz, K., Warm, E.J., Whelan, A.J., Schumacher, D.J.: Entrustment decision making: extending Miller’s pyramid. Acad. Med. 96(2), 199–204 (2021)
Vallejo, V., Wyss, P., Rampa, L., Mitache, A.V., Müri, R.M., Mosimann, U.P., Nef, T.: Evaluation of a novel Serious Game based assessment tool for patients with Alzheimer’s disease. PLoS One 12(5), e0175999 (2017)
Vassiliou, M.C., Feldman, L.S., Andrew, C.G., Bergman, S., Leffondré, K., Stanbridge, D., Fried, G.M.: A global assessment tool for evaluation of intraoperative laparoscopic skills. Am. J. Surg. 190(1), 107–113 (2005)
Verma, V., Baron, T., Bansal, A., Amresh, A.: Emerging practices in game-based assessment. In: Game-Based Assessment Revisited, pp. 327–346. Springer, Cham (2019)
Yeo, H.L., Dolan, P.T., Mao, J., Sosa, J.A.: Association of demographic and program factors with American Board of Surgery qualifying and certifying examinations pass rates. JAMA Surg. 155(1), 22–30 (2020)
Acknowledgements
The authors would like to thank Yulin Cai, Ruonan Chen, Pichy Jumpholwong, Ananya Anand, and Sherry Wren for their contributions to the development of the ENTRUST platform.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Melcer, E.F. et al. (2023). ENTRUST: Co-design and Validation of a Serious Game for Assessing Clinical Decision-Making and Readiness for Entrustment. In: Cooper, K.M.L., Bucchiarone, A. (eds) Software Engineering for Games in Serious Contexts. Springer, Cham. https://doi.org/10.1007/978-3-031-33338-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-33338-5_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-33337-8
Online ISBN: 978-3-031-33338-5
eBook Packages: Computer ScienceComputer Science (R0)