Abstract
Background
Residents’ understanding of diagnostic error may differ between countries. We sought to explore the relationship between diagnostic error knowledge and self-study, clinical knowledge, and experience.
Methods
Our nationwide study involved postgraduate year 1 and 2 (PGY-1 and -2) Japanese residents. The Diagnostic Error Knowledge Assessment Test (D-KAT) and General Medicine In-Training Examination (GM-ITE) were administered at the end of the 2014 academic year. D-KAT scores were compared with the benchmark scores of US residents. Associations between D-KAT score and gender, PGY, emergency department (ED) rotations per month, mean number of inpatients handled at any given time, and mean daily minutes of self-study were also analyzed, both with and without adjusting for GM-ITE scores. Student’s t test was used for comparisons with linear mixed models and structural equation models (SEM) to explore associations with D-KAT or GM-ITE scores.
Results
The mean D-KAT score among Japanese PGY-2 residents was significantly lower than that of their US PGY-2 counterparts (6.2 vs. 8.3, p < 0.001). GM-ITE scores correlated with ED rotations (≥6 rotations: 2.14; 0.16–4.13; p = 0.03), inpatient caseloads (5–9 patients: 1.79; 0.82–2.76; p < 0.001), and average daily minutes of self-study (≥91 min: 2.05; 0.56–3.53; p = 0.01). SEM revealed that D-KAT scores were directly associated with GM-ITE scores (ß = 0.37, 95% CI: 0.34–0.41) and indirectly associated with ED rotations (ß = 0.06, 95% CI: 0.02–0.10), inpatient caseload (ß = 0.04, 95% CI: 0.003–0.08), and average daily minutes of study (ß = 0.13, 95% CI: 0.09–0.17).
Conclusions
Knowledge regarding diagnostic error among Japanese residents was poor compared with that among US residents. D-KAT scores correlated strongly with GM-ITE scores, and the latter scores were positively associated with a greater number of ED rotations, larger caseload (though only up to 15 patients), and more time spent studying.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
INTRODUCTION
In 2004, the Japanese Ministry of Health, Labor and Welfare established a new postgraduate clinical training program, mandating that residents complete a 2-year internship after graduating from medical school and passing the National Examination for Medical Practitioners. In those 2 years, residents are expected to achieve a basic set of skills for primary care practice, obtained through rotations in internal medicine, surgery, and emergency medicine during the first year, and pediatrics, obstetrics and gynecology, psychiatry, and community medicine during the second year. In 2010, the required departments for rotation were reduced from seven to three: internal medicine, emergency department (ED), and community medicine. Residents are allowed to rotate on surgery, anesthesiology, pediatrics, OB-GYN, or psychiatry as electives. After this 2-year residency, they choose their subspecialty program.
Knowledge of diagnostic errors such as cognitive bias or heuristics is an important factor in preventing such errors. No studies to date have explored experiential variables, such as workload, ER rotations, or minutes of study per day, that might improve D-KAT scores. In a previous study, we found that GM-ITE scores were improved with optimal resident caseload, ED rotations, and access to online resources.1 The purpose of the current study was to assess diagnostic error knowledge among residents throughout Japan and to compare the results with the benchmark scores in a previous US study.2 We also evaluate the relationships between diagnostic error knowledge and self-study, clinical knowledge, and experience.
METHODS
We conducted a nationwide study of Japanese PGY-1 and -2 residents. The Diagnostic Error Knowledge Assessment Test (D-KAT) and General Medicine In-Training Examination (GM-ITE) were administered at the end of the 2014 academic year. Immediately after the D-KAT and GM-ITE were conducted, the participants completed a questionnaire regarding their clinical experience and self-study habits, assessing ED rotations per month (the average frequency of night duty per month), the mean number of inpatients handled at one time, and mean daily minutes of self-study.
Diagnostic error knowledge was assessed using the D-KAT, a 13-item multiple-choice test developed and validated by Reilly et al. in 2010.2 The GM-ITE is a clinical knowledge test that uses a methodology similar to that of the US Internal Medicine In-Training Examination (IM-ITE).3 The purpose of the IM-ITE is to provide residents and program directors with an objective, reliable, and valid assessment of clinical knowledge through a multiple-choice examination.4 , 5 Our GM-ITE, which was designed and written by a committee of experienced physicians assembled by the Japan Institute for Advancement of Medical Education Program (a non-profit organization in Tokyo, Japan), included 87 questions testing a wide range of clinical knowledge, from clinical skills and practical medical knowledge to psychosocial care of patients.6
The mean D-KAT scores were compared between Japanese and US residents using Student’s t test.2 In addition, the associations between D-KAT scores and gender, PGY, ED rotations per month, mean number of inpatients handled at any given time, and mean daily minutes of self-study were analyzed with or without adjusting for GM-ITE scores (calculated by the domains other than the D-KAT score). Linear mixed models were used to account for hospital variability as random intercepts, with Huber-White (sandwich) variance estimators. Answers of “unknown” (for ED duty and inpatient caseload) were retained in the analyses, whereas non-responders for either ED duty, inpatient caseload, or self-study (N = 224) were excluded from mixed modeling. We also conducted the same analyses for the D-KAT score as a dependent variable, with and without adjusting for the GM-ITE score (except for the D-KAT score). A two-tailed p value of <0.05 was considered statistically significant. Analyses were performed using SAS version 9.4 software (SAS Institute Inc., Cary, NC). Structural equation models were built to test for the direct and indirect determinants of D-KAT performance using standardized estimates of effect and standard goodness-of-fit measures (STATA 14; StataCorp LP, College Station, TX).7 The study was approved by the Ethics Committee of the Tsukuba University Mito Medical Center.
RESULTS
A total of 2652 residents (1123 PGY-1 and 1529 PGY-2) from 258 teaching hospitals participated in the examination; 777 (29.3%) of participants were women. The voluntary survey response rate was 91.6% (2429/2652). Among respondents, the weighted average number of ED rotations was 3.94, with 3% having no rotations, 8% with 1–2, 70% with 3–5, and 17% with 6 or more rotations. By self-report, residents managed an average of 6.9 inpatients (weighted average value): 9% reported managing 4 or fewer, 64% reported 5–9, 19% noted 10–14, and 6% reported 15 or more. Residents reported spending an average of 42.2 min studying (weighted average value), with 39% spending less than 30 min, 38% between 31 and 60 min, 13% 61–90 min, and 4% 91 min or more of daily self-study.
The mean (± SD) D-KAT score among Japanese PGY-2 residents (6.2 ± 1.6) was significantly lower than that of their US PGY-2 counterparts (8.3 ± 1.5; p < 0.001).2 Similar to US resident scores, GM-ITE scores increased from PGY-1 to PGY-2 (6.1 to 6.2, respectively; p = 0.018).8 Using linear mixed models, the D-KAT scores were found to be associated with GM-ITE scores and inpatient caseload (Table 1). The GM-ITE scores were associated with D-KAT scores (estimate of change in score, 0.64; 95% confidence interval, 0.45–0.83; p < 0.001), ED rotations (≥6 rotations: 2.14; 0.16–4.13; p = 0.03), inpatient caseloads (5–9 patients: 1.79; 0.82–2.76; p < 0.001), and average daily minutes of self-study (≥91 min: 2.05; 0.56–3.53; p = 0.01; Table 2).
SEM models indicated a direct relationship between GM-ITE scores and D-KAT performance (ß = 0.37, 95% CI: 0.34–0.41; Fig. 1), and suggested that D-KAT scores were indirectly affected by the number of ED rotations (ß = 00.06, 95% CI: 0.02–0.10), inpatient caseload (ß = 00.04, 95% CI: 0.003–0.08), and average daily minutes of study (ß = 00.13, 95% CI: 0.09–0.17). Our model was well fitted to our data (root mean square error of approximation [RMSEA]: 0.03; comparative fit index [CFI]: 0.98; Tucker-Lewis index [TLI]: 0.96).
DISCUSSION
It is unclear why Japanese residents scored more poorly on diagnostic error knowledge than US residents. We can infer, however, that graduate and postgraduate medical education on clinical reasoning has not reached a level similar to that in the US. Medical education in clinical clerkships in Japan has long been provided during subspecialty rotations that occur in the PGY years following the first 2 years. This long-held educational tradition has brought fewer educational opportunities for clinical reasoning and even postgraduate clinical training because of the short training period in the department of general medicine (GM) or general internal medicine (GIM).6 Compared to subspecialty physicians, teachers of GIM or GM departments may be better able to provide their students and residents specific clinical problem-solving knowledge, including that for diagnostic error.
Our study suggests that self-study, time spent in ED rotations, and caseload are all directly associated with GM-ITE scores, and are indirectly associated with D-KAT scores through the relationship between GM-ITE and D-KAT scores. A greater number of ED rotations and minutes of self-study and a larger inpatient caseload appeared to contribute to general improvement in clinical knowledge and skill. An appropriate caseload in both emergency and inpatient settings may provide residents with the experience needed to reflect on various cognitive biases, leading to improved clinical and diagnostic reasoning. In addition, the development of independent curricula, such as simulation-based training, reflective practice, or active metacognitive review, may be useful.9,10, – 11 Our SEM models allowed us to explore the potential causal relationships between our variables, but given the nature of our cross-sectional data, our results can only be speculative. Our findings should be validated in a larger longitudinal study.
In conclusion, diagnostic error knowledge among Japanese residents was poor compared with that among US residents. GM-ITE performance was related to scores on D-KAT. Diagnostic error knowledge was indirectly related to increased clinical experience and self-study. Resident diagnostic error performance may improve with greater opportunities for reflection on inpatient and emergency patient conditions and additional independent curricula.10 , 11
References
Kinoshita K, Tsugawa Y, Shimizu T, et al. Impact of inpatient caseload, emergency department duties, and online learning resource on General Medicine In-Training Examination scores in Japan. Int J Gen Med. 2015;8:355–60.
Reilly JB, Ogdie AR, Von Feldt JM, et al. Teaching about how doctors think: a longitudinal curriculum in cognitive bias and diagnostic error for residents. BMJ Qual Saf. 2013;22:1044–50.
McCoy CP, Stenerson MB, Halvorsen AJ, et al. Association of volume of patient encounters with residents’ in-training examination performance. J Gen Intern Med. 2013;28:1035–41.
Kanna B, Gu Y, Akhuetie J, et al. Predicting performance using background characteristics of international medical graduates in an inner-city university-affiliated Internal Medicine residency training program. BMC Med Educ. 2009;9:42.
Perez JA, Greer S. Correlation of United States Medical Licensing Examination and Internal Medicine In-Training Examination performance. Adv Health Sci Educ Theory Pract. 2009;14:753–8.
Shimizu T, Tsugawa Y, Tanoue Y, et al. The hospital educational environment and performance of residents in the General Medicine In-Training Examination: a multicenter study in Japan. Int J Gen Med. 2013;6:637–40.
Kline RB. Principles and practice of structural equation modeling. New York: Guilford Press; 1998.
Kay C, Jackson JL, Frank M. The relationship between Internal Medicine Residency Graduate performance on the ABIM certifying examination, yearly in-service examinations and attending evaluations. Acad Med. 2015;90(1):100–4.
Schmidt HG, Mamede S. How to improve the teaching of clinical reasoning: a narrative review and a proposal. Med Educ. 2015;49:961–73.
Graber ML, Kissam S, Payne VL, et al. Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf. 2012;21:535–57.
Singh H, Schiff GD, Graber ML, et al. The global burden of diagnostic errors in primary care. BMJ Qual Saf. 2017;26:484–94.
Acknowledgements
The authors would like to thank Takeshi Asakura for help in translating this manuscript into English, and we thank all members of the GM-ITE production committee for their support. We would also like to thank Jeffrey Jackson, MD, MPH, for his editorial work and assistance with the SEM models.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Rights and permissions
About this article
Cite this article
Nishizaki, Y., Shinozaki, T., Kinoshita, K. et al. Awareness of Diagnostic Error among Japanese Residents: a Nationwide Study. J GEN INTERN MED 33, 445–448 (2018). https://doi.org/10.1007/s11606-017-4248-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-017-4248-y