Introduction

The practice of utilizing something as mundane as completion of required, routine, or administrative tasks is a commonplace measure of medical students’ level of conscientiousness as it relates to expected professional behavior [1]. The term conscientious is defined by dictionary sources including Cambridge, Oxford, and Merriam-Webster as the desire or sense of responsibility to do a task well, thoroughly, and reliably. Our survey of the medical education literature revealed that research conducted over the past two decades has successfully established conscientiousness indices as a predictive proxy for professional behavior. These professionalism behaviors relate to lapses or failure to meet duties and responsibilities, such as missed deadlines, unexcused absences, and tardiness [2,3,4,5,6,7,8,9]. This has important implications as the literature shows that unprofessional behavior displayed by students during medical school foretells disciplinary actions from state medical boards, up to and including license revocation [10]. Physicians identified during medical school for professionalism lapses were twice as likely to be disciplined by state medical boards as practicing physicians than matched controls without documented professionalism issues [4]. In a study that compared disciplinary sanctions from state medical boards to medical students’ early concern notes, the researchers found striking similarities. Of 516 instances of physician disciplinary actions, 77% (398) were related to failure to meet professional responsibilities. In the same vein, of 110 early concern notes of medical students, 59% (44) were related to failure to meet professional responsibilities for years 1 and 2 and 82% (54) for years 3 and 4 [7]. Repeatedly, significant correlations between conscientiousness indices and unprofessional behaviors have been illustrated by many researchers [6, 8, 11, 12].

The focus of conscientiousness research as it relates to academic performance has been observed to a lesser extent. One study in a graduate medical education setting explored the implications of conscientiousness on exceptional professional behavior [13]. This work entailed performing assessments of professionalism among 148 internal medicine, post-graduate year-1 residents using feedback from peers, senior residents, faculty, medical students, and non-physician professionals. Findings showed that residents who had professionalism scores within the top 20% achieved higher median scores on the in-training examination and the mini-CEX evaluation (a direct observation assessment of trainee-patient interactions). Further, residents that completed 70% of their evaluations were four times more likely to score in the top 20% of the professionalism ratings than residents that completed only 50% of their evaluations. More directly related to our study, Wright [14] examined the relationship between end-of-year examination outcomes and medical students’ completion of simple administrative tasks (providing a passport photograph; N = 366). Of the 27 non-compliant students, 13 failed the exam. Rudland [5] related the academic performance of 85 fourth-year medical students to course evaluation completion rates. Students that completed end-of-course evaluations within the required time frame were deemed responders (59%). Students who responded after receiving prompts to complete were deemed late responders (34%) and those who did not comply at all were deemed non-responders (7%). The academic performance for the late responders and non-responders groups were lower than the responders group; 18% of responders needed to retake a course examination compared to 33% of late responders and non-responders.

As evidenced in this brief review, few researchers have linked conscientiousness to academic achievement; yet, the relationships illustrated by their findings have important implications that impel further investigation. If completion of routine tasks predict educational achievement in undergraduate and graduate medical education, strategies could be developed for early identification of at-risk learners. Preventing or addressing academic underperformance is of paramount importance, particularly when early identification is possible [15, 16]. Preliminary evidence suggests there is a correlation between noncompliance and academic performance [5, 13, 14]. The predictive potential of noncompliance of routine tasks would enable institutions to develop interventions aimed at preventing or minimizing future academic challenges. We believe  that researchers should move beyond noncompliance in terms of professionalism lapses and begin to explore its academic implications. Consequently, in this study we explore the predictive value of a noncompliance index (NCI) to subsequent performance on Step 1 board examinations and competency-based academic promotion decisions. We anticipate that our findings will help others develop early detection strategies to prevent remediation or even academic failure.

Materials and Methods

We used a retrospective, cohort-based design to explore whether medical students’ behavior, as measured through the completion of required tasks, may predict USMLE Step 1 performance and/or decisions made about their academic promotion decisions. Participants were medical students from eight class cohorts from 2011 through 2018, at the Cleveland Clinic Lerner College of Medicine of Case Western Reserve University (CCLCM). The selection criteria included those who completed years 1 and 2 of medical school (years 1 and 2), took USMLE Step 1, and consented to release program evaluation data for research purposes.

Data Sources

After obtaining ethical approval from the Cleveland Clinic’s Office of Institutional Review Board, we used existing medical school records for each student in three areas, student background, conscientiousness measures, and performance measures as outlined below. We coded gender as male or female and aggregated Medical College Admission Test (MCAT) scores in each of the domains (verbal, physical science, and biological science); the scores were then used as covariates for the statistical analyses (Table 1).

Table 1 Student characteristics
Full size table

Noncompliance Index Measures

The noncompliance index measures consisted of missing end-of-course evaluations, tardy submissions of concept appraisals (CAPPs), and non-completion of self-assessment questions (SAQs). CAPPs are required weekly essays used to assess students’ application and depth of medical knowledge [17]. SAQs are required weekly multiple-choice questions (25–30 items) used to help students self-assess their breadth of medical knowledge [17]. Unlike CAPPs, the SAQ scores are not reported to course directors. Further, CAPPs are assessed by faculty; thus, students may view CAPPs as a higher stake assessment. It is worth noting that our index measured compliance; the actual scores or content of these assessments were not taken into account, only whether the tasks were completed.

The missing end-of-course evaluations were coded as 1 for every instance, during years 1 and 2, when a student did not electronically submit a course evaluation, within the 2-week allotted time frame, as required by the medical school’s professionalism policy. Tardy submission of required CAPPs were coded as 1 for every instance when a student did not electronically submit CAPPs by the 8-AM Monday deadline.

Every instance when a student accessed an SAQ during years 1 and 2 and immediately exited without attempting to answer a single question was coded as 1. If a student scored zero, based on performance (student answered all questions incorrectly) such instance was not used in the noncompliance index. That is, noncompliance was coded only when students made no attempt to answer any questions on a given quiz, which demonstrated that a zero score was not reflective of the student’s performance or knowledge, but rather unprofessional behavior.

Student Performance

Student performance indicators consisted of students’ first attempt score on the USMLE Step 1 and overall promotion decisions which includes either advancement or remediation. CCLCM medical students take USMLE Step 1 approximately 6–8 weeks after completing year 2. Competency-specific promotion decision during years 1 and 2 entails having students submit a summative portfolio at the end of year 1, year 2, and year 5 to document their performance in nine competencies. A promotion committee reviews each student’s summative portfolio and makes competency-based decisions about individual performance [18]; these decisions include pass, concerns, remediation, or fail (Table 2). Required formal remediation during medical school occurs when a student requires closer monitoring to address performance deficits while continuing with the medical school curriculum.

Table 2 Association among noncompliance index year 1 and promotion decisions
Full size table

For this investigation, we selected performance decisions made about students’ performance on the competencies of professionalism, communication skills, and medical knowledge. We dichotomized the decisions into (a) pass or (b) concerns/remediation. Similarly, we dichotomized the overall promotion decisions of either advancement or remediation into (a) pass or (b) concerns/remediation/fail, respectively. A code of 1 was assigned for every instance when a student was placed in remediation by the school’s promotion committee.

Statistical Analysis

Descriptive statistics summarize study participant characteristics, mean ± standard deviation, or count (percent) as appropriate. NCI comparisons among genders were conducted with independent t tests. The association among NCI and USMLE Step 1 scores, adjusting for both gender and MCAT score, was explored with multivariable linear regression models. The Wilcoxon rank-sum tests investigated associations among NCI and subsequent promotion decisions.

Scores for missing course evaluations, tardy CAPPs, and zero-score SAQs were summed for each student to measure the incidence of these behaviors during year 1, year 2, and years 1 and 2, with higher scores denoting greater incidences of non-completion of required tasks. We subsequently combined these variables to a form a noncompliance index (NCI), which had a Cronbach’s alpha of 0.713 for the sample NCI scores from years 1 and 2. A level of 0.05 was utilized for defining statistical significance. All analyses were performed with and without outliers. Conclusions remained unchanged and thus, we present the results of the full dataset only. All statistical analyses were performed with SAS version 9.3 (SAS Institute, Cary, NC).

Results

Characteristics of students included in this study are summarized in Table 1. Distributions of the noncompliance index (NCI) for year 1 and years 1 and 2 are positively skewed. The majority of students scored below 2. However, (4%) 10 students scored above 5 on the NCI in year 1, and (15%) 38 students scored above 5 on the NCI in years 1 and 2. One outlier is present who scored 28 on the NCI year 1 and 52 on the NCI years 1 and 2. All three components of the NCI contributed to the overall score. The impact of their respective scores influences ranks SAQs, CAPPs, and course evaluations as increasing areas of negligence.

The data do not support an association among NCI scores and gender (NCI year 1 female mean = 1.43 vs. male mean = 1.44, p = 0.98; NCI year 1 and year 2 female mean = 2.73 vs. male mean = 2.63, p = 0.86). NCI scores are associated with scores on USMLE Step 1. Adjusting for both gender and MCAT scores, each point increase in NCI year 1 results in a decrease of 1.6 points in USMLE Step 1 scores on average (95% CI − 2.7 to − 0.5, p = 0.005). Similarly, each point increase in NCI year 1 and year 2 results in a decrease of 1.3 points in USMLE Step 1 scores on average (95% CI − 2.0 to − 0.6, p = 0.0005). MCAT is also associated (p < 0.0001) with USMLE Step 1. Each point increase in MCAT results in an increase of 2.5 points in USMLE Step 1 scores on average (95% CI 1.8 to 3.2). Associations among NCI year 1 and subsequent promotion decisions are presented in Table 2. A student’s NCI year 1 is predictive of their overall year 2 promotion decision (p = 0.02) when considering both referrals and portfolio-based decisions. Higher NCI year 1 scores are also associated with overall promotion decision in year 1 (p = 0.01), medical knowledge competency in year 1 (p = 0.04), overall promotion decision in year 2 (p = 0.0002), and all three competencies in year 2 (medical knowledge, p = 0.0006; professionalism, p = 0.02; and communication skills, p = 0.004) when considering portfolio-based decisions only.

Associations among NCI years 1 and 2 and subsequent promotion decisions are presented in Table 3. Similarly, for overall and specific competencies, higher NCI year 1 and year 2 scores are associated with remediation in years 2 thru 5 and in year 2; NCI year 1 and year 2 scores are associated with concerns/remediation/fail.

Table 3 Association among noncompliance index and promotion decisions in years 1 and 2
Full size table

Discussion

Our data showed that the vast majority of our students displayed conscientious (compliant) behavior; nonetheless, the size of our program enables us to focus on those who may need further guidance. The data we gathered over a period of 8 years allowed us to develop strategies to help advise students. These strategies include remediation, physician advising, and noncompliance awareness. The NCI used in this study included missing course evaluations, shown to correlate with unprofessional behavior [3, 6], poor test performance [5], and delayed/incomplete knowledge assessments, CAPPs, and SAQs [17]. We hypothesized that medical students who did not complete these routine activities, as denoted by higher NCI scores, were more likely to require formal remediation and receive lower scores on USMLE Step 1.

A notable finding involved detecting a significant relationship between students’ NCI scores and USMLE Step 1 performance. For instance, in this study, each unit increase in NCI score, on average, resulted in a 1.6-point decrease (year 1) and a 1.3-point decrease (years 1 and 2) in a student’s USMLE Step 1 score, after adjusting for gender and MCAT performance. To our knowledge, this is the first study to link unconscientious student behavior to lower USMLE Step 1 performance. We recommend including USMLE Step 1 as a variable in future studies of conscientious behavior to illuminate underlying reasons for this relationship.

The use of a longitudinal design revealed that unconscientious behavior during year 1 of medical school predicted unfavorable student performance in year 2 for overall promotion to year 3 and adverse competency-specific decisions in three competency domains (medical knowledge, professionalism, and communication skills). Furthermore, combined year-1 and year-2 NCI scores were predictive of students being placed in formal remediation in year 2 and years 3 to 5 by the school’s promotion committee. Formal remediation at CCLCM requires that medical students develop and implement learning plans, with direct oversight of the promotion committee, until generating sufficient assessment evidence to document satisfactory performance [18]. Consequently, most CCLCM students would prefer to avoid the additional scrutiny and effort that remediation requires in order to meet performance expectations and graduate from medical school. These findings suggest NCI scores may be used as a coarse measure to identify and counsel at-risk medical students about the potential impact unconscientious behavior may have upon performance and career advancement, particularly as it has been shown that medical students may continue to demonstrate unprofessional, irresponsible behaviors decades after graduating from medical school [4].

We now track NCI scores for each student and informally communicate worrisome NCI patterns to students’ assigned advisors as a vehicle for advisors to initiate conversations with their students about possible stressors and time management strategies. These notifications to advisors occur rarely, as most CCLCM students consistently complete NCI tasks.

Our medical school’s emphasis on competency-based assessment provided an opportunity to explore the relationship of students’ conscientious behavior upon multiple performance domains. We anticipated that higher NCI scores would predict adverse, performance-based decisions for the medical knowledge and professionalism competencies and would not be associated with performance on the communication skills competency, which was not the case in year 2. We learned that the promotion committee systematically assigned a “pass with concern” performance decision if a student submitted a poorly constructed summative portfolio (communication competency), which may explain this unexpected observation of NCI scores predicting performance on the communication skills competency.

Finally, this study’s findings provided opportunities to discuss the importance of completing routine tasks with both students and faculty. Each year, we present NCI data to first-year students at a class meeting and discuss completion of routine tasks at advisor and promotion committee meetings to emphasize the impact conscientious behavior may have upon one’s professional career, regardless of gender. These discussions have helped communicate and reinforce our medical school’s commitment to professional, responsible behavior.

This study has several limitations. Our longitudinal, cohort-based design relied on routinely collected data and provided a coarse measure, at best, of conscientious behavior. Though other studies have linked the completion of course evaluations to performance lapses [3, 6], our NCI measure included SAQs and CAPPs, which are unique to our medical school program. While the NCI predicted USMLE Step 1 and competency-specific performance decisions in this analysis, we cannot discern if these patterns will generalize to future classes after communicating study findings to both students and faculty. Additionally, we combined some categories of overall promotion decisions to form a dichotomous variable (i.e., pass and concerns/remediation/fail) for statistical analyses. We recognize that “pass with concerns” and “pass with remediation” represent different performance levels; yet, we believe the collapsed category of concerns/remediation/fail retains the meaning of deficient student performance.

Conclusions

We observed that we can make much of the mundane! Students who consistently neglected to perform routine tasks and exhibited unconscientious behaviors were more likely to attain unfavorable medical school performance assessments and score lower on USMLE Step 1 exams. Previous findings have shown that unprofessional actions by students during medical school can lead to disciplinary actions up to and including professional license revocation when they become practicing physicians [4, 10]. While this study highlights the negative outcomes of noncompliance with mundane matters, we believe that our noncompliance index may be used to identify positive outcomes as well. For example, would students that score low—higher degree of compliance—on our noncompliance index prove more likely to become chief residents of their residency programs? Do these students make better physician leaders or seek out leadership responsibilities within their hospital systems? Do students with lower noncompliance index scores enjoy more career success such as promotions or chair positions? These are a few encouraging outcomes for possible areas of research. Given our findings, other medical schools may wish to explore whether similar, school-specific noncompliance indices, can identify potential at-risk students and examine its association with academic performance and behavior more broadly. Our findings may also generalize to other health professions and graduate medical education as most training programs routinely collect administrative compliance data that provide snapshots of learners’ completion of expected behaviors, such as meeting deadlines, honoring policies, and completing course evaluations. These readily available data can be compiled easily to create program-specific conscientiousness indices that may identify struggling learners in sufficient time to provide anticipatory guidance and recommend resources for improvement. Additionally, noncompliance metrics may provide educational programs with a feasible, systematic approach to encourage learners to reflect upon the impact unconscientious, irresponsible behaviors may have upon their performance and future careers. The potential uses of data collected from these indices are numerous, from career advising to the development of highly professional future physicians. We believe emphasizing conscientious behavior, at a programmatic level, helps foster and reinforce professional environments essential for learning; thus, we recommend exploring all options and utilizing all available tools—however mundane.