Introduction

Since attrition rates in North American medical schools are traditionally low and the number of applicants is increasing [1], emphasis on the admissions process to evaluate competent potential physicians is important [2]. Matriculation rates for medical school applicants in 2018–2019 were 40.9% [1]. Competitive admissions to medical school have increased in the last ten years, with over 40 schools adding multiple mini-interviews to go beyond traditional indicators of academic promise [3].

Medical School Admissions

Over the past 60 years, there has been a shift in the admissions process, from a statistical approach focused on Medical College Admission Test (MCAT) scores and grade point averages (GPA) to a holistic admissions process. Holistic review does not disregard or abandon the notion of statistical review. Rather, it incorporates the broad scope of the applicant’s life experiences, attributes, achievements, and how these characteristics may impact the medical school environment [4]. Additional selection methods and predictors of academic success during medical school have included but are not limited to consideration of applicants’ personal statements, letters of reference, situational judgment, and personality tests [5]. Patterson and colleagues, in a broad systematic review, found using references and personal statements were not as effective or as fair compared to other methods such as situational judgment tests. The use of personality tests for selection of medical students needs further investigation to determine its validity [5].

Communication and the ability to connect with patients have become increasingly emphasized as important skills for successful physicians [6, 7]. Therefore, it is imperative to consider these factors in the admissions process. As a result, multiple mini-interviews (MMI) are becoming more common as selection criteria, placing emphasis on non-cognitive abilities such as cultural competence and empathy [8,9,10,11,12,13,14]. Assessing non-cognitive abilities provides an opportunity to gain insight unavailable from the application and traditional interview process.

Multiple Mini-Interviews

MMI is an interview format consisting of multiple short stations, typically in a timed circuit, to obtain a score of each applicant’s soft skills [2, 8,9,10,11,12,13,14,15]. Medical schools using MMI for admission typically determine a unique set of evaluation components. Generally speaking, the main components target non-cognitive abilities, such as (a) cultural sensitivity, (b) empathy, (c) ethics, (d) honesty and integrity, (e) responsibility, (f) communication, (g) professionalism, (h) ability to handle stress, and (i) problem-solving skills [8,9,10,11,12,13,14]. Previous research has evaluated the reliability of MMI for medical school applicants and found that MMI was a more reliable way of assessing applicants [13, 16, 17]. Several studies found that compared to traditional interviews, MMI provides a more objective measure of applicants’ non-cognitive abilities [5, 18,19,20,21,22,23,24,25].

Defining Academic Success

Traditional success in medical school is defined as obtaining high scores on institutional and national certification exams [26, 27]. However, these success measures do not account for the non-cognitive skills measured by MMI which are an essential component for any interpersonal interaction. The disconnect between academic achievement and the non-cognitive skills needed for future success as a practicing physician may be a factor of burnout seen in medical students [28, 29]. Therefore, there is a need to expand the definition of success in medical schools to include not just academic achievement in the biomedical sciences but also the development and assessment of non-cognitive skills measured by MMI.

Purpose

The purpose of this research study is to understand the relationship among variables that contribute to acceptance into a southeastern medical school, as well as success in medical school performance. Three research objectives guide this study:

  1. 1.

    Examine the relationship among variables to create a predictive model of acceptance into medical school.

  2. 2.

    Examine the relationship between MMI scores and traditional academic success in medical school (defined as biomedical course scores).

  3. 3.

    Examine the relationship between MMI scores and clinical competency and academic accolades (defined as Objective Structured Clinical Examination (OSCE) scores and honors nomination).

Methods

Medical School Admissions Process

This study involves the admissions process of a medical school located in the southeastern USA. Admission to this medical school remains highly competitive with over 4000 applications received annually to fill a class of 100 students. As a result, less than 10% of applicants receive an invitation to interview.

Applicants interested in the medical school submit an application for admission through the American Medical College Application Service (AMCAS). The AMCAS applications are reviewed to determine which applicants will be asked to complete a supplemental application. While supplemental applications are completed by students with a broad range of GPA and MCAT scores, matriculating students to the school have an average GPA of 3.7 and MCAT score of 509. In the interview year of 2016–2017, the school received 4277 AMCAS applications and 93% were sent supplemental applications (n = 3979), with 2376 supplemental applications completed and returned. Once supplemental applications are received, Student Affairs and Admissions staff evaluate the completed applications, in order to determine who is invited for on-campus interviews, based upon several factors including academic performance, letters of recommendation, experience in healthcare settings and research, extracurricular activities such as volunteering and/or leadership experience, and distance traveled (i.e., overcoming adversity/hardships and unique personal experiences within the applicant’s journey to medical school). If invited to campus for an interview, they spend half of the day attending presentations by the medical school dean and associate deans of admissions and academics. Applicants complete tours of the school and have opportunities to interact with current medical school students. During the second half of the day, they participate in a series of interviews, traditional structured interviews and MMI. The two structured interviews are performed by various members of the admissions committee. They last approximately 30 min each and are used to assess cognitive (e.g., medically related experiences and academic record) and non-cognitive applicant qualities (e.g., maturity, motivation for a medical career, intellectual curiosity, interpersonal skills, etc.). These structured interviews are given a score by the interviewers and discussed within the admissions committee meeting.

MMI Development and Process

Each applicant also participates in four MMI: three with standardized patients (SP) in one of seven randomly selected developed scenarios, and one with a member of the admissions staff. These scenarios were developed by the Simulation Center’s staff over the course of three years. They do not require any medical knowledge and focus on addressing three general areas: communication, ethics, and professionalism. SPs are trained by Simulation Center staff. For the 2016–2017 interview season, 23 SPs were trained and conducted 1161 MMI for the school. Each SP performed approximately 50 MMI (mean 50.48, sd 38.82, median 44, range 1–140). Although the scenarios are randomly selected for the interview days, some of the scenarios require specific SP characteristics, and based on SP availability, some of the scenarios were performed less often (average number of performances of each scenario; mean 165.86, sd 47.59, median 192, range 85–196).

During the MMI process, applicants spend approximately forty minutes in MMI sessions—rotating through four stations. Each station is five minutes with three minutes of prep time and a two-minute break between stations. A total of sixteen applicants participate in each MMI session, amounting to approximately 2 hours total. Each station presents a highly structured, topically focused scenario that encourages applicants to assume the role of a character described in the prompt and engage in role play with an interviewer. At the conclusion of each scenario, interviewers rate the applicants on their skill and ability to address the conflict presented and articulate a response that reflects elements of emotional intelligence or psychological maturity. Figure 1 presents the process of admission and MMI described herein.

Fig. 1
figure 1

Process of medical school admissions and multiple mini-interviews with applicant instructions, descriptive prompts, and interview cycle options included

Full size image

Each portion of the applicant’s application and performance during the interview day are discussed during the committee meeting in an effort to determine a complete and rich picture of the applicant to determine matriculation readiness and fit into the environment of the school. Admissions committee members are provided a voting rubric for each applicant discussed during the admissions committee meeting. The voting rubric is divided into categories: (a) extracurricular activities, (b) commitment to medicine (characterized by clinical and/or shadowing experiences), (c) letters of recommendation, (d) individual interview, and (e) distance traveled. GPA, MCAT, and MMI scores were reviewed for each applicant in the context of the applicant discussion. At the conclusion of the interview day, the admissions office collected all scores from committee members to generate an average admissions committee score.

Pre-clinical Curriculum and Assessment

Upon matriculation into the medical school, students begin immediately learning clinical skills through emergency medical technician (EMT) training and certification. During the first two years of study, students continue to serve as community EMTs by completing one 12-hour shift each month. Upon completion of the EMT certification, students begin their study of foundational principles of medicine. The first-year curriculum focuses primarily on the normal human body within five biomedical science modules (Foundations of Medicine, Structure & Function 1 and 2, Neuroscience, and Defenses and Responses) and a year-long clinical module (Integrated Practice of Medicine) that focuses on clinical skills and reasoning. In the second year, students follow an organ-based approach focused more on disease processes in seven biomedical science modules (1. Biomedical Principles of Disease and Therapy; 2. Hematology/Oncology; 3. Mind, Brain, and Behavior; 4. Cardiovascular/Pulmonary/Renal; 5. GI/Hepatic; 6. Endocrine and Reproductive; 7. Musculoskeletal/Dermatology/Rheumatology) and a second year of the Integrated Practice of Medicine. Content related to behavioral, social, and population health, as well as lifestyle medicine, supports all four years of the undergraduate medical curriculum. Evaluation of progress through the curriculum in the biomedical sciences modules is provided through both formative and summative assessments. Generally, summative assessments are computerized multiple-choice tests with questions that mirror the National Board of Medical Examiners Step 1 exam. The number of summative assessments in each module depends on the calendar time spent in the module. Each summative exam covers four weeks of class material or less; therefore, if a module is longer then multiple summative assessments are given. Additionally, some modules in the first year also incorporate lab practical exams. Final grades in a module are the average of the summative assessments offered. For the Integrated Practice of Medicine course, the major summative assessments are Objective Structured Clinical Examinations (OSCEs). In the first year, OSCEs are performed at the end of the academic year in May. In the second year, OSCEs occur twice; once in December and once in March.

All courses are pass/fail; however, numerical grades are kept determining class ranks. In the Integrated Practice of Medicine course, students may also pass with honors. To pass with honors, students must have a score that is at the mean or above on the OSCE(s), be “satisfactory” on other assessments, and be nominated by at least one of their small-group faculty members. All nominations are reviewed by the course directors to determine whether a student will receive the honors notation.

Study Design

This portion of a longitudinal study followed one cohort of students who participated in MMI at this medical school during the 2016–2017 application cycle through their initial two pre-clinical years.

Ethical Approval

This study was approved by the Institutional Review Board of the university (Pro00069266).

Participants

Applicants who were offered an interview at the medical school and participated in the MMI process were included in this study. Individuals who were missing admissions data were excluded.

Variables

This study explored several variables assessed in the admissions process, including MCAT score, GPA, extracurricular activities, letters of recommendation, commitment to medicine, distance traveled, interview, and MMI score. These variables were scored and averaged to create an overall admissions committee score (possible score of 100), which guided the admissions committee in its decision-making process for each applicant.

The MMI score is presented as a continuous variable from 1 to 5 representing the average of applicant scores from the four stations. Traditional academic success, clinical competency, and academic accolades were also outcomes of interest. Traditional academic success was defined as module exam grades and final module grades. These grades are presented as categorical data obtained from the medical school. Academic accolades is a binary variable defined as obtaining honors nomination. OSCE score, which evaluates students on criteria ranging from the environment created in the patient room to the presentation of the care plan, is presented as a continuous variable and total score ranging from 0 to 532 possible points.

Statistical Analysis

Modeling Medical School Acceptance

Using a multiple logistic regression analysis to explain the association between the aforementioned admissions covariates and acceptance, the full model contained age, commitment, extracurricular activities, GPA, interview, distance traveled, letters of recommendation, MCAT, MMI, and sex (note that age and sex were included, though these were not factors defined by the admissions committee for inclusion in acceptance decisions). Backward elimination variable selection was performed, resulting in a final model containing only statistically significant predictors of acceptance.

Modeling Academic Success

The hypothesis of interest is whether the MMI score is associated with the different grades across modules. While the MMI score (predictor) is continuous, grades (outcomes) are categorical and potentially correlated, with categories in the ordered set of assigned grades {A, B+, B, C+, C, F}. Given the natural ordering of the categorical outcomes and their potentially correlated nature across outcomes, the approach proposed in this manuscript is a multivariate multinomial ordered probit model to assess the joint relationships between sets of outcomes (letter grades) and the predictor (MMI). Analyses were performed for each of the following sets of outcomes: (1) only exams across all modules; (2) only final grades across all modules; (3) exams and final grades jointly for each module; and (4) univariate analyses for each of the outcomes.

In general notation, where Y represents a set of outcome variables v = 1, .., V for each individual i = 1, .., 100, the model takes the following functional form:

$$ {\displaystyle \begin{array}{c}{Y}_{i,v}={G}_{i,v}\Longleftrightarrow {\theta}_{i, Gi,v-1}<{\alpha}_v+{\beta}_v\ast {\mathrm{MMI}}_i+{\eta}_{i,v}<{\theta}_{i, Gi,v}\\ {}{G}_{i,v}\in \left\{A,B+,B,C+,C,F\right\}\end{array}} $$

where Gi,v is the observed letter grade corresponding to individual i for outcome v, βv is the regression coefficient representing the association between MMI score and outcome v, and αv is the intercept. The latent thresholds, θ, which map MMI scores to outcome letter grades, are assumed to be outcome-dependent and, for identifiability, set to zero for the boundary between the first two letter grades in increasing order (i.e., between F and C for this study), and the error term, η, is assumed to be multivariate normal with common (unrestricted) covariance matrices across outcomes. Positive values of v reflect a positive association between MMI score and letter grades for outcome v.

Modeling Clinical Competency and Academic Accolades

Simple linear regression models were used to assess the relationship between MMI score and OSCE scores for each of the three periods when data was collected, and simple logistic regression models were used to assess the relationship between MMI score and honors nomination for the two periods available, once at the end of each year.

Results

Sample Characteristics

A total of 387 individuals participated in the interview process with nearly equal numbers of males (50.1%, n = 194) and females (49.9%, n = 193). The average age of all interviewees was 22.60 years old (sd 2.237). Among the 387 individuals interviewed, 17 were removed from the analysis for incomplete admissions data (i.e., missing MCAT score). Approximately 46% (n = 171) of those interviewed (with complete data) were accepted, and 100 of them matriculated. For those who were admitted to the school, MMI scores ranged from 2.86 to 4.96 (IQR = 3.92–4.36), with a mean score of 4.12 and standard deviation of 0.36.

MMI and Medical School Acceptance

Figure 2 portrays a heatmap of observed correlations among covariates with the exception of sex, which is binary, for the resulting sample of 370 individuals interviewed who had complete admissions data. This diagram highlights interesting features of the data. For example, some covariates are highly significantly strongly positively correlated, such as commitment and interview (r = 0.95); letters of recommendation and interview (r = 0.93); letters of recommendation and commitment (r = 0.91); MMI and commitment (r = 0.39); MMI and interview (r = 0.38); MMI and letters of recommendation (r = 0.38); MMI and extracurricular activities (r = 0.32); and MMI and distance traveled (r = 0.21). Conversely, other factors demonstrate significant negative correlations, including age and GPA (r = − 0.35); MCAT and distance traveled (r = − 0.29); distance traveled and GPA (r = − 0.16); and MCAT and interview (r = − 0.12). MMI was not significantly correlated with GPA (r = − 0.07, p = 0.1744); MCAT (r = − 0.05, p = 0.3516); and age (r = 0.04, p = 0.4359).

Fig. 2
figure 2

Heatmap of correlations among continuous covariates considered for the analysis (a subset of which form the final model explaining acceptance), with the exception of sex (binary). Lighter colors indicate higher correlations, as represented in the color key, which also overlaps a histogram of observations by range of correlation values

Full size image

Table 1 contains the parameter estimates, standard errors, and p values for each of the covariates in the final model after backward elimination variable selection was implemented. The resulting model is

$$ \mathrm{logit}(p)=-60.22+1.84\ast \mathrm{female}+4.49\ast \mathrm{MMI}\ \mathrm{score}+1.09\ast \mathrm{MCAT}\ \mathrm{points}+0.25\ast \mathrm{GPA}\ \mathrm{points}+1.93\ast \mathrm{commitment}+2.64\ast \mathrm{letters}+1.37\ast \mathrm{interview}+0.81\ast \mathrm{distance}\ \mathrm{traveled} $$

where p is the probability of acceptance.

Table 1 Parameter estimates, standard errors, and corresponding p values for the final multiple logistic regression model for acceptance, resulting from a backward elimination variable selection procedure
Full size table

For example, an individual who is male with MMI score = 3.53, MCAT points = 15, GPA score = 20, commitment points = 4.33, letters of recommendation score = 4.0, interview score = 3.75, and distance traveled score = 0 has logit(p) = 0.8988, or p = exp(0.8988)/(1 + exp(0.8988)) = 0.7107. This applicant profile has a 71.07% chance of acceptance conditional on his observed factors.

The only covariate in the final model that is not introduced in the admissions committee-defined decision-making process is sex (female = 1). Table 2 contains the means, standard deviations, and p values for the mean differences for males vs. females. Among the covariates in the final model, females are significantly more likely to have a better MMI score, commitment to medicine, letters of recommendation, and interview compared to males, though females are significantly less likely to have a higher MCAT score compared to males. This “combined” interaction effect can explain the gender significance within our results.

Table 2 Means, standard deviations, and p values (for the mean differences) by sex (n = 188 females (F) and n = 182 males (M)) across covariates among interviewees
Full size table

Figure 3 contains density plots of the fitted model probabilities for each of the two acceptance clusters (those accepted and those not accepted). The left panel in Fig. 3 portrays the expected probability of acceptance for those who were accepted (with a large cluster of probabilities of acceptance around 1), while the right panel shows the expected probability of acceptance for those who were not accepted (with a large cluster of probabilities of acceptance around 0). This figure shows a strong fit associated with a great predictive power of the final model.

Fig. 3
figure 3

Density plots for the fitted probabilities from the final model for those accepted (left panel) and those not accepted (right panel)

Full size image

Academic Success

Table 3 contains a descriptive summary of the observed MMI scores across letter grades of the outcome variables for the biomedical science grades from year 1.

Table 3 MMI scores, standard deviations, and counts (n) across outcome variables (exams and final modules) and letter grades within each outcome variable
Full size table

The estimated coefficients for MMI score, βv, across each of the four aforementioned analyses within the “Methods” section are reported in Table 4. While there are small negative associations between MMI score and most of the outcomes, they are not statistically significant, as reflected in Table 4. However, most of the estimated coefficients, both at the univariate and multivariate levels, show negative associations. Since the predictor, MMI score, is common to all analyses, and the outcomes are all measured using the same letter grade scale, the magnitudes of the coefficients, βv, are comparable within and across analyses.

Table 4 Coefficient estimates (p values) for associations between MMI scores and the corresponding outcome (rows) using multinomial ordered probit models with different subsets of outcomes (columns). Within each column, outcomes that are modeled jointly are highlighted in the same shade
Full size table

Clinical Competence and Academic Accolades

None of the regressions resulted in an MMI score being statistically significant at the 5% level. Therefore, average MMI does not appear to be associated, at any of the time periods, with either OSCE scores or honors nominations; nor are there any significant differences among the coefficients over time. Table 5 shows parameter estimates, corresponding standard errors, and p values for each of the regressions.

Table 5 Parameter estimates (for covariate average MMI), as well as corresponding standard errors and p values, for each of the regressions (linear [total OSCE score responses] and logistic [honors nomination responses]) across time periods
Full size table

Table 6 contains the MMI scores and standard deviations, as well as corresponding p values, from t tests for differences in MMI means by honors nominations in both academic years. All analyses confirm that the MMI score is not significant to explain the variability in OSCE scores nor honors nomination status for any of the time periods in the sample.

Table 6 Mean MMI scores, standard deviations, and p values for the differences in means for both time periods of honors nominations
Full size table

Discussion

This study has indicated our admissions process is influenced by MMI, letters of recommendation, commitment to medicine, traditional interview, extracurricular activities, GPA, MCAT, and sex. We also found females were more likely to score higher in MMI compared to males, leading to a higher likelihood of acceptance into medical school [30]. Further, this study suggests that MMI is highly significantly positively correlated to less cognitive admissions criteria including commitment to medicine, traditional interview, letters of recommendation, extracurricular activities, and distance traveled, while MMI is weakly negatively correlated (though non-significant) with more cognitive admissions criteria including GPA and MCAT score. A prior study showed a significant negative correlation between emotional intelligence and undergraduate GPA when assessing admissions [31], which aligns with some of our findings, though the negative correlations in our study are non-significant. Though, prior studies also show no significant correlations between emotional intelligence and admissions selection criteria [26]. With mixed results, further research regarding MMI, other admissions criteria, and their influence on medical school acceptance is needed.

Traditional Academic Success

Prior studies reported mixed results on associations of non-cognitive abilities with traditional academic success among matriculated students, indicating low linear correlations, as well as negative, positive, and no associations [26, 31, 32]. Our study found negative associations, though non-significant with the exception of one module final grade, between MMI score and biomedical science exam and module final grades. Results may reflect potential associations not sufficiently uncovered due to a low sample size. A larger sample may be needed to draw sufficient conclusions on the associations between MMI and traditional academic success.

Clinical Competency and Academic Accolades

While we did not find that MMI predicted higher scores on clinical competency and academic accolades, further discussion revealed that these measures may not be fully aligned. OSCE scores incorporate some of the non-cognitive skills assessed during the MMI process; however, additional constructs may be adding noise to the analysis. Without examination of similar constructs of variables between MMI and OSCE, we cannot draw conclusions of the relationship between the two. Positive correlations have been found between MMI interpersonal scores and communication rubrics on OSCE for residents [33]. Also, we note the lack of standardization awarding honors nominations to students. Because these designations do not have a specific process, receipt of honors nomination is subjective.

Limitations

We noted limitations in this research study. In understanding traditional academic success, there is not a predefined or enforced policy on attendance in didactic courses at the medical school. Scores may be affected by inconsistent class attendance. This study did not account for or address students who may have received accommodations during their courses or within OSCEs. Knowing this information can encourage researchers to normalize their knowledge of the scores collected for each student. Further, some variables used in this study to predict acceptance into medical school were notably absent. Reporting on race, ethnicity, socioeconomic status, and sexual and gender identity minority status are self-identifiable categories in the AMCAS. Our research team did not have access to these variables during data analysis. Future analyses may lead to investigation of whether these factors are associated with differences in MMI scores [10]. Including race/ethnicity and socioeconomic status would provide further insights as other studies have found that MMI do not disproportionately negatively impact those who are underrepresented in medicine [10, 34, 35]. However, these other studies were not conducted in the South and therefore it is important to determine if there are regional differences. Interestingly, no studies to date could be identified investigating MMI outcomes for sexual and gender identity minorities, which gives rise to interesting questions in the future. Finally, neither validation of the MMI scenarios nor inter-rater reliability studies has been conducted for all of the SPs within the Simulation Center; therefore, the results should be viewed with caution. However, the Simulation Center has conducted quality improvement investigation to determine inter-rater reliability of some of the SPs who are part of the MMI, but not all. Internal reports indicate low variability between SPs in their scoring. These results have not been published and have only been used for training purposes.

Lessons Learned

After the initial implementation of MMI in the admissions process, several changes have been made. The evaluation rubric was revised to improve the ease of grading applicants. This checklist-style rubric will disallow comparison between the first and subsequent cohorts; however, it may provide better insight into future cohorts. The revised rubric grants credit for how the applicant performs, identifies key problem areas, and notates areas of excellence. Additionally, this rubric has been implemented into OSCE grading, which will allow for standardized assessments of students’ non-cognitive abilities throughout medical school.

Future Implications

As we continue to follow this cohort through medical school, there are future data points that should be explored. We would like to follow these students through their third and fourth years, examining their academic course scores, OSCE, and SHELF exam scores. We would also like to examine STEP 1 and STEP 2 scores, as measures for residency placement and residency choice.

One final important factor not considered is healthcare pipeline program participants, who comprise 10% of students who matriculate to this medical school. Pipeline programs traditionally provide high school and/or college students with access to resources preparing them for careers in medicine. A future study could focus on this group of applicants who have received training and practice on the interview process prior to applying to medical school. Though beyond the scope of this manuscript, the logistic regression used for modeling medical school acceptance could also be used to identify outliers, which would represent applicants with acceptance decisions that significantly differ from those implied by the model’s fitted values based on their covariate set. This information could serve committees to automatically monitor and identify individuals who may need a second look/review of their application materials and enhance the institution’s intra- and/or inter-year coherency of their decision-making process. It could also serve to identify potential common reasons driving such differences.

MMI in a holistic admissions process may help to identify those students with non-cognitive skills needed to become successful physicians. Focusing only on assessing success in a traditional academic manner overlooks the applicant whose success incorporates non-cognitive skills and abilities. Furthermore, if medical schools do not consistently assess and evaluate these components from MMI (e.g., cultural sensitivity, empathy, decision-making) then curriculum effectiveness and student non-cognitive skill development cannot be determined. Integrating MMI into a different school of thought redefines successful applicants and students, opening up opportunities for those who might be traditionally passed over for admission.