Keywords

1 Introduction

National medical competency examinations have been conducted to evaluate medical school education. The Indonesian Medical Doctor National Competency Examination (IMDNCE) is a national medical competency exit exam established in 2014 [1] The IMDNCE consists of multiple-choice questions using computer-based testing methods (MCQs-CBT) and an Objective Structured Clinical Examination (OSCE). This high-stakes OSCE is an essential tool for assessing clinical skill competency [1]. It is designed to evaluate students' history-taking skills, physical examination, communication, and professionalism. The OSCE has been used worldwide to teach and assess learners' competencies, particularly in healthcare [1, 2].

Formative OSCEs are assessments for learning, and it does not determine whether the students pass or fail. It is conducted to make the student familiar with the concept of OSCEs. They are unlike summative OSCEs, which are learning assessments that count toward a grade and formally assess clinical skills and knowledge required for graduation [3]. Formative OSCEs contribute positively to final summative examination performance [4]. During formative OSCEs, students are given the experience of interacting with standardized patients and their teachers at each station [5].

Although OSCE is well-established as an effective assessment tool for clinical competence, feedback after OSCE is vital in improving the student's clinical skills. This impact is not only for assessment but also can be used for learning purposes when it is accompanied by proper feedback. Feedback is an essential element of the educational process [6]. Feedback is a continual process between the teacher and student [7]. Good feedback practices may strengthen students' self-regulative ability [6]. Formative feedback gives information to the learner to improve their learning [8]. Formative feedback in OSCE to the learner can narrow the gap between actual and desired performance. Feedback is a fundamental learning tool in medical education, whereas excellent and effective feedback enhances student motivation and satisfaction [8].

Feedback forms can range from a simple judgment of correctness, identification of parts that could be improved, and also inviting ideas [9]. While formative feedback can be verbal or written, written feedback can take the form of comments, questions, corrections, and others to develop student understanding and provide a correction [9].

Due to the nature of the formative feedback, some students might see formative feedback as the only way to do better at summative assessment, thus not taking the formative assessment seriously without the thought that formative feedback is essential to help students reach the milestone of performance and narrowing the gap between actual and desired performances [10]. Although formative feedback is underutilized due to the complexity and minor variance, its benefit in highlighting the drawback of a curriculum or educational program is significant [10]. This study aimed to explore the outcome of simple feedback in national high-stakes OSCE by analyzing the passing status and the score of high-stakes OSCE (post-feedback; P2) compared to local formative OSCE (pre-feedback; P1) after being given the feedback in P1 OSCE.

2 Methods

This study was conducted with a quasi-experimental design on final-year medical doctor (MD) profession students of Atma Jaya Catholic University of Indonesia from 2017 until 2019 who undertook the P2 OSCE. Before the P2 OSCE was conducted, the P1 OSCE from the faculty was done to assess the readiness of the MD students. Both examinations were conducted with the same method. Each student was required to complete an entire cycle of 12 stations in a single session of both P1 and P2 OSCE. The topic of each station varied about a clinical case or procedure, and the examiner might not usually be an expert on the subject. The examiners were general practitioners with minimal qualifications of Master's degrees or specialists.

The examiner for each OSCE station had to follow the specific scoring sheet (actual mark) from the grading format checklist and observe the student's overall performance sheet (global rating) in the station for borderline regression purposes as a standard-setting procedure. The scoring sheet consists of three to eight criteria (history taking, physical examination, laboratory workup, differential and working diagnosis, pharmacotherapy plan, non-pharmacotherapy plan, communication, education, or professionalism) to assess specific clinical skills of each scenario. The global rating was scored as: (4) superior performance; (3) passing performance; (2) borderline performance; and (1) not-pass performance. Both P1 OSCE and P2 OSCE used the same grading format.

The borderline regression method is one of the latest methods to evaluate students' performances at each station by completing a checklist and a global rating scale. The checklist mark from examinees from each station is then regressed on the attributed global rating scores, providing a linear equation. The global score representing borderline performance (e.g., 2 on the global performance rating scale) is substituted to predict the pass-fail cut score for the checklist marks.

The formative written feedback was given in P1 OSCE regarding the positive and negative performances of the students during the examination and the suggestions for improvement for P2 OSCE. These written formative feedbacks were collected from each station and were sorted and put together based on the student's ID. It was given to each student individually approximately two weeks before the P2 OSCE, and the students were given an optional practice period until one week before the P2 OSCE. The feedback was meant to give the students the self-reflections needed before the P2 OSCE to improve their performances.

Statistical analysis was performed using STATA version 14.1 (STATA Corp., College Station, TX, USA). The differences in the P2 passing rate between each batch were compared using Fisher's exact probability test. The passing status frequency of the P1 and P2 OSCE results were compared using Fisher's exact probability test. The score of P1 and P2 OSCE was compared using the Wilcoxon signed-rank test. A p-value <0.05 was appraised as statistically significant.

3 Results

This study was conducted on 520 MD students from 10 batches of P2 OSCE. There were no significant differences between each batch of the P2 OSCE (Table 1).

Table 1 OSCE result of each batch
Full size table

There were 98 students who failed the P1 OSCE but passed the P2 OSCE, while only five students who failed P1 OSCE also failed the P2 OSCE. Only one student passed the P1 OSCE but failed the P2 OSCE (Table 2).

Table 2 P1 compared to P2 OSCE result
Full size table

There was a significantly higher median in the score of P2 OSCE (80.56 out of 100) than P1 OSCE (71.56 out of 100) (p-value < 0.001) (Fig. 1). The positive predictive value (PPV) of P1 OSCE as a predictor of passing P2 OSCE was 99.76%, and the negative predictive value (NPV) of P1 OSCE as a predictor of failing P2 OSCE was 4.85%.

Fig. 1
A box plot represents the P 1 and P 2 scores of each batch out of 100. It depicts the lower and upper quartiles range of O S C E P 1 and P 2 scores between 60 and 85, with extensions for outliers up to a score of 40. Values are approximated.

P1 and P2 scores of each batch. There were significant differences (p-value < 0.001, Wilcoxon signed-rank test) between the P1 OSCE score and the P2 OSCE score

Full size image

4 Discussions

The results of this study showed that the formative OSCE was beneficial for the student in their readiness to take the summative OSCE. Formative OSCE before the summative OSCE had a positive effect on increasing students' scores. Our findings may support the implementation of formative OSCEs to prepare students for high-stakes summative OSCEs. There were significant differences (p-value < 0.001, Wilcoxon signed-rank test) between the formative OSCE scores and high-stakes OSCE scores. Formative OSCE contributes positively to final summative examination performance [4]. During formative OSCE, students are given the experience of interacting with standardized patients and their teachers at each station [5].

This study analyzes MD students' improvement after written formative feedback from formative OSCE to high-stakes (summative national) OSCE. After giving written feedback about their weaknesses and strengths, the students had two weeks to prepare for the high-stakes OSCE. Formative feedback can enrich the learning process for students to improve their performances and give self-reflections apart from a pass/fail environment. It is essential to give time for them to encourage their self-assessment and reflection on strengths and areas that need improvement [11]. Formative feedback in OSCE to the learner can narrow the gap between actual and desired performance. Feedback is a fundamental learning tool in medical education, whereas an excellent and effective feedback enhances student motivation and satisfaction [8].

Feedback effectiveness is influenced by many factors, such as the characteristic of feedback sources (stringency/leniency), the feedback recipient, and the message itself [12, 13]. Rebel and colleagues [14] stated that removing pass/fail results can give students specific input on their skill performances to narrow the gap between actual and desired performances. The formative feedback from a clinically experienced physician could help the students to improve their skills in history taking, physical examination, communication, and especially clinical reasoning [2]. A study by Alkhateeb and colleagues [15] found that a single P1 OSCE does not increase and change the pass rate of P2 OSCE. This finding is discordant with our findings that showed there was an increase in the passing rate and the score of P2 compared to P1 OSCE. The difference in results might be because of the selection bias in the study by Alkhateeb because the participation of formative-OSCE was voluntary. Our study has a similar result to the study by Chisnall [16], which showed an increase in pass rate and scores. A previous study by Bandiera [17] also showed the positive effect of feedback of past examination performances on future performances. This finding might be due to the potential of formative feedback, which provides the student with a rich and meaningful learning experience [10].

The good PPV but poor NPV can imply that the students who passed P1 OSCE were more likely to pass the P2 OSCE, while those who failed the P1 OSCE were less likely to fail the P2 OSCE. This finding showed that P1 OSCE improves performance and prevents failure in P2 OSCE. The students also perceived the P1 OSCE as a positive and valuable activity that could help their preparation for P2 OSCE [16].

This study did not analyze the effectiveness of the feedback to the students and assess each feedback given to improve the performance in P2 OSCE [12, 13]. This is a limitation of this study, although many publications support the finding of this study. The design of this study also does not control whether the examiners on P1 and P2 OSCE were generalists or specialists.

As a feedback provider, the teacher has an essential role in encouraging learners' self-assessment and reflection, so the examiner's background should be in accordance with the clinical topic of the examination station [4]. The study conducted by Perron [2] found that the quality of feedback during formative OSCEs depends on the tutor's profile. Specialists reported less training in providing feedback than generalists, whereas generalists are more learner-centered and pay more attention to communication and professionalism during feedback.

Medical students may receive feedback in various clinical and non-clinical contexts where opportunities for feedback on clinical competencies are valuable [18]. Many studies reported face-to-face feedback from the examiner immediately after the OSCE with additional time allocation, audio or video recordings of the examiner providing generic or personalized feedback, and written OSCE feedback [19, 20]. Ngim [18] reported that medical students prefer written feedback compared to face-to-face feedback. In our study, the written feedback was unstructured or lacked guidelines to create high-quality feedback. It caused a wide variance between examiners [21], which becomes another limitation of this study. The excellent quality of feedback should be specific, balanced, and constructive and should describe the gap in student learning and observed behavioral actions in the exams [2].

The primary goal of formative low-stakes assessment is to support learners' progress. There are different perceptions about low-stakes assessment from the learner and teacher's points of view. Learners often do not appreciate the value of low-stakes assessments to guide their learning. If teachers do not fully understand the meaning and purpose of assessment, low-stakes assessments and their potential learning benefits are useless [22, 23] Van der Vleuten and Schuwirth proposed a programmatic assessment model to provide a holistic overview of students' competency development for formative feedback and summative decisions [24]. Multiple low-stakes assessments with constructive feedback can inform high-stakes performance decisions that have substantial consequences for learners [22]. Bok [24] found that the programmatic assessment is not easy to implement. It needs to train the students and supervisors to provide assessment and feedback. Hence, they both have the same perception about the function of low-stakes assessment and the feedback in the learning process and its contribution to high-stakes decisions. Enhancing feedback quality is also needed as the strategy for better implementation of programmatic assessment, such as using modern technology or scoring rubrics on the assessment form [24].

5 Conclusions

Assessment and feedback are essential components of medical education. Formative OSCE before the summative OSCE had a positive effect on increasing students' scores and passing status. Our study found that simple formative feedback in formative OSCE before summative OSCE could increase the percentage of passing rates and scores in summative OSCE. Structured written feedback is still needed to narrow variance feedback between examiners.