Keywords

1 Introduction

The Structured Oral Case Analysis (SOCA) test is an oral assessment method in which the examinee must analyze a case to be able to answer related questions and the answers are recorded based on a structured marking grid [1]. The SOCA test can assess problem-solving skills, critical thinking, clinical reasoning and decision-making, professionalism and ethics, and academic communication skills. Like other oral tests, the SOCA test is considered to have a good educational impact, namely facilitating student learning (assessment for learning) [1,2,3,4,5].

Most of the time, the SOCA test is regarded as the most challenging assessment in the faculty of medicine. Relevantly, the score is frequently less satisfactory [4]. In evaluating the unsatisfactory SOCA test scores, we can use the presage-process–product (3P) model of teaching and learning developed by Biggs [6]. According to the model, the presage factors that exist in students and their learning environment interact with each other to produce or influence the process factor, namely the student learning process, which in turn will affect the product factor, namely student learning outcomes. Thus, it can be understood that the unsatisfactory scores on the SOCA test (product factor) can be caused by problems in the learning process (process factor) or students' characteristics and learning context factors (presage factor).

According to Thorndike's Law of Readiness [7], when a student is faced with a stimulus in the form of a learning environment that demands independence and proactivity, student readiness is necessary to respond to it to be able to perform well during the learning process and gain a satisfactory result. Therefore, one of the presage factors that can affect the SOCA test score is the level of student readiness in carrying out the self-directed learning (SDL) process, known as Self-Directed Learning Readiness (SDLR). Knowles in 1975 proposed that SDL itself is a process of individuals taking the initiative to analyze learning needs, formulate learning goals, identify the resources needed for learning, select and implement appropriate learning strategies and evaluate their learning outcomes with or without the help of others [8]. The Self-Directed Learning Readiness (SDLR) in Nursing Education (SDLRNE) [9] is the most frequently used instrument to assess SDLR and has three constructs, namely self-management, desire to learn, and self-control [10].

Examining the correlation between students’ SDLR scores and assessment scores is necessary. After the positive correlation is confirmed, the institution can take steps to help students with low SDLR scores. However, if the SDLR score does not correlate with the SOCA test score or if the students’ SDLR scores are already high, then the institution needs to test other presage factors and process factors. The competencies tested through the SOCA test are trained mainly through the problem-based learning (PBL) group discussion method during the learning process. Many studies on medical students show that PBL group discussions encourage students to apply SDL strategies through cognitive, metacognitive, and resource management strategies [11, 12]. Thus, problems in conducting PBL group discussions might explain why the high SDLR score as a presage factor is not followed by the suitable learning process that supports the achievement of good academic achievement.

1.1 Methods

The Faculty of Medicine in Indonesia, including the Faculty of Medicine, Universitas Jenderal Soedirman (FoM UJS) has implemented a competency-based curriculum. The curriculum is structured as a series of 3–6 weeks length blocks. The Introduction to The Human Body Systems Block (IHBS Block) is the third block of semester 1, which lasts for 3 weeks. In this block, students learn an overview of the human body, from chemical to organismal levels. The learning methods used were lectures, PBL group discussions, and hands-on laboratory practices. The summative assessment methods used were multiple choice questions (MCQs), Anatomy and Histology laboratory test, and SOCA tests. Students were excited because this was the first block for them to learn basic medical science materials, meaning that it had greater relevance to a career in medicine as compared to the previous two blocks. However, the test results were unsatisfactory. We hypothesized that the students’ SDLR scores have influenced it. Thus, we aimed to study the correlation between SDLR scores with SOCA test scores.

Using total sampling, this cross-sectional study was conducted on 126 participants of IHBS in the 2021/2022 academic year. The Indonesian language SDLR questionnaire [13], an adaptation of the SDLRNE questionnaire developed by Fisher et al. [9] was used in this study. The questionnaire consists of 3 constructs spread over 36 valid and reliable statements related to SDL characteristics. The self-management construct consists of 13 statements, the desire to learn to the construct consists of 10 statements, and the self-control construct consists of 13 statements. For each of the statements, respondents were asked to choose their responses among a 5-point Likert scale, ranging from 1 (absolutely not agree) to 5 (absolutely agree). This was the most appropriate instrument since in testing the validity and reliability, the developer used the context of the medical faculty. In addition, the researcher did not find another SDLR measurement instrument in Indonesian in the literature. The SOCA test questions were reviewed by medical education experts and related lecturers and were declared valid. The SOCA test score before the remedial test was obtained from the block’s administrator.

Univariate analysis for the SDLR scores was conducted to know the SDLR score’s mean for overall statements and each construct, the mean to the maximum score in each construct to know the percentage toward the desired result, as well as the score’s mean in each statement to identify the statements with the lowest and with the highest score in each construct. Univariate analysis for the SOCA test scores to know its distribution was based on the following grade categories: A (≥80.00); B (66.00–79.99); C (56.00–65.99); D (46.00–55.99); E (<46.00).

The bivariate analysis presented a crosstabulation between SDLR scores that were categorized as low (36–83), moderate (84–131), and high (132–180) and SOCA test scores that were categorized based on grade. The correlation between the SDLR score and the SOCA test score was analyzed using the Pearson correlation test with a significance level of <0.05.

2 Results

2.1 Univariate Analysis

Of the 126 IHBS Block participants, 122 people filled out the questionnaire (96.82% response rate). Most of the respondents (69.7%) were women. Both SOCA test scores and SDLR scores were normally distributed. The mean for the SDLR score was 133.57 (95–168). Based on Nyambe, Harsono, and Rahayu in 2016, it was regarded as a high SDL level (≥132). The distribution of SDLR scores within each construct is shown in Table 1. When we compared the mean for the SDLR score in each construct to the maximum score for that construct, the biggest percentage was in the construct of desire to learn (77.2%), followed by the construct of self-control (76.66%), and the least was in the construct of self-management (45.10%). Table 1 shows the statements with the lowest and the highest average score in each construct.

Table 1 Distribution of the SDLR score within each construct
Full size table

The mean for the SOCA test score was 42.79 (11.00–79.00). Most of the students (61.5%) got an E grade. Students who passed (got grades B and C) were 23.8%, while 76.2% did not pass (got grades D, C, and E). No student achieved a grade of A.

2.2 Bivariate Analysis

The cross-tabulation between the SDLR and SOCA test scores is seen in Table 2. Table 2 shows that of all students, most (55%) had a high SDLR score, 45% had a moderate SDLR score and none had a low SDLR score. We can also see that most students with high and moderate SDLR levels had SOCA test scores of grade E.

Table 2 Crosstab between SDLR and SOCA test scores
Full size table

The data normality test and data variance test on SDLR scores and SOCA test scores met the requirements for parametric tests. The Pearson correlation test for the correlation between SDLR and SOCA test scores got a correlation coefficient (r) = 0.037, which is a very weak correlation strength, with a p-value of 0.686. Thus, there was no significant correlation between SDLR and SOCA test scores.

3 Discussions

3.1 Most Students Had a High SDLR Level

The finding that most students had a high SDLR level and no student had a low SDLR level is different from the literature review regarding the SDLR level of medical students in Asia, which reported that the SDLR score at the beginning of university entry was still low but then increased when entering the third level [13]. The difference is probably due to the differences in the learning environment characteristics and learning methods used. The previous study used the SDLRNE questionnaire developed by Fisher, King, and Tague in 2001, which categorized SDLR scores as high (>150) and low (<150). However, we could not automatically interpret that our results were equal to the low SDLR level due to the difference in the two instruments used. Additionally, we are aware that the mean score of 133.57 was only slightly above the lower border of the high SDLR category, which is 132–180.

Dolmans et al. [14] found that SDLR was influenced by the amount of experience in implementing student-centered learning. This is supported by Fisher et al. [9] who explained for a person to be self-directed in a specific context, they must first have the relevant knowledge to a certain extent. In this study, most of the respondents had high SDLR levels. This may indicate that most of the 2021 batch of new students were accustomed to various student-centered learning methods. One of the contributing factors might be the implementation of the 2013 curriculum (K-13) in elementary to high school. The 2013 curriculum requires students to be more active, creative, and innovative in solving problems at school [15]. However, further studies are needed to test the assumption regarding the effect of the implementation of the K-13 on Indonesian students’ SDLR.

3.2 There was No Correlation Between the SDLR Score and the SOCA Test Score

The positive correlation between SDLR and learning achievement is supported by Demak and Pasambo [16] who found a weak positive correlation and by Gayathridayawarsi et al. [17] in 2019 who found a moderate positive correlation. However, we found no significant correlation between the SDLR score and the SOCA test score.

Based on the 3P model of teaching and learning by Biggs et al. [18], SDLR as one of the presage factors generally has a direct effect on the process factor, and thus indirectly affects the test score, which is one of the product factors. This indirect relationship is supported by Zulharman et al. [19] who found that the effective contribution of SDLR in improving learning achievement was only 7.6%, indicating the average learning achievement of a group of students who had a high SDLR would be higher than those with moderate SDLR, although the difference was not statistically significant. Therefore, as mentioned above, if we found that the SDLR score does not correlate with the SOCA test score or if the students’ SDLR scores are already high, then the institution needs to examine other presage factors. It is also important to look into the details of the SDLR statements, and whether there are some aspects in each SDLR construct with a low score that might affect the process factor.

3.3 The Problem with the Presage Factor

Another presage factor that moderates the relationship between SDLR and SOCA test scores is academic self-efficacy. Students with low academic self-efficacy will not automatically apply self-directed learning skills in their learning process, even though they have a high SDLR level [20]. Based on the average score on the construct of self-control, most students could realize their limitations and tended to lack confidence in their abilities. They were able to identify their limitations because, in the first block, namely Personality Development and Professionalism Block, students were taught self-potential analysis and self-evaluation. Based on the academic self-efficacy theory, e.g. students’ personal belief in their capacity to achieve educational tasks to the expected level, students’ lack of self-confidence could be caused by several reasons. They could have experienced failure before or have seen other people fail even though they had abilities and had tried hard. There might be a lack of persuasion or positive suggestions, advice, and guidance from people around. Being stressed, fearful, and anxious without abilities to manage them could reduce their self-efficacy [20]. FoM UJS might relate to those reasons. Similar to other oral tests, the SOCA test causes anxiety more than written tests. This was because of some factors, such as the extent of support from parents or people surrounding them, students’ level of knowledge regarding the principles of the SOCA test as an assessment method, students’ level of preparation [21], failure or success experiences in the previous SOCA test, and the emotional status during SOCA test preparation [22]. In this study, the students had undergone the first 3 blocks or completed about the first 3 months of their journey as new medical students. They might still struggle to adapt to their new learning environments at the campus as well as at home, especially if they are from other towns, and now living apart from their parents and family. They might still search for the most suitable way to conduct learning methods and prepare for assessment methods that are different from those in high school. Especially knowing that the IHBS block is the first block in the medical faculty curriculum to have the SOCA test as an oral assessment, students have no previous experience in conducting SOCA tests. Those adaptation issues might influence their academic self-efficacy.

Another adaptation issue is time management. Based on the average value for each question in the self-management construct, it can be concluded that most students could determine work priorities, but were less able to manage their time well. Learning how to prioritize activities was taught as part of a student orientation program for the new students and in the previous block, namely, the Learning How to Learn Block. However, this block was much busier than the previous two blocks. They might experience difficulties in managing time for finishing many assignments, not only those related to the laboratory practicums but also the lengthy orientation program organized by their seniors. They might need longer time in fulfilling the assignments because of difficulties in understanding the content materials, problems related to the availability of learning resources, and lack of opportunities for clarifications that support understanding. A fully online learning environment nature could worsen the case, as described by Curelaru et al. [23] who found some disadvantages of online learning as perceived by university students during the COVID-19 pandemic. Those disadvantages are health and psychosocial problems (e.g., stress, anxiety, decreased motivation, isolation/loneliness, and apathy) and learning process problems (e.g., misunderstandings, a lack of feedback, additional academical requirements, a lack of challenge, and disengagement).

The time management problem might hinder the students to adapt learning habits that were suitable to their learning style. In other words, students might use unsuitable learning habits just because everybody used them, and then this may affect their SOCA test scores. As argued by Radha and Muthukumar [24], the suitability of learning styles with learning habits will improve the assessment of learning outcomes. Further studies are needed to study the students’ learning style as part of their presage factor and study habits as part of their process factor, as well as the concordance between them.

Obstacles in the aforementioned adaptation issues might lower students’ academic self-efficacy. Therefore, institutions could have supported new students with specific instruction concerning academic and life skills in managing those problems. This could be done during the campus orientation program right after students’ enrollment. The importance of a well-organized campus orientation program in increasing students’ SDLR is following Thorndike's Law of Readiness [7] that explained when a student is faced with a stimulus in a form of a learning environment that demands independence and proactivity, student readiness is necessary to respond to it to be able to perform well during the learning process and gain a satisfactory result.

Another presage factor in the Biggs 3P model is the learning context factor. Leatemia et al. [25] found that the curriculum system, lecturer experience, student background, and cultural factors are factors that can affect SDL. In addition, presenting SOCA test answers requires good academic communication skills, since it is one of the skills assessed in the SOCA test. Students equipped with good public speaking as their prior skills will have an advantage. However, even though a student was not talented in public speaking or presenting, he/she could train him/herself through group discussion activities.

3.4 The Problem with the Process Factor

Problems in conducting PBL group discussions might explain why the high SDLR score as a presage factor is not followed by the suitable learning process that supports the achievement of good academic achievement.

We found that in the desire to learn construct, most students were less able to evaluate new ideas critically. Students could learn from their mistakes, probably partly because they have learned about reflective learning in the previous block, which was the Learning How to Learn Block, and were encouraged to apply it at the end of each PBL session. Students were asked to reflect on the PBL process that has been going on by analyzing the process that went well and what needs to be improved for the next PBL. The students tended not to be able to think critically about new ideas or knowledge even though critical thinking course material was taught in the Learning How to Learn Block. This is likely because this capability needs repeated practice and takes a longer time to be internalized. It could also be possible that critical thinking processes have not been practiced as part of their study habits because of poor time management or inappropriate learning management as mentioned earlier. The’ time management problem could also hinder students to apply a deep learning approach, thus applying superficial learning by pursuing assignment submission without implementing critical thinking.

The unsatisfactory SOCA test score indicates a lack of students’ ability to think critically. Solving SOCA test questions requires critical thinking skills in analyzing a case. This skill could be honed, especially through PBL discussions. Students could also practice answering comprehension or clinical application questions during individual learning. The importance of critical thinking practice is highlighted by Karatas and Barbay [26] who found that self-directed learning levels can be predicted by critical thinking disposition, academic achievement, and general self-efficacy in order of priority. Further research using qualitative methods is needed to explore students’ learning strategies in conducting PBL group discussions as part of strategies in preparing for the SOCA test.

3.5 Implications of the Study

This study has several implications. First, the institution needs to examine other students’ presage factors, such as students’ background and cultural factors, new medical students’ level of knowledge at the time of enrolment, and students’ preference of learning style. Second, the institutional presage factors, such as the curriculum system, lecturer experience, and the SOCA administration need to be evaluated [25]. Intervening through modifying the institutional presage factors would be more manageable. We have ensured that the SOCA questions and answer rubrics were valid in terms of content and face validity. This consistency would be good to be maintained for every SOCA test. Third, the institution should consider having more than one SOCA examiner for each examinee and ensure that they reach an agreement on the scoring. Fourth, to increase the educational impact of the SOCA test as an oral test [1,2,3,4,5], we suggest that the faculty, especially persons in charge of the Block deliver a debriefing session, both for the students and the examiners. Although everyone involved in the SOCA test could put their thoughts in the Block evaluation form, direct and timely feedback should be better. Specific feedback should inform the students: what has been done well and what should be anticipated or avoided as well as how to do it to achieve good results in the next SOCA tests. Fifth, the institution should facilitate students’ practice in time management as new medical students to increase their self-efficacy. This remedial approach can be done through training and mentoring by faculties or senior students. The tutors should also give examples, opportunities, and supports for critical thinking and presenting skills practices during PBL group discussions for the students to be more prepared for the SOCA test.

3.6 Limitations of the Study

The SDLR scores obtained in this study were limited to students’ readiness in a particular block, so the results might be different from the SDLRs in other blocks or contexts. This study did not explore and could not control other student presage factors, learning context, and process factors that could mediate the relationship between SDLR scores and SOCA test scores.

4 Conclusions

Most students failed the SOCA tests, regardless of their high and moderate levels of SDLR. The statistical analysis confirms that there was no significant correlation between SDLR and SOCA test scores. We propose that the institution equips new students with adaptation management skills to increase their self-efficacy. Tutors might facilitate the students to make the most out of PBL tutorials, especially to exercise presentation skills and critical thinking to prepare for the SOCA test. It is also important to have debriefing sessions to provide timely feedback, both for the students and the tutors. Further study is needed to clarify some assumptions mentioned. Although the study was conducted within one block only in FoM UJS, other institutions with a faculty of medicine with similar curricula may relate to our findings and suggestions.