Keywords

1 Introduction

Programmatic assessment is a novel way to approach assessment as a more comprehensive decision-making process based on multiple and triangulated data concerning student performance. Given the strengths and weaknesses of each assessment instrument, the assessment decision of pass/fail should be made on something other than individual assessment but instead made after thoroughly considering all data collected regarding student performance. Within the scheme of programmatic assessment, students will first undergo training activities, and afterwards, there will be numerous assessment activities in which each activity will produce a data point. All data points will be collected and subjected to intermediate and final evaluations, with each aimed to evaluate students’ progress and end achievement, respectively. Throughout training and assessment activities, students will be supported with supporting activities such as supervision, mentoring, reflection, and feedback. All these concepts and components of a programmatic assessment are elaborated in the work of van der Vleuten and colleagues [1].

Portfolios hold an essential role in a programmatic assessment. It functions as a platform to collect all data points, record, and reflect on student progress and achievement. By means of a portfolio, a longitudinal assessment of student performance can be made based on triangulated data points on assessment. An electronic portfolio (e-portfolio) is considered a robust system to collect all data points, including their feedback [2, 3] Overall, as Heeneman et al. [3] summarized, an e-portfolio serves as a repository of all student data to facilitate the administerial aspects of the assessment processes, to provide aggregated data, and facilitate student reflection on their learning. Therefore, an e-portfolio functions more than just as an assessment tool but also as a learning tool, aligned with the focus of programmatic assessment as an assessment for learning [3].

Despite the potential benefits of a programmatic assessment as widely discussed in the literature, especially ones that originated from the Western context, less is known about how programmatic assessment will fit into the context of medical education in the Eastern setting, including Indonesia. Applying a programmatic assessment requires a shift of mindset, especially in how we see, consider, and utilize assessment data. Unfortunately, studies on the development and use of programmatic assessment in Indonesia are still lacking. A study by Ainin et al. [4] was conducted to develop an instrument to assess programmatic assessment implementation in health professions education institutions but has yet to examine the actual implementation of programmatic assessment. Therefore, more studies are required to establish the best ways to implement programmatic assessment in our own setting.

As a part of a larger project aiming to implement programmatic assessment as an undergraduate medical national exit exam in Indonesia, one of the first steps to be taken is to develop an e-portfolio system. We used the design thinking approach to develop the e-portfolio since it will be widely used throughout medical schools in Indonesia. Design thinking is an approach where we try to create an innovation, either a product or a solution, based on a deep understanding of the potential users, identification of problems and iterative process of prototyping and testing [5]. Thus, the current study aimed to design a national-level e-portfolio system for undergraduate medical education in Indonesia.

2 Methods

We conducted participatory action research [6] as an iterative process to develop an e-portfolio design using the Stanford five-step design thinking approach [5]: empathize, define, ideate, prototype, and test. These steps were conducted to be able to clearly define the priority features of the system by valuing opinions from stakeholders, evaluating its feasibility and acceptability, and improving the system iteratively [5, 7].

The application of learning and assessment methods, such as portfolios, should follow the best practices as well as consider local contexts and cultural backgrounds in its implementation [8]. Best practices in developing and utilizing e-portfolios were reviewed by the research team based on several theoretical frameworks: programmatic assessment, facilitating feedback provision and reflective practice. We conducted literature reviews on portfolio systems and implementation (i.e., van der Schaaf et al. [9]; Moores et al. [10]; Driessen et al. [11]; Buckley et al. [12]) and results of related pilot studies about portfolios conducted in Indonesia, which were still very limited [13, 14].

Intensive and iterative internal discussions were then conducted among the research team members and the information technology (IT) system developers in order to develop the baseline systems of the e-portfolio, consisting of the student interface, teacher interface, and backend system for the administrator. A linear-sequential life-cycle software development based on the Waterfall model (Fig. 1) was implemented in this project to complete each sequential stage and downward to the system deployment [15]. Initially, use-case diagrams were created to represent the interrelation between functional requirements. Non-functional requirements were also identified in this stage. The system development continued to design, implementation, pilot testing, and deployment within two months. The initial version of the system that was developed was built upon several generic features:

Fig. 1
A waterfall model represents the e-portfolio system development requirements, designs, implementation, testing, and maintenance during weeks 1 to 3, 3 to 4, 4 to 6, 7, and 8, respectively.

Waterfall model with the corresponding schedule for e-portfolio system development

Full size image
  1. 1.

    The e-portfolio system used the National Medical Doctor Competences (NMDC) as the final competence framework to comprehensively assess all competency areas and accommodate different medical schools’ final learning outcomes.

  2. 2.

    Each medical school would be trained to prepare assessment data points and their connection towards their final learning outcomes and the NMDC.

  3. 3.

    The e-portfolio system allowed data points re-grouping according to the NMDC.

  4. 4.

    The piloting stage was planned to use retrospective data points from the past two semesters.

  5. 5.

    Self-reflection prompts were provided in the student interface.

  6. 6.

    Feedback prompts were provided in the teacher interface, including future plans for students’ learning.

  • Empathize

The process of empathizing was conducted by inviting stakeholders: medical teacher representatives from 93 medical schools in Indonesia, endorsed by the Ministry of Education, Culture, Research, and Technology Republic Indonesia, as well as the Indonesia Medical Education Association (IMEA). The workshop was conducted online using the Zoom Meeting platform. The workshop aimed to provide a venue for developing a shared vision/perception and further discussing the feasibility of e-portfolio implementation. The workshop began with a short presentation reviewing outcome-based education (OBE), programmatic assessment, and portfolio concepts and implementations, followed by a demonstration of the baseline e-portfolio systems for each interface. Workshop participants were then assigned to six breakout rooms with facilitators to further participate in focus group discussions to explore the current curriculum practices in their respective medical schools, potential data points, as well as expectations and concerns of the e-portfolio implementation. Data from the focus group discussions were audio recorded and analyzed thematically to identify perceptions and needs of medical school institutions’ stakeholders towards the e-portfolio implementation.

  • Define

While engaging prospective stakeholders in focus group discussions, several problems and potential challenges in e-portfolio implementation were identified. The issues were related to four curricular aspects: OBE implementation, prior experience and familiarity towards the e-portfolio system, mentorship process, and faculty development program, as follows:

  1. 1.

    Implementation of outcome-based education

    Synchronization and implementation of OBE in each medical school varies and remains challenging, especially regarding aligning the assessment instruments or tools into the final learning outcomes. This challenge illustrated that the starting point for e-portfolio implementation would be different for each medical school and that the system needed to be made flexible enough to be adapted to the current curriculum structure at each medical school.

  2. 2.

    Prior experience and familiarity towards the e-portfolio system

    Generally, stakeholders had a positive perception towards the generic system being demonstrated and looked forward to its implementation nationally. The ability to include summative and formative assessment results as data points was mentioned by stakeholders. However, stakeholders’ prior experience and familiarity with e-portfolio also varied. Only a few stakeholders had previous experience in piloting portfolios for clinical education. Some other stakeholders wondered how this e-portfolio system would differ from their respective schools' learning management or academic systems. Therefore, integration of this e-portfolio system with other existing systems was expected. Stakeholders also highlighted potential challenges concerning limited time and human resources in their respective medical schools to manage the backend data.

  3. 3.

    Mentorship process

    Stakeholders identified that the current challenges in conducting mentoring processes between academic counsellors and students were due to the large number of students in the program. The intensity, frequency, and quality of mentoring remain a challenge, especially for clinical students. Relationships between faculty members and students at a high-power distance affected students and faculty engagement in mentorship programs. Both academic counsellors and students often needed clarification about what to discuss during the mentoring program. Students' engagement in the mentorship program could have been higher. Academic counsellors found it challenging to decide to what extent they needed to assist students' learning and provide feedback.

  4. 4.

    Faculty development program

    Stakeholders identified the need for a continuous and longitudinal faculty development program in conducting a thorough and sustainable mentoring process and to maintain faculty members' motivation to utilize data in the e-portfolio systems thoroughly, being engaged in feedback dialogue to support student’s personal and professional development. A faculty development program for the portfolio assessment team should also be provided.

  • Ideate

We conducted the ideation stage using the ‘value proposition canvas’ [16] as depicted in Fig. 2.

Fig. 2
A combined square and circle diagram lists the products and services, gain creator, pain relievers, gain, pain, and jobs to be completed.

The results of the ideation stage in ‘value proposition canvas’ [16]

Full size image
  1. 1.

    Gains: Easily accessible platform across teacher generations was necessary; the e-Portfolio platform should have distinguished aims among other platforms (Learning Management System [LMS] and academic platforms) and help monitor students’ academic performance development.

    Gain creator: Using spider web (radar chart) according to NMDC for easy visualization; The spider web was created automatically by the system according to the alignment of data points (assessment scores- quantitative); Comprehensive monitoring is conducted by asking students to reflect upon their academic performance and provide additional learning evidence upon reflecting on their performance followed with monitoring by academic supervisors using SOAP framework

  2. 2.

    Pains: OBE practice was found to vary across institutions; Institutions mentioned limited time and human resources in their respective medical schools to manage the backend data; Feedback provision from faculty members was lacking; Reflective ability of students was still varied, and students’ engagement in mentorship program was lacking.

    Pain Relievers: the starting point for e-portfolio implementation will be different for each medical school (i.e., number of students and faculty members involved, types of assessments as data points, recorded period as data points, etc.), and the system needs to be made flexible enough to be adapted to the current curriculum structure at each medical school. Each medical school was expected to assign one e-portfolio coordinator (faculty member) and one e-portfolio administrator to be in charge of the piloting program; reflection has to be submitted by students before the mentoring session on their last semester performance for each competence with reflection prompts provided, while the feedback framework for the academic counsellor was also provided.

  3. 3.

    Customer Jobs: Engage students and teachers in reflective dialogue for students’ professional development; Increase stakeholders’ familiarity with e-portfolio utilization for professional development.

    Product/services: System development consisted of features that encourage student reflection and feedback provision of teachers; Workshop for academic supervisor as well as for administrator were conducted; Guidebook for academic supervisors and students were provided; a Helpdesk for backend data development and collective/direct upload of curriculum matrix by IT development team was also provided

3 Results

The generic e-portfolio features above reflect the application of OBE and programmatic assessment. Enhancement of the features was conducted in the ideation stage by applying the results of the define stage, leading to the prototyping phase. The developed national e-portfolio integrates summative assessment results, maps students’ performance with NMDC, and consolidates them with reflective inquiries. Based on the development approach above, a prototype of the national e-portfolio was established to incorporate several features clustered into the SOAP mnemonic below, which stands for S (Subjective—Student Reflection), O (Objective—Assessment Outcomes), A (Assessment—Diagnosing Learning Issues), and P (Plan—Formulating Improvement and Learning Plan).

3.1 Subjective—Student Reflection

This section allows students (Fig. 3) to conduct reflections based on their learning achievements and experience in the recent period, which is visualized through a radar chart according to each competency area in the NMDC. The reflection process applies a constructive reflection approach to incorporate the description of experiences, learning processes and difficulties that occurred. The academic advisors are able to see students’ reflections before conducting advisory meetings. Hence, they are aware of the student’s point of view and use it as the foundation for providing feedback and further discussion.

Fig. 3
A screenshot of the self-reflection form. It includes the options to choose a batch, meeting number, description of your feelings and experiences, areas that need improvement, choosing semester, date, accomplishments, and future plans.

Self-reflection form, allowing students to evaluate their current state [17]. This figure was taken from the Intellectual Property Rights Min. of Law Rep. of Indonesia by Utomo et al. [17], No. EC00202267866

Full size image

3.2 Objective—Assessment Outcomes

This section provides information on the student's academic performance. The distinctive approach of this national e-portfolio from other regular academic information systems is the attempt to quantitatively map the results of students’ assessments (i.e., examination scores, scores acquired in problem-based learning (PBL) discussions, etc.) into the graduate learning outcomes in students’ respective institutions and the national competencies achievement. Both students and advisors can monitor the achievements using radar charts (Fig. 4).

Fig. 4
A radar chart of national competencies consists of self-awareness, effective communication, disease management, professionalism, clinical competencies, informational management, and basic biomedical sciences. A triangle denotes the graduate profile with clinician, researcher, and academic roles.

NMDC achievement is illustrated in a radar chart in conjunction with the institutional graduate profile

Full size image

3.3 Assessment—Diagnosing Learning Issues

Based on the Subjective and Objective data obtained and observed, the academic advisor as mentor evaluates students’ progress, clarifies any learning issues, and provides feedback to be further discussed with each mentee. The written feedback in the system is used for advisors to provide mentoring advice and as guidelines for students in creating further action plans for their study. During this mentoring phase, students can also clarify any issues to the advisor. The ‘Assessment’ is a collaborative process facilitated by the advisor (Fig. 5).

Fig. 5
A table lists the batch, name, and semester, with another table below consisting of 6 columns with headers including number, block, national competence, criteria, class average, and score. The Print tab is highlighted below on the left.

Objective data points originating from summative and formative assessments were recorded

Full size image

3.4 Plan—Formulating Improvement and Learning Plans

After completing the ‘Subjective’, ‘Objective’ and the ‘Assessment’ phases, the reflective journey comes to the ‘Plan’ phase, where students and advisors determine points for future improvement and plan subsequent learning goals. The process is collaborative and is documented in the e-portfolio. Both students and advisors should agree with the learning plans before submitting the advisory meeting report. The ‘Plan’ section can be reviewed in the subsequent advisory meeting (Fig. 6).

Fig. 6
A table with options of the selected batch, student, student's progress related to learning outcomes, current status compared with the initial plan, selected semester, description of the student's status, and plan. Below are the mandatory checkboxes for the declaration with a cancel button.

An academic advisor is responsible for facilitating students to develop their future study plans

Full size image

The national e-portfolio has been introduced to 34 medical schools to evaluate its adaptability towards curricular variation among medical schools. The current e-portfolio can facilitate medical school curricula with different graduate outcomes, since most medical schools already have assessment mapping towards the respective outcomes and the national competencies framework.

A training was also delivered to faculty advisors and institutional portfolio administrators. Hence, each medical school has distinct access to their respective e-portfolio to maintain data confidentiality. The training allows each medical school to operate and modify the e-portfolio independently.

4 Discussion

The use of e-portfolio benefited students and advisors in maintaining learning achievement and institutions or medical schools to evaluate the application of outcome-based curricula and align the assessment towards the expected competence. The portfolio may serve as an instrument to document students’ progress in achieving the graduate outcomes of their medical education [18]. Introduction of the e-portfolio has triggered medical schools to reflect and re-examine whether OBE and programmatic assessment principles have been sufficiently incorporated into their curricula. Additionally, the process of developing the backend data of the e-portfolio has been used by medical schools to verify the alignment of each assessment instrument towards the course learning outcomes and graduate outcomes.

However, this piloting program shows that the initial process of creating, data input, and establishing the e-portfolio was rigorous and labor-intensive. There is a need for each institution to assign dedicated faculty members and supporting staff to prepare the backend data and further operate the e-portfolio. Moreover, faculty development programs are also required to ensure the appropriate use of portfolios [11], and to enhance the benefit of e-portfolio towards students’ learning advancement.

The application of e-portfolio and programmatic assessment requires students and advisors to be familiar with reflective inquiries. The e-portfolio should be managed and interpreted appropriately. Otherwise, it might become an assessment pile and thus will make all intensive work and resources in vain. Hence, there is a need to provide training in reflective practice for students and faculty members so that the value of the e-portfolio might be enhanced. Even at a national level, a single assessment cannot necessarily reflect and predict medical graduates’ performance [19]. Hence, using a portfolio with a comprehensive attempt to portray competency attainment and record the collaborative process between mentees and mentors during its implementation might also provide a better understanding and prediction of the graduates’ competencies.

5 Competing Interests

The authors declare no competing interests related to the study and manuscript writing.