Background

An undeniable link exists between planning and assessment, and those colleges and universities which are considered effective institutions of higher learning [1]. In addition to an institution’s own interest in providing an excellent education for students, schools are held accountable by requirements of the state and federal government, accrediting agencies, and the public. Institutional standards and accountability, of modest consequence in the past, have now become significant nationwide. Accreditation bodies, such as LCME and Middle States, require that specific standards are met. Students can provide valuable feedback for both internal information and for meeting necessary external requirements.

Institutions usually query students with surveys on the last day of a course asking participants to Likert scale typical statements posed in medical education such as, “The course syllabus clearly defined course learning objectives” or “The course content matched stated course learning objectives” [2]. In considering just these two survey items, there seems to be a cause for concern, or at the very least, room for improvement. Courses in medical school are typically quite long and cover many objectives. Materials and resources tend to be voluminous. Is it reasonable to expect students to provide useful feedback to such questions? Is it likely that the results of this assessment can be used to improve the learning environment and opportunities for students? Further, what opportunity does a student have to comment on something which is not asked within the survey? Have the students truly been given a voice?

Another potential weakness to most traditional assessment is survey response rate. The American Society for Quality (ASQ) reports that “customers are most likely to respond if they are angry about something they would like to change [3]. Anger motivates people to action more than contentment [4]. On the other hand, data from Thorpe [5] and Adams [6], who looked at nonresponse in online course surveys, determined that the majority of responses came from students scoring at the highest end of the grade range and those scoring lower were much less likely to reply. Either way, the responding population may provide a skewed view.

Another difficulty medical schools face is that courses are typically taught by numerous instructors. To fairly assess a course which is so in depth by recalling on its last day, the many face-to-face and online sessions, outside material sources, and associated teachers, would require herculean efforts from the students and an unwise leap of faith from those who rely on the data provided.

Something Different—Course/Faculty Assessment Program

At SUNY Downstate, something different has been implemented—the Course/Faculty Assessment program (CFA). The CFA is a student team assessment program which evaluates both courses and associated faculty instructors. In this context, the term assessment is being used to mean the gathering of information concerning the functioning of students, administration, and the institution. The term functioning refers to the purposes of the school: social (to facilitate student learning) and development (to advance knowledge, and to contribute to community and society) [7].

Based on the premise that evaluation of a representative sample can produce results which are generalizable and applicable to the greater population [8], a sampling of students is gathered. In our large school, 200 students are divided into groups of 20–25 and randomly assigned to assess one unit or course during their preclinical education. These students, who form a CFA group, attend an orientation meeting, during which they are trained about the Course and Faculty Assessment process and in delivering professional feedback.

The Process

Continuous quality improvement (CQI) by W. Edwards Deming [9] is the basis for the innovative process. Using the principles of CQI, the Mayo Medical School [10] developed a course evaluation process, which was expanded and modified to meet the needs of SUNY Downstate College of Medicine and a few other large medical schools where this program has been implemented. Changes were made to the CFA design to accommodate the specific curricular design and the pedagogical practices of each school.

The goal of the CFA is to examine the curriculum as experienced by students. In other words, the CFA gets at what the learning experience is like for those doing the learning. The process begins and ends with students. Participants are asked to take ethnographic notes, recording anything noteworthy as they attend classroom sessions and labs. This process sets the CFA program apart from many conventional assessments in that students write down their concerns in situ, as they go through their day, rather than relying on recall at the end of the course.

At orientation, students are trained on how to participate in the CFA process. Students are instructed to develop consensus opinions and record their comments in professional language, ready to go into the reports. Students are asked to include both praise and complaints and they are reminded that words of praise can be more motivating than criticism. Since students are informed that they will present their findings in a face-to-face session with faculty and administration, they are very cognizant of the need for appropriate, professional language and students have always risen to the occasion. Also, during CFA orientation, faculty and administration have an opportunity to suggest any areas that they would like the group to consider, such as the complexity and quantity of material, the quality of the exams, resources, classroom sessions, and new initiatives. However, it is made abundantly clear that all topics are open to discussion. At orientation, a round robin style conversation is suggested, where each student has an opportunity to contribute to the discussion. Group members then meet weekly, throughout the course, to discuss their findings.

After the course concludes, the CFA team develops two brief reports (one to two pages each). One report includes comments and recommendations regarding the course and the other focuses on the faculty. The group is responsible for making a recommendation to improve any situation, regarding a course, about which a critical comment is made. Called “The Golden Rule,” this obligation requires the group to problem solve as a team. “The Golden Rule,” a term used by Deming [9], is a nod to the request for ethical behavior. It is that sentiment which frames the discussion with the group as to how to make constructive comments.

The course report is made public to the school community, posted within the school’s Learning Management System, or made available to the student government. This is an important aspect of the process as it lends transparency and accountability. Faculty comments are handled privately and delivered to the appropriate parties, as determined by the institution, such as unit directors and department chairs.

With eight years of experience, students have demonstrated that the following group structure works well. However, each group of students is welcome to develop a structure of their own.

Student Participation: Roles and Responsibilities

  • Two group leaders

  • running the meetings

  • coordinating report generation

  • overseeing attendance

  • One or two recorders

  • compiling information for reports

  • One timekeeper

  • maintaining the flow of the meeting

  • One group liaison

  • culling email input from the rest of the student body to be considered at meetings (an email account is set up for each group)

  • Student Ambassador (a volunteer from a previous group or a member of student government)

  • guiding the group as needed

Each student participates once in his or her preclinical years. Participation is mandatory, considered part of each student’s professional responsibility, and can be considered a component of the medical professionalism competency.

Faculty and Administrative Participation

The Course Director, a faculty member assigned to guide the academic aspects of the course, is invited to attend the first (and only the first) CFA meeting, during which orientation takes place. He or she is available via phone and email throughout the course to respond to questions or assist the CFA group and to address any immediate problems related to the course. Most faculty members in leadership roles are readily available to students, but CFA students know that they have a “hot line” to course administrators.

The CFA Coordinator is an administrative assistant who handles program logistics and the CFA Advisor, the Associate Dean of Assessment, is available to respond to questions and conduct orientation.

Presentation of Findings: the Post-course Round Table Meeting

CFA group leaders present their findings at a post-course Round Table meeting, which typically takes place after grades have been determined and distributed. Since data are gathered in advance of grading, this does not seem to influence student comments. The Course Director, key faculty, administrators, and any other interested students attend. At these meetings, students read their reports to the assembled group. Attendees are welcome to ask questions to clarify their understanding of student comments. All agree that these sessions are successful exchanges, but this positive outcome required a change in culture (Fig. 1).

Fig. 1
figure 1

Course faculty assessment cycle

Full size image

Outcomes

Benefits for Students

There are many benefits of the Course Faculty Assessment program. Students have an opportunity to reflect on their learning processes, to participate in collaborative problem solving, and to author professional feedback. Students know that their comments can initiate improvements, not only for future classes but for their own class. In addition, the CFA offers students an opportunity for leadership. Roles such as group leader, recorder, timekeeper, and group liaison enable students to contribute to the learning community without taking on an onerous commitment. Those who take on these roles often elect to continue their participation by volunteering to be a student ambassador for future groups. These leadership roles are noted in each student’s Dean’s Letter and can be counted toward participation in a medical education pathway, if the school has one in place. Further, CFA groups have made suggestions which have improved student performance in subsequent years.

For example, just last year, a CFA group identified that only 19% of students had immunology as part of their undergraduate studies and were therefore underprepared for the immunology presented early in medical school. With this in mind, some introductory sessions were added to the curriculum and students were successful in answering more challenging NBME questions in this area of study.

However, the most significant benefit to be highlighted here is that students like the CFA program. Survey and focus group data indicate that students feel the CFA program is worthwhile, and that the school is responding to their concerns and listening to their needs.

Benefits for Faculty

Faculty members benefit from the CFA program as well. First, the students provide detailed, actionable feedback which can be used to inform pedagogy, curricular design, and administrative processing. Second, the CFA process has increased the dialog between students and faculty/administration. Third, the CFA process has eliminated outlying student comments. In the past, students might share a small number of negative remarks, but the institution was often reluctant to take any action, knowing these were the opinions of very few. Now, critical comments result from consensus opinions among the CFA group and are likely to reflect the student body’s views. Likewise, compliments now come from a group of many, rather than from the occasional outspoken few. Also, the administration is less reliant on Likert survey responses. And lastly, comments from students can initiate faculty development.

Benefits for Administration

Insights from the CFA provide a unique perspective of curriculum as experienced by students. A tremendous amount of planning goes into curricular design, scheduling, faculty selection, and faculty preparation; yet, there are times when despite all best efforts, the learning experience is not optimal. Members of the CFA group tell us exactly what worked and what did not and their suggestions for improvement are by and large, excellent. Further, this detailed feedback helps to initiate and support administrative decisions.

Discussion

During implementation of this unique program, several challenges arose. First and foremost was the need to create a culture of assessment. Early on, some faculty and administrators felt criticized, became defensive, and even argued as students presented their findings. What changed those sentiments were conversations with faculty, and the support and leadership from the administration. At each Round Table meeting, all parties are reminded of the rules and terms of the relationship between the students and those of us who work educate them. It is made clear to students that the school is interested in their point of view—what their experiences of day to day learning are like and faculty and administration are subtly reminded not to respond defensively. All participants have become accustomed to the process. Students take the assessment process seriously and faculty have come to truly appreciate and in fact count on the feedback from students.

For any school interested in implementing this program, the CFA process is tremendously flexible. For example, students can prepare a formal report, or simply talking points for the Round Table discussion. Faculty can ask the group to address specific concerns, if they choose. CFA groups can be integrated with existing student government structures, which can work well, and some schools have used this process to gather clerkship feedback.

Assessing the Assessment

In keeping with the notion of Continous Quality Improvement [9], our Course/Faculty Assessment process is continually monitored and evaluated. Periodically, run focus groups give students and faculty an opportunity to share what they think about the process, policies, and procedures. Participants are encouraged to make suggestions and several have been implemented.

More than 80% of student recommendations have been put in place at SUNY Downstate. This high number indicates that students comments are on-point and students make excellent recommendations. Further, the school is listening and those meaningful suggestions are put into practice. Although student feedback should be only a piece of a comprehensive course and faculty evaluation at any school, it is an integral piece, which schools often struggle to gather [11].

The Course/Faculty Assessment Program has successfully created a culture of assessment. Faculty and administrators are more than simply receptive to student views, they eagerly await their comments. The CFA has changed the way data is gathered and how it is used. It has changed our students, faculty, and administrators.