New Formative Evaluation Methodology on Rotating Machinery Diagnostics

Abstract

Advanced Technologies in Analysis and Diagnostic of Machinery is taught in the Master in Industrial Mechanics at Universidad Carlos III de Madrid. The difficulty of this subject is that most of the theoretical contents are advanced, and at the same time new for the students. Besides their training is uneven and the number of teaching hours is short, which makes that sometimes the contents are not easy to consolidate. At the year 2017, it was observed that most students did not participate when a question was asked by the professor. Besides, the average final exam mark was low. For this reason, from 2018 it was decided to remove the group work, that had a strong experimental component, and avoid the real measurements on the part of the students. In its place it was introduced an individual work, that involves the diagnosis of rotating machinery using vibration signals given to the students (they do not measure them). The application of the personal work is used as example during the classes, to reinforce the theoretical contents. Until that moment, the evaluation strategy was purely summative. In 2020, a new formative evaluation strategy was introduced. It is based on online questionnaires that encourage students to keep up-to-date with the course. The questionnaires were introduced in the practical sessions, accounting for 10% of the final grade. This provided valuable information for the teacher, because it is possible to see in real time if the concepts have been consolidated or not, and rethink strategies. The implementation of the questionnaires is useful also for self-evaluation of the students, since they can detect gaps in knowledge. It has been observed that the final exam grade has notably improved with the implementation of these actions.

1 Introduction

The subject ‘Advanced Technologies in Analysis and Diagnostic of Machinery’ is part of the Master in Industrial Mechanics in the second term of the second course. The course comprises 4 ECTS in 14 teaching weeks, so students have an hour and a half of class a week.

In this subject, students learn diagnosis of mechanical systems using vibration analysis. In this way, they learn advanced techniques for signal processing and mechanical systems identification based on data analysis. In general, the training of the students is uneven, and most of them students have never studied mechanical vibrations. Thus, sometimes it is difficult for them to consolidate the knowledge of this advanced concepts in 21 h of class.

The evolution of the number of students from 2017 to 2022 is shown in Fig. 24.1.

Fig. 24.1

Evolution of the number of enrolled students from 2017 to 2022

The weekly classes alternate theory and exercises sessions each week. The exercises constitute half of the teaching hours because they are essential to understand the advanced theoretical concepts explained. The exercise part is focused on examples extracted from [1]. The exercises are programmed in Matlalb® with friendly interfaces, showing real and visual examples to strengthen the knowledge acquired in the theory sessions.

At the year 2017, it was observed during the course that most students did not participate when a question was asked by the professor after every exercise. Besides, at the end of the course the average final exam mark was low. That course, the students did a group work, which proposal was presented in [2], where they had to take vibration measurements from a fault simulation machine and perform its diagnostics based on the processing of the signals taken. It was concluded that the group work done was useful to learn how to measure, but did not help improve the understanding of the theoretical contents explained.

For this reason, and due also to the increasing in the number of students (from 13 to 27) in 2018, another type of work was proposed. The idea of the new work was to substitute the group work by an individual work, giving the students vibration signals avoiding the measurement work on their part. This way, the main work would focus in the signal processing tasks and in the diagnosis. Case Western Reserve University Database [3] signals were given to the students to perform bearing diagnostics. Using the new proposal, the final exam mark average improved in the successive years. However, it was detected that students did not follow the course up to date.

In 2020, the purely summative evaluation strategy was substituted by a new formative evaluation strategy, based on the introduction of online questionnaires for the students after each exercise or example. The value of the questionnaires’ marks is 10% of the final grade.

Both strategies have shown to be positive for improving the knowledge of the students.

2 Methodology

To improve the understanding and the motivation of the students with the course, two different types of actions were taken:

• (a) 2018: introduction of an individual work.

• (b) 2020: introduction of online questionnaires.

• (a) Introduction of an individual work

The substitution of a group work by an individual work was performed to avoid the consequence that only few students of the group really worked and understood the principles used.

Besides, the new individual work removed the experimental part that was required in the previous work, presented in [2], that was useful to learn how to measure, but did not prove to be as effective in learning other types of theoretical principles.

The objective of the work is for students to perform bearing diagnostics using freely available vibration signals provided by CASE WESTERN RESERVE UNIVERSITY in [3], which are widely used in research to evaluate new techniques and parameters for machine diagnostics. In this way, students are encouraged to explore the state of the art and to go beyond the techniques explained and proposed in the course. The design of the work is carried out as a script to help the student to process signals, and to extract conclusions about the optimal conditions for the diagnosis purpose.

Approximately halfway through the term, once the basic knowledge has been imparted to be able to understand and approach it, the script is shared with them during a collective tutorial. From then on, all the contents of the course are presented with the example of the application of the work, proposing reflections and debates on certain issues, as a form of interactive formative evaluation that provides valuable information to both students and teacher. In this process they are encouraged to go beyond the knowledge imparted in the course and to delve into the state of the art.

At the end of the term, students submit an individual report in which they explain the knowledge applied, reflections and conclusions obtained. This report is also finally evaluated with a numerical grade.

• (b) Introduction of online questionnaires

The proposed questionnaires ask theoretical questions to verify that the students follow the class and that they understand it. It consists of a series of multiple-choice questions that help to strengthen the concepts studied in the theory classes and thus, consolidate them.

Until year 2019, these questionnaires were answered orally and jointly with all the students, so only some answered the questions, not making it possible to see who really understood it and who did not (see Fig. 24.2). The final grade of the course was not very high.

Fig. 24.2

Methodology before introducing the novelty of tests

To improve both the final grade and the interest and motivation of the students, it was decided to apply these synchronous tests online to improve the participation in class, using Moodle platform. In this way, everyone must answer the questionnaires (since it counts for the final grade of the course) and force them to be attentive. It was immediately observed that the students paid much more attention, participated more and were more motivated. This allows both the student and the teacher to know in real time who has answered correctly and who has not, so it is possible to emphasize more in those concepts that people have failed the most and repeat the main concepts when necessary. With the implementation of this methodology, it has been possible to improve in numerous aspects, which are all advantages as shown in Fig. 24.3.

Fig. 24.3

Methodology after introducing the novelty of tests

These advantages are:

• Higher final grade. Students demonstrated more knowledge in the final exam, obtaining higher final grade.

• More positive attitude. Students pay more attention in class and participate more.

• More participation in class. Students asked more doubts and answered the questions that the teacher asked.

• Better control of student’s knowledge. Knowing in real time how many people answered the test right or wrong allows the teacher to delve deeper into the concepts that have not been clear.

• More entertaining classes. Students are attentive and alert to answer the questionnaires well and the participation of all makes it more enjoyable.

• To improve communication between teachers: digital questionnaires are accessible for all the teachers, thus all of them can follow the learning process and draw conclusions at each topic.

• To encourage the self-evaluation on the part of the student: the student is able to control the learning process and to detect misunderstandings, and gaps in knowledge.

This formative evaluation is of an interactive type, since it occurs in an integrated way with the learning process. In this way, the student regulates the learning, and on the other hand, the teacher can detect learning problems and replan strategies.

3 Results

To see if the actions taken have been useful for the students, the average grade of the final exam has been collected for the last years. The final mark is not considered because there have been changes in the weighting of the exam and work during the years. From 2018 to 2021, the new individual work was done. In 2022 the group work proposed in [2] was done again. Regarding the questionnaires, they have been used from 2020 to 2022.

Figure 24.4 shows the evolution of the final exam grade average (out of 10) at the first call from 2017 to 2022. It can be observed that the substitution of the group work by the individual work from 2018 to 2021 helped understanding the theoretical concepts and improving the final exam marks, that decreased again in 2022 when the group work is introduced.

Fig. 24.4

Evolution of the final exam mark average (out of 10) with the actions taken

Another consequence that can be obtained from the actions taken is that the number of students that attend the final exam at the first call have improved considerably with the improvement of their knowledge and understanding, reaching values of 100% from the introduction of the questionnaires in 2020, as can be seen in Fig. 24.5.

Fig. 24.5

Evolution of percentage of attendance at the first call exam

4 Conclusions

The implementation of the actions has demonstrated to be very useful for the development of the course.

Students are more participative and the knowledge acquired is better. The introduction of the tests has allowed their self-evaluation, and encouraged them to keep up to date. Since the implementation of the tests in 2020, 100% of the students attend the exam at the first call. The performance of an individual or a group work also affects the final exam mark, seeing that when students do an individual work, the understanding of the concepts is better, avoiding that only few students of the group really work.

In general, many improvements have been detected in the students' attitude and training.