Introduction

Academic quality in higher education

Student satisfaction in a higher education setting is considered as one aspect of academic quality, which is also a vital monitoring indicator of internal study programme quality. In general, linking consumer satisfaction and service quality is an approach often used in a wide range of fields (Athiyaman 1997).

Most studies on quality in higher education focus on academic factors more than on administrative ones (Onditi and Wechuli 2017). Mostly they are centred on effective course delivery mechanisms, as well as on the quality of courses and teaching. Carney (1994), for example, suggests several determinants influencing the image of a higher education institution, such as variety of courses, academic reputation, average class size, students’ academic qualification, as well as their personal qualities, encounters between the staff and the students, quality instruction, career planning, students’ social life, financial support to students, location and equipment, etc. On the other hand, Athiyaman (1997) emphasizes the importance of library services, consultation for students, teaching, students’ wellbeing, the existence of technical facilities such as computers or gyms, but also the level and difficulty of course contents and even student workload.

Higher education institutions in many European countries like the United Kingdom and Germany have become similar to service providers, and students can be regarded as customers (recipients of services) (Crawford 1991; Oldfield and Baron 2000). However, Sunder (2016, p. 1094) explains that there is evidence in the literature that the concept of “student as customer” is not entirely accepted. He argues that the students’ willingness to learn may be a driving force for their success as well, while a customer behaves according to agreed rights and obligations. Onditi and Wechuli (2017) note that the perceived academic quality may be regarded as a form of an attitude that is not the same as satisfaction. It results from a comparison between expectations and perception of performance. In higher education, the quality, in general, could be regarded as a product of evaluating service encounters, such as those with administrative staff, teachers, librarians, security staff, etc. Zeithaml et al. (1990) even propose that the academic quality should be defined as the conformance to student specifications. Oldfield and Baron (2000) see three main criteria that should be considered by a higher education institution to be perceived as a provider of a quality service: (i) enabling students to fulfil study obligations, (ii) acknowledgement of students’ desirability instead of essentiality, and (iii) functional delivery of encounters with students.

The relationship between students’ satisfaction and quality of higher education institutions

Higher education institutions have been challenged with several trends and demands in the last few decades, ultimately affecting their operation, management, and responsiveness to society. Thus, they must try harder to deliver their primary role and equip students with skills relevant to their future. In this light, Europe’s New Skills Agenda (European Commission 2016) claims that Higher education institutions need to provide society with graduates (human capital) who own relevant and up-to-date skills, which are a “pathway to employability, prosperity, and competitiveness.”(European Commission 2016, p. 2). Underneath this straightforward demand, there is a central issue: how to provide quality and relevance to what graduates learn? Besides, an increase in the number of higher education institutions in the world has led to intense competition (Isani and Virk 2005). Butta and Rehman (2010) note that only those institutions that provide quality education can excel. Quality factors can significantly influence students’ choice of decisions to enroll in a study programme but also to attend it.

Quality in higher education, in general, is a relative concept, perceived differently by different stakeholders (Sunder 2016). While students are the primary target group which higher education institutions serve, their perceptions of academic quality are regarded as the most important aspect when assessing the actual quality of teaching and services they receive. As Srivastava and Beri (2016) state, research has confirmed a positive linkage between service quality and student satisfaction. Some authors even consider the direction of causality from student satisfaction towards quality of service (e.g., Parasuraman et al. 1988; Cronin and Taylor 1992), while others take student satisfaction as a dimension of service quality (e.g., Rust and Oliver 1994). On the other hand, Sureshchandar et al. (2002) note that satisfaction is a multidimensional concept, which should be operationalized differently than service quality. Srivastava and Beri (2016) conclude that, when the students are satisfied, their education can be taken as quality education. This may be the reason why many universities monitor their academic quality by measuring student satisfaction. By following student satisfaction, they can also capably handle student’s expectations.

Butta and Rehman (2010) note that extensive research has been carried out studying the factors that influence students’ retention, as well as their satisfaction. Good academic quality education provides better learning opportunities. Satisfaction or dissatisfaction may, therefore, strongly affect students’ success or failure (Aldridge and Rowley 1998). DeShields et al. (2005) state that, by focusing on identifying and satisfying students’ needs and expectations, higher education institutions recognize the factors of student satisfaction, such as student academic achievement, faculty performance, classroom environment, learning facilities or institution reputation (Butta and Rehman 2010). These factors can be seen as focal points for management to consider while improving academic quality.

Along with the implementation of the Bologna process, quality assurance processes have gained considerable importance. The European Association for Quality Assurance in Higher Education has provided standards and guidelines to form a European quality assurance model (Kauko and Berndtson 2013). According to the Standards and Guidelines (ESG 2015), higher education institutions have the responsibility, among others, to monitor the progress and achievements of students and to perform surveys on students’ satisfaction with their courses and programmes. Emphasis on students’ satisfaction and performance is a mandatory part of internal (and external) quality assurance systems that provide a basis for continuous improvement. Consequently, higher education institutions have considerable responsibility for keeping students satisfied with their study, although this is not an easy task.

Aims of the study and research question

The concept of student satisfaction in this paper will be considered as a component of academic quality in higher education. The data used in the study is based on several monitoring approaches by which student feedback on teaching, their workload and the data about the study effectiveness were collected. The feedback of teachers plays a significant role in delivering a quality teaching (Lackey and Neill 2001).

The central research question driving this research is: How are students’ satisfaction and students’ learning related? We limit our research to the analysis of satisfaction with the delivery of the courses. The key elements defining learning and students’ effectiveness on which we base our research include the intensity of student involvement in e-classrooms (blended-learning approach), overall student workload, average grades achieved by students in a course, and the number of attempts they have had to take an examination to pass it over one academic year. The variables were used in a decision tree analysis as feature and target variables to explore their dependencies. We have not yet found any similar research that links students’ performance, satisfaction and study activities in the e-classroom with the suggested data mining technique.

Linking student satisfaction, student performance, and student involvement

Student satisfaction

Quality assessment of teaching has been a long-standing issue in higher education (Brusoni et al. 2014). For higher education institutions, the quality of their core service depends on the delivery of teaching (Douglas et al. 2006, p. 254). Teaching activities are the most important aspect when dealing with the measurement of student satisfaction, which in turn influences the overall quality perceptions of the service received by students. Research by Hill et al. (2003) highlighted that the crucial aspects affecting students’ satisfaction are classroom delivery by a teacher, feedback provided to students during the course, provision of feedback on assignments, as well as relations between teachers and students. Similarly, Stodnick and Rogers (2008) found that the most important dimensions of quality impacting the satisfaction with the course are reliability in the instructor’s way of lecturing, assurance of the instructor’s competence and the teacher’s empathy. Hasan et al. (2008) confirmed that, by improving the service quality, student satisfaction might potentially improve also.

According to Eliot and Shin (2002; p. 198), student satisfaction can be defined as “favourability of a student’s subjective evaluation of the various outcomes and experiences associated with education”. In this case, the emphasis is on subjective perceptions of teaching quality from the student perspective. Elliott and Healy (2001, p. 3) further explain that “student satisfaction results when actual performance meets or exceeds the student’s expectations”. A positive subjective evaluation of past educational experience is a good predictor of intention to continue studying (Douglas et al. 2006, p. 254). Brusoni et al. (2014, p. 12) noted that the term “excellence” (in teaching) could be identified by student satisfaction and through the performance of students in assessment. On the other hand, it can be determined by factors such as the inspirational nature of individual teachers, the organisation of presentations, the interaction with students and the match between the information provided to students and the learning objectives of the course.

Calvo-Porral et al. (2013) found that quality is perceived significantly different in private compared to public higher education institutions, and that tangibility and empathy dimensions have the most substantial influence on student’s perceived quality. The tangibility dimension is associated with facilities and equipment, and the empathy dimension concerns the attitudes of the teaching and administrative staff towards students. Yusoff et al. (2015) further identified 12 aspects that drive student satisfaction: professional environment, student assessment and learning experiences, classroom environment, lecture and tutorial facilitating goods, textbooks and tuition fees, student support facilities (including information technology facilities), business procedures, relationship with teaching staff, knowledgeable and responsive faculty, staff helpfulness, feedback and class sizes. Overall, “satisfaction” in the eyes of a student is a complex concept with foundations in the subjective perceptions of the study experience.

Student performance

Assessment of students’ performance can be regarded as an aspect of quality, as a result of learning outcomes having gained considerable attention in the European higher education area. The European Qualifications Framework for Lifelong Learning (EQF) was established to make it easier to understand and compare what people have learned (European Commission 2015). According to Skrbinjek and Dermol (2016, p. 128), a “student can achieve general and specific competencies through achievement of a measurable learning outcome”. Competencies achieved through the study can be used to define expected programme outcomes that a graduate gains after completing that study programme (Skrbinjek and Dermol 2016). Expected learning outcomes are, therefore, a measurable output of a study programme used to describe a graduate’s career profile. Moreover, Duque (2014) found that students’ learning outcomes (knowledge and skills acquisition) depend not only on perceptions of higher education quality but also on student involvement (efforts and productive interactions with other educational actors), which have a powerful effect on overall satisfaction with the educational experience.

However, retention is also closely linked to student assessment and students’ capacity to progress through education. Duque’s (2014) findings suggest that the more satisfied the students are, and the higher their perceptions of achieving the required learning outcomes are, the less likely they are to drop out from their studies. Similarly, Elliott and Healy (2001, p. 10) discussed that most students strive for high grades and, if they receive them, they will less likely drop-out. This issue may be more problematic when teachers give higher grades to enhance the level of student satisfaction (Elliott and Healy 2001, p. 10), when higher grades may not reflect the acquisition of the right learning outcomes and may influence the quality of the service received.

Student involvement

Student involvement in study activities is another important aspect of student satisfaction. Astin (1999) defines student involvement as “the amount of physical and psychological energy that the student devotes to the academic experience”. According to student involvement theory, the focus of a teacher shifts from content and teaching techniques to “what students are actually doing” (p. 526). This concept is firmly connected to student-centered learning, which supports the learning-outcome approach. Students have to be actively engaged in study activities, in such a way that they co-create their learning experience in partnership with the teacher and share the responsibility for their learning (Efimenko et al. 2018).

Still, a few questions remain to be answered: how to measure student involvement? Or how to track student involvement during the conduct of a study programme? A web-based course can provide some assistance. During web-based learning, student involvement can be monitored by their online activities in the e-classrooms (like the Moodle platform). Considering the learning benefits of web-based courses, a blended learning approach, combining face-to-face instruction and eLearning, is widely used as a teaching technique. Sadeghi et al. (2014) showed that blended learning is an effective way to increase students’ learning rate. Therefore, it is expected that the extent of blended learning will grow nationally and internationally. (Alebaikan 2012; Kim and Bonk 2006).

Woo and Kimmick (2000) tested for satisfaction and test scores differences on a web-based course compared to a traditional course. They did not find any significant differences in test scores or overall student satisfaction with a course. However, in our study, we do not control for environmental variables, as in the Input-Environment-Outcome approach (see Thurmond et al. 2002), where the primary purpose is to monitor for input differences, resulting in a more accurate estimate of environmental variables affecting student outcomes. Our study is more focused on finding connections between various aspects of student learning.

Research methodology

Methods of data analysis

In this study, we first check the causal relationships with the use of regression analysis. This approach gives us an initial insight into the relations between dependent and independent variables. However, since linear regression is a linear model, the data has to have a linear shape. When the data has a non-linear shape, a linear model cannot capture the non-linear features, and the findings of the analysis may be indecisive. Besides, in our case, we are more interested in classification as predictions of labels then in predictions of quantity, which is the primary role of a regression analysis.

Moreover, the reliance of regression analysis on a polynomial (like a straight line) to fit a dataset presents a real challenge when it comes to building a classification capability. Linear regression is also weaker than algorithms related to decision trees regarding reducing error rates. For these reasons, in our study, we also use decision trees, which do a rather good job at capturing the non-linearity in the data by dividing the space into smaller sub-spaces depending on the questions asked. Decision trees also bring in the capability to handle a dataset with a high degree of errors and missing values.

We used a machine learning analytic tool called Orange for our statistical analysis, developed at the University of Ljubljana, Faculty of Computer Sciences. We analysed the data with the data mining technique, using a decision tree induction as a regression method. Our target variable was the student satisfaction with the course, and our feature variables were the average students’ course grades, the average number of students taking the examination in the study year, average student workload at the course level and student involvement in the e-classroom.

Decision tree induction is a widely used algorithm in a variety of domains for knowledge discovery and pattern recognition using a data mining approach (Barros et al. 2015). Decision trees are an efficient nonparametric method, which can be applied either to classification or regression tasks. They are hierarchical data structures for supervised learning whereby the input space is split into local regions to predict the dependent variable (Alpaydin 2010). A decision tree can be seen as a graph G = (V, E) consisting of a finite, nonempty set of nodes (vertices) V and a set of edges E. Root and internal nodes hold a given data set attribute (or a set of attributes), and the edges correspond to the possible outcomes of the analysis. Leaf nodes can either hold class labels (classification), continuous values (regression), (non-) linear models (regression), or even models produced by other machine learning algorithms. For predicting the dependent variable value of a specific instance, one has to navigate through the decision tree. Starting from the root, one has to follow the edges according to the results of the tests over the attributes. When reaching a leaf node, the information it contains is responsible for the prediction outcome. For instance, a traditional decision tree for classification holds class labels in its leaves (Barros et al. 2015). Several approaches have been developed in the last three decades that are capable of providing reasonably accurate, if suboptimal, decision trees in a reduced amount of time. Among these approaches, there is a clear preference in the literature for algorithms that rely on a greedy, top-down, recursive partitioning strategy for the growth of the tree (top-down induction).

The main limitation of our study is the focus on courses as a unit of research. Consequently, the sample is smaller, including 49 courses in two academic years 2016/2017 until 2017/2018 at one faculty. Our research can be, therefore, marked as a case study and a starting point for future research.

Description of the data and variables

We used data from the faculty’s data warehouse database, which includes all faculty students’ data. The data warehouse was developed by experts who established a connection between the faculty’s information system and the data warehouse, where all data is easily accessible and manageable from SharePoint. The target group of observation was students enrolled in the programme economics and business at bachelor’s level. The data was provided by course level (also with the aim to protect the personal information of students) for all 49 courses, out of which 16 courses were delivered in the first year of study, 18 courses in the second year and 15 in the third year. All the courses were implemented as a blended learning experience. We used continuous data for initial testing of a causal relationship with regression analysis. For decision tree induction, the data was discretised (2 intervals and equal-width discretisation). In Table 1, we present the discretised data for all 49 courses.

Table 1 Data on the variables
Full size table

We used the following variables: (i) SATISFACT - average student satisfaction with the course, (ii) GRADE - the average students grade of the courses - all grades in the scale from 1 to 5 (knowledge does not meet minimal criteria) up to 10 (exceptional results without or with negligible faults) were included in the calculation, (iii) EXAPPROACH - average number of students’ attempts to pass the examination in one academic year, (iv) EINVOLV measuring average student’s participation in the e classroom with the sum of responses and their active engagement (views and posts) in the e-classroom modules divided by the number of enrolled students, and (v) WORKLOAD - average student workload at the course level.

The values of GRADE and EXAPPROACH were retrieved from the faculty data warehouse. SATISFACT was taken from the students’ survey analyses and calculated as the mean values of satisfaction values of eight items, such as teacher’s instruction, methods used, study materials and other factors. WORKLOAD was also taken from the students’ survey (share of students with a high perception of workload were doubled, and the percentage of students with an adequate impression of workload was added). The data on active student involvement in the e-classroom (EINVOLV) were taken from the e-classroom statistics report (views and posts reports).

The course satisfaction survey contains many variables, but this research considered eight items from which the average value was calculated for each course (see Table 2). Once the course ended, students enrolled in the courses were responding to the question: How would you rate the course structure, content and material?

Table 2 Questions measuring student satisfaction: How would you rate the course structure, content and material?
Full size table

Results

Initial regression analysis

Multiple linear regression analysis was used to test if the independent variables GRADE, EXAPPROACH, WORKLOAD and EINVOLV significantly predict the dependent variable SATISFAC. The results of the regression indicate the four predictors explain 22% of the variance (R2 = 0.19, F (4, 44) =2.60, p < 0.05). However, from the model, we can assume that only GRADE significantly predicts student satisfaction (β =0.11, p < 0.05), and all other independent variables do not statistically significantly predict the dependent variable. Except for GRADE (β = −0.12, p < 0.05), we can assume that the strength of the impact of all other independent variables is negligible and not statistically significant (Table 3).

Table 3 Summary output of regression analysis
Full size table

These findings correspond to the general perception that students are more satisfied if the grades are high. However, our further analysis reveals a different perspective.

Decision tree induction

The results of the decision tree induction analysis are visualized in Fig. 1. From Fig. 1 (right side) we can assume that the probability of low satisfaction with the courses equals to the situation in which the average grade is low (GRADE < 7.01), the requirements for involvement in e-classroom activities high (EINVOLV ≥295.27), and overall student workload high (WORKLOAD ≥0.9). On the other hand (Fig. 1, left side), we can assume that the probability of high satisfaction with a course equals the situation in which the requirements for involvement in e-classroom activities are high (EINVOLV ≥295.27), the average grade is high (GRADE ≥ 7.01), and overall student workload low (WORKLOAD <0.9).

Fig. 1
figure 1

Decision three; Target variable: course satisfaction

Full size image

Another important observation is that EXAPPROACH is not a decisive variable, which means that the majority of students pass the examinations in less than 1.6 attempts. This finding needs to be interpreted within the faculty’s context, as teachers may grade students during the course progress and following the accomplished activities. Therefore, the final examination may be a matter of formality.

As can be seen from Fig. 1, in the case of low satisfaction, the average grade might not be of crucial importance, which means that most of the students may be dissatisfied with a course when the requirements for involvement in the e-classroom, as well as the student workload, are both high. Similarly, when satisfaction is high, the extent of participation in e-classroom may be somehow irrelevant, which means that a student may be satisfied with a course if the average grade is high and student workload low.

The model of decision tree induction offers reasonably prediction probabilities - the probability of a correct prediction based on the decision tree classification is about 64% for high satisfaction and about 61% for low satisfaction.

Conclusion and discussion

In our study, we found that the student workload stemming from student involvement in in-class and homework activities, as well as the extent of activities in the e-classroom, in most cases relate negatively to the level of student satisfaction. This relation is apparent especially in the case of low satisfaction. However, in the case of high student satisfaction, the decrease of in-class or homework activities may be significant, while the increase of the extent of activities in the e-classroom may also slightly increase the probability of high student satisfaction. This finding might lead to the conclusion that students are much more satisfied with a course when the average grades are high and when the workload is not too high, and when a part of the workload moves to the environment of the e-classroom.

Both analyses – regression analysis as well as decision tree induction – emphasized the role of the average grades students achieve in the courses, which is in line with the study of Duque (2014). The relationship seems to be positive, which means that an increase in a grade brings an increase in the level of student satisfaction. However, grades do not seem to be extremely relevant in the case of students’ low satisfaction. The probability of low satisfaction is only slightly higher when the average grades are low. Such findings may imply that dissatisfied students are most probably those students with low average grades, as well as with low expectations regarding the grades. Such students seem to be especially sensitive about the extent of work related to in-class and homework activities, as well as activities in e-classrooms. Nevertheless, student workload can be regarded as a good predictor of student satisfaction.

Otherwise, the variables defining student satisfaction with a course seem to be more or less expected. Students would be more satisfied if they get a higher grade, which is very generic and expected. The findings of our study are throwing some doubts on student-centered learning and student involvement paradigms, as approaches positively influence commitment to learning and student satisfaction with the courses. Teachers have to be more careful when designing courses regarding student activities, as many activities may lead to low satisfaction and subsequently low academic quality. Such finding implies that conditions for high student satisfaction, which ultimately leads to good quality education, will be satisfied if student’s workload, involvement and grades are balanced regarding the right amount of the various activities that match the grade and learning outcomes. Similarly, Machado and Afonso (2018) found positive associations between satisfaction with interactivity, and with performance and students’ learning results.

Further research should be done in this area. The limitations of our methodological approach might have biased the results of the study. For further analysis, we suggest increasing the amount of data (number of courses, more study years, etc.), shifting the focus of the study to the individual student level, as well as including other data-mining techniques such as logistic regression or clustering. Moreover, as Calvo-Porral et al. (2013) suggest, quality is differently perceived in private and public higher education institutions. As a result, a comparative analysis can provide an understanding of the differences and deviations in satisfaction level.

This research has, however, provided valuable insights into students’ satisfaction with their courses, as well as the relationship between students’ satisfaction, study performance and involvement in study activities with a data mining approach. Our findings may help teachers to improve their learning styles and reflect on their work as part of the total quality management process as reflected in the work of Sfakianaki et al. (2018). Furthermore, results can be used by faculty management to improve the academic quality of departments, programmes and institutions. After all, the students are the ultimate judge.