Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

In Italian universities, teaching evaluation is in part based on students judgments concerning aspects related to courses and considered of preeminent interest for university management. A questionnaire is generally used to collect such data. The students judgments are expressed as a score on an ordinal scale.

Even if a synthetic measure of quality is required, there is no single methodological solution for aggregating individual scores. Until now several approaches have been proposed in order to define a synthetic measure of teaching quality by using student evaluations, see among others [1, 5, 18].

A possible solution is to use Structural Equation Models (SEM) [3, 14] that are used for describing and estimating conceptual structures where some latent variables, linked by linear relationships, are measured by sets of manifest variables. A double level of relationships characterizes each SEM: the first involves relationships among the latent variables (structural model), while the other considers the links between each latent variable and its own block of manifest variables (measurement model).

Given that both the quality of teaching and student satisfaction cannot be observed directly but can be measured through several real indicators, they can be treated as latent variables.

SEM applications in both evaluation and teaching quality measurement have been widely used [6, 11, 12, 15, 16].

Several techniques can be used to estimate model parameters in SEMs, which can be grouped under two different approaches. The first is the so-called covariance-based approach, based on the search for the best parameters in reconstructing the observed covariance matrix of manifest variables. A number of estimation techniques are used to estimate model parameters, including the maximum likelihood approach, which has long been the point of reference for SEM estimation. However, especially in quality evaluation studies, some limits impair its application: using maximum likelihood estimation for covariance-based SEM requires that the manifest variables follow a multinormal distribution and may lead to non-unique solutions (i.e. the model is not identifiable). Especially in social research, this distributional hypothesis is very hard to verify. Indeed, since the manifest variables are often judgments expressed on ordinal scales, they cannot properly be considered continuous variables and they are unlikely to meet the multinormal distribution hypothesis. Other estimation techniques that do not require a multinormal assumption can be used to estimate SEM parameters in a covariance-based approach, such as the Unweighted Least Squares. Nevertheless, all these techniques are based on the covariance matrix and do not allow individual behaviour to be directly taken into account.

A different approach is the component-based one. Following this approach, model estimation is basically geared to determining the latent variable scores, i.e. values of the latent variable for each individual in the sample. The main aim is to identify a latent variable explaining at the same time both its own block of indicators and the relationships between blocks. Among the component-based techniques, the most widely used method is the PLS Path Modeling algorithm (PLS-PM), also called the PLS approach to SEM [20, 24]. PLS-PM does not rely on a specific distributional hypothesis. Moreover, according to Tenenhaus [19] it provides systematic convergence of the algorithm; it allows data to be managed with a small number of individuals and a large number of variables; it provides a practical interpretation of the latent variable estimates; and it represents a general framework for multi-block analysis.

For these reasons we propose to use PLS-PM for SEM estimation in teaching evaluation, also because, since we are interested in describing students opinions, the explorative approach (typical of component-based methods) is much more coherent than the strong confirmatory one (typical of covariance-based methods).

We note that this is our contribution since in the literature of SEM application to students evaluation of teaching PLS-PM has never been used before.

2 PLS Approach to Structural Equation Models

The PLS approach to Structural Equation Models uses an iterative algorithm to obtain latent variable estimates through a system of multiple and simple regressions. The iterative algorithm works by alternating inner and outer estimates of the latent variables. In more formal terms, given the generic latent variable (ξ q ), the outer estimation of the latent variable (v q ) is obtained as a linear combination of its own manifest variables x pq :

$$\textbf{v}_{\!q}{}_\propto \pm \sum _{p=1}^{P_{q}}w_{\!pq}\textbf{x}_{pq}$$
((1))

where P q equals the number of manifest variables associated to the q-th latent variable and w pq represents the outer weight, i.e. the weight associated to each manifest variable to obtain the latent variable estimate.

In the second step (inner estimation), each latent variable is computed by considering its relations with the other latent variables. In other words, for a given outer estimate of the latent variables obtained in the previous step, the inner estimate z q of each latent ξ q is obtained as:

$$\textbf{z}_{q}\propto \sum_{q'} e_{qq'}\textbf{v}_{q'}$$
((2))

where v q′ is a generic latent variable connected to the q-th latent variable and \(e_{qq'}\) is an inner weight, usually obtained as the sign of the correlation between the outer estimates of the q-th latent variable and the q′-th latent variable (centroid scheme). The symbol ∝ means that each estimate of the latent variable has to be standardized, both in the outer and inner estimates.

The iterative procedure goes on to compute the outer weights (w pq ). Each of these weights is then used in the following outer estimate of the latent variable (equation (1)). Two different schemes are available to compute the outer weights according to the nature of the latent variables. If the latent variable is obtained as a reflective construct (mode A), i.e. if the observed variables are assumed to be the reflection of a latent concept, then the latent variable is considered a predictor of the manifest variable. Thus, each relation in the block is a simple linear regression model and may be expressed as follows:

$$\textbf{x}_{pq}=\lambda_{pq}{\boldsymbol\xi}_{q}+{\boldsymbol\epsilon}_{\!pq}$$
((3))

where λ pq is the generic loading (i.e. the correlation coefficient, if the manifest variables are scaled to unit variance) associated to the p-th manifest variable linked to the q-th latent variable, and ε pq is a residual term.

Indeed, in a reflective block each manifest variable is considered to be the reflection in the real world of an underlying concept, that is the latent variable. As a consequence, the generic outer weight w pq used in the outer estimate of the latent variable is the regression coefficient of the simple linear regression of each manifest variable on the inner estimate of the corresponding latent variable. The inner estimates of the latent variables being standardized, each outer weight (for a reflective block) is the covariance between each manifest variable and the corresponding latent variable as follows:

$$w_{\!pq}=Cov(\textbf{x}_{pq},\textbf{z}_{q})$$
((4))

In a formative scheme (Mode B), instead, each latent variable is formed by its own manifest variables. In other words, the latent variable is a function of its own indicators. In this case, a multiple linear regression model defines the relation between the latent and manifest variables:

$${\boldsymbol\xi}_{q}=\textbf{\textit{X}}_{q}{\boldsymbol{\omega}}_{q}\!+{\boldsymbol{\delta}}_{q}$$
((5))

where X q is the matrix of the manifest variable linked to the q-th latent variable, ω q is the vector of the weights associated to the q-th latent variable, and δ q the residual term.

Hence, in a formative scheme the outer weights in the iterative procedure are the regression coefficients of a multiple regression model of the inner estimate of each latent variable on its own manifest variable. For each block, the vector containing the P q outer weights is:

$$\textbf{w}_{q}=\left(\textbf{X}'_{q}\textbf{X}_{q}\right)^{-1}\textbf{X}'_{q}\textbf{z}_{q}.$$
((6))

After updating the outer weights, they are used to obtain a new outer estimate of the latent variables.

These steps are repeated until convergence between inner and outer estimates is reached. The final estimate of the generic latent variable (i.e. the latent variable score, \(\hat{\boldsymbol\xi}_{q}\)) are then computed. Then, the structural relations among the endogenous latent variable scores (\(\hat{\boldsymbol\xi}_{\!j}\)) and the exogenous one (\(\hat{\boldsymbol\xi}_{m}\)) are estimated by using standard multiple/simple linear regression models.

For a generic endogenous latent variable ξ j in the model, the structural model can be written as:

$${\boldsymbol\xi}_{\!j}=\sum_{m=1}^{M}b_{\!jm}{\boldsymbol\xi}_{m}+{\boldsymbol\zeta}_{\!j}$$
((7))

where ξ m is the generic exogenous latent variable impacting on \({\boldsymbol\xi}_{\!j}, b_{\!jm}\) is the OLS regression coefficient (path-coefficient) linking the m-th exogenous latent variable to the j-th endogenous latent variable, ζ j is a residual term, and M is the total number of exogenous latent variables impacting on the j-th endogenous latent variable.

As already stated, the PLS-PM is considered a soft modelling approach since no hard distributional hypotheses have to be made either with regard to the manifest variables or to the latent variable scores.

Unlike other estimation techniques used in the SEM framework, the PLS-PM is more prediction-oriented. Thus the quality of the model has to be evaluated in terms of prediction capability. Since two sub-models comprise each SEM, four different indexes have to be used to assess the prediction capability of the model (one measuring the performance of the measurement model, one considering the structural model and the last measuring the goodness of fit of the whole model):

  • the average communality index (measurement model goodness of fit index);

  • the redundancy indices and the R 2 values of each structural relation in the model (structural model goodness of fit index);

  • the goodness of fit index (GoF, goodness of fit index for the model as a whole).

For each block the measurement model quality is assessed by using the average communality index. This index is computed as the average of the squared correlations between each manifest variable in the q-th block and the q-th latent variable score:

$$Com_{q}=\frac{1}{P_{q}}\sum _{p=1}^{P_{q}} cor^{2}\left(\textbf{x}_{pq},\hat{{\boldsymbol\xi}}_{q}\right).$$
((8))

The average communality index is a measure of the capability of each latent variable score in explaining the variances in the manifest variables.

The quality of each relation in the structural model is measured by using the R 2 value. Moreover, for each endogenous block the redundancy index may be computed as:

$$Red_{\!j}=Com_{\!j} \times R^{2}\left(\hat{{\boldsymbol\xi}}_{\!j}, \hat{{\boldsymbol\xi}}_{m:{\boldsymbol\xi}_{m} \rightarrow {\boldsymbol\xi}_{\!j}} \right).$$
((9))

This index provides information on the part of the variability of the manifest variables linked to the j-th endogenous latent variable explained by the M exogenous latent variables impacting on it.

The global model quality is measured by means of the goodness of fit index (GoF) proposed by Amato et al. [2]. This index was constructed to provide a measure of model quality by considering model performance in both the measurement and structural models. Indeed, the GoF index comprises two parts:

$$GoF=\sqrt{ \frac{\sum\nolimits_{q=1}^{Q}\sum\nolimits_{p=1}^{P_{q}} Cor^{2}\left(\textbf{x}_{pq},\hat{{\boldsymbol\xi}}_{q}\right)} {P} \times \frac{\sum\nolimits_{j=1}^{J}R^{2}\left(\hat{{\boldsymbol\xi}}_{j}, \hat{{\boldsymbol\xi}}_{m:{\boldsymbol\xi}_{m} \rightarrow {\boldsymbol\xi}_{j} }\right)} {J}. }$$
((10))

The first term refers to the quality of the measurement model, while the second takes into account the performance of the structural model. J is the total number of endogenous latent variables in the model and P is the total number of manifest variables in the model, with \(P=\sum_{q=1}^{Q}P_{q}\).

3 Applying PLS-PM to Students Evaluation of Teaching

3.1 The Data and Model Specification

We show an example of teaching quality evaluation using a Structural Equation Model estimated by a PLS-PM algorithm. The analyzed data are the judgments expressed by 7,369 students attending courses at the Faculty of Humanities at a university in southern Italy. Judgments were collected through questionnaires distributed to the students during usual daily teaching activities in the academic year 2004/2005.

Each questionnaire is a statistical unit.

Observations do not cover the totality of enrolled students, nor are they a random sample: they were selected not by a sampling procedure, but they are the students present at one lesson of all courses (on different days). This means, for example, that each student could have filled in the questionnaire even more than once.

The structure of the questionnaire is based on a standard set of questions, as stated by the National University Evaluation Committee (CNVSU) [8] to ensure universities to have a common database recording students opinion on teaching (so that comparisons among universities, faculties, courses, etc. may be made).

The CNVSU questionnaire is organized in 5 sections. We believe that each of these sections can be considered a latent variable, such that the 15 questions can be treated like manifest variables for each of them, in an SEM sense (see Table 4.1).

Table 4.1 The logical structure of the CNVSU questionnaire
Full size table

In particular, we consider that the latent variable Interest and satisfaction is the only endogenous latent variable in the model. In other words, we suppose that Interest and satisfaction can be explained by all other aspects, so that its estimated score can be interpreted as a measure of students’ evaluation of teaching effectiveness.

In the measurement model, manifest variables are connected to the corresponding latent variables according to a reflective scheme: responses are supposed to be a logical consequence (the “reflection”) of the latent factor they are connected with.

A preliminary study [4] showed that in such a model the manifest variables describing the block Teaching and study activities are correlated with at least two different dimensions while each block should express one latent concept (see composite reliability analysis in Table 4.3). In order to avoid this inconsistency and based on the analysis of the covariance matrices among the manifest variables and among manifest and latent variables, we specified a different model, whose structure is shown in Table 4.2.

Table 4.2 Measurement model definition
Full size table

In the new model, the variable v7 (Understanding of lecture given student’s preliminary knowledge) and the block Programme organization were dropped and variable v10 (Proportion between studying load and number of credits) was moved from Teaching and study activities to Course organization block. Finally, according to the redefinition of the model, the block Teaching and study activities has been renamed Teaching. The final model is shown in Table 4.2 and in Fig. 4.1.

Fig. 4.1
figure 4_1_191533_1_En

An SEM model for students evaluation of teaching

Full size image

3.2 The Results

XLSTAT software by Addinsoft [25] was used to perform PLS-PM analysis involving only reflective indicators and the centroid scheme for the inner estimation. Since each reflective block represents only one latent construct, it needs to be unidimensional. This is why a preliminary exploratory analysis for verifying the composite reliability of blocks is required. Two different measures are available to test block unidimensionality in PLS-PM framework: Dillon-Goldstein’s rho and Cronbach’s alpha. According to Chin [7], Dillon-Goldstein’s rho is considered a better indicator than Cronbach’s alpha as it is based on the results from the model (i.e. the loadings) rather than on the correlations observed between the manifest variables in the dataset. A block is considered homogeneous if this index is greater than 0.7 [23].

As shown in Table 4.3, all five blocks of manifest variables can be considered unidimensional. Indeed, the Dillon-Goldstein Rho index is always greater than 0.7.

Table 4.3 Composite reliability
Full size table

Once the composite reliability is verified, we may look at the relationships between each manifest variable and its own latent variable. Table 4.4 shows the weights of the relationships between each manifest variable and its own latent variable, together with the average communality index, i.e. the ability of each latent variable to explain its own manifest variables. Since this index is always higher than 0.5, we can conclude that globally all the latent variables are powerful at explaining their own manifest variables.

Table 4.4 Normalized outer weights and average communalities
Full size table

The normalized weight measure the impact of the corresponding manifest variable in computing the latent variable score as an index. It is evident, for example, that the manifest variable v13 (Lecture hall) is the most important driver in computing the latent variable Facilities. The same occurs for manifest variable v16 (Overall satisfaction) with respect to the latent variable Interest and satisfaction, and for latent variable Teaching with the two manifest variables directly tied to lecturer’s quality and ability (v8 and v9).

As the distribution of PLS estimates is unknown, conventional significance testing is impossible. However, testing may be accomplished by Bootstrap methods [9]. The results of the bootstrap estimation of the standardized loadings of manifest variables are shown in Table 4.5.

Table 4.5 Measurement model estimates: loadings
Full size table

Tables 4.6 and 4.7 and Fig. 4.2 show the results of the structural model estimates. Table 4.6 shows the correlation and regression coefficients linking each exogenous latent variable to the endogenous Interest and satisfaction. We can conclude that all path coefficient estimates of the structural model are significant.

Table 4.6 Impact and contribution of exogenous latent variables on the endogenous Interest and satisfaction
Full size table
Table 4.7 Goodness of fit index for the structural model
Full size table
Fig. 4.2
figure 4_2_191533_1_En

Impact of exogenous latent variables on Interest and satisfaction

Full size image

According to the results in Table 4.6 the structural equation may also be written as follows:

Interest and satisfaction \(= 0.605 \times\) Teaching \(+\ 0.105\ \times\) Facilities \(+\ 0.077\ \times\) Course organization

Looking at the path coefficients (see Fig. 4.2 and Table 4.6), we note that students interest and satisfaction mainly depend on teaching quality (path coefficient = 0.642 and contribution to R 2 higher than 80%) while the quality of facilities and course organization have lower effects (path coefficients: 0.108 and 0.091). This is probably due to how data were collected. Since the questionnaires were distributed during the course, students attached more importance to characteristics intrinsic to that course than to general matters: aspects related to lectures prevailed very largely.

The goodness of fit indices for both the structural and measurement models are very satisfactory with an absolute GoF value of 0.537 and an equal contribution of measurement and structural models in constructing it (see Tables 4.7 and 4.8).

Table 4.8 Goodness of fit index for the whole model
Full size table
Table 4.9 Goodness of fit index for the structural model
Full size table

Finally, in Table 4.9 some descriptive statistics for latent variables scores (computed on a 0–100 scale) are shown. Recalling that the individual score of latent variables can be interpreted as the quality level perceived by a student, we can conclude that for both the latent variables Teaching and Course organization the students are fairly satisfied. Instead, the latent variable Facilities does not reach a very satisfactory level.

4 Concluding Remarks

In this chapter we used an SEM estimated by a PLS-PM algorithm to define and compute an index for measuring student’s evaluation of teaching effectiveness in universities. The proposed approach provides individual values of the index: for each student we compute a score that represents the measure of his/her perception of teaching quality. Moreover, a major advantage of using the PLS-PM approach is that it is possible to derive the weighting system for observed indicators by a data-driven procedure, once the structural and measurement models have been specified.

This issue can be set in a composite indicator framework [17]. In this perspective, the PLS-PM estimation provides a double-level weighting system [22]. Indeed, the results can be interpreted as follows: path coefficients represent the impact of the exogenous latent variables on the composite indicator (Interest and satisfaction), while the normalized weights are the weights for simple indicators (manifest variables). Together they define the coefficients of the final linear combinations (aggregation functions) for computing the composite indicator (latent variable score) at individual level.

Finally, we note that in order to consider more homogeneous contexts and compare results it would be interesting to perform PLS-PM analysis for separate groups of students according to a priori information (for example by using external variables such as the programme attended) or by running a so-called response-based clustering algorithm such as REBUS-PLS [10, 21] or FIMIX-PLS [13]. This latter issue may be an interesting topic for further work.