Keywords

1 Introduction

Distance education is the learning process in which instructors and learners are not physically present at the educational institution (Kaplan and Haenlein 2016). The flexibility of distance education curricula allows more learners to participate in the educational process, as it reduces the barrier of limited time imposed by personal responsibilities and commitments, and geographical distance, and provides access to the educational process to people from many different geographical areas and socio-economic backgrounds (Oblinger 2000; Masson 2014). Distance education programs can be both innovative (Masson 2014), and just as effective as the traditional face-to-face learning programs (Nguyen 2015), especially if the former is conducted using digital means (e-learning technologies), and the instructor has specialized knowledge and experience in online teaching methods, which are quite different from the traditional ones (Masson 2014). Furthermore, most textbooks are available in digital forms. Also, all e-learners have equal access to the digital education, regardless of demographic factors, such as socio-economic status, place of residence, gender, origin, age, or tuition (Casey and Lorenzen 2010). Digital distance education, in comparison with the traditional education, enables learners to learn in the most appropriate for each of them way and pace, to follow the courses according to their needs and background, focusing on and spending more time in subjects in which they have weaker knowledge (Kirtman 2009). Through asynchronous and synchronous digital distance learning, students can have continuous access to the educational material at the time and pace they desire, and also can have collaboration (e.g., for group assignments and projects) flexibly (Masson 2014). Digital distance education can be highly beneficial for learners and instructors during a pandemic, as there they do not have to move to the educational institution and have direct contact, which contributes to the mitigation of the transmission of the disease (Masson 2014).

However, in addition to the above advantages of digital distance education, asynchronous and synchronous, there are also some disadvantages. According to Al-Saleh (2013), the lack of direct interaction, cooperation, and communication between the learner and the instructor, makes it more difficult for learners to ask questions and additional information from the instructor, which has negative impact on the quality of the education of the former. In addition, weaknesses and problems of the technological infrastructure in educational institutions, especially with respect to the high speed and availability access of the students and instructors to the educational content, as well as to synchronous e-learning sessions, may lead to further problems in the education quality (Jawida et al. 2019).

The Greek government rapidly after the outbreak of the Covid-19 pandemic in order to mitigate its spread adopted a central policy for all universities of shutdown, and at the same time of continuing their educational activities through asynchronous and synchronous online teaching, based on e-learning platforms and online educational material (Bao 2020; Crawford et al. 2020). In particular, in early March 2020 a Legislative Act titled ‘Urgent Measures for Handling the Negative Consequences of the Appearance of Covid-19’ was issued by the Greek government (Government Gazette Α, 55, March 11th, 2020), which in Article 12 included the shutdown of all Greek universities, and at the same time the continuation of their educational activities using digital distance learning methods. A few days later (16/3/2020) the Ministry of Education issued a relevant Administrative Circular titled ‘Application of Distance Learning in Higher Education Institutions’, which included guidelines for the practical implementation of digital distance learning, using not only asynchronous e-learning methods (upload educational content to an electronic platform), but also synchronous ones. For the universities that already had e-learning platforms with sufficient capacity this should start immediately, while the remaining ones were given quite strict deadlines for upgrading their e-learning platforms within a short time period, and also the option of using e-learning platforms that had been offered by Google and Microsoft.

It is quite important to evaluate the application of these digital distance learning policies of the Greek government for the universities during the Covid-19 period, which have been extensively and intensively debated (and there has been strong confrontation among politicians about them), and investigate the degree of their success or failure, and also identify their strengths and weaknesses. In general, it is important to gain as much knowledge as possible from the application of these digital distance learning policies of the Greek government in the universities during the Covid-19 period, which can be quite useful for making the required improvements of them. This is going to be highly beneficial, as digital distance learning is expected to be used extensively in the future in higher education, as both instructors and students have become more familiar with this education method: a) for postgraduate programs and continuous education; b) for the provision of cross-departmental courses (attended by students of several different departments, which are located in different cities); c) for the provision of higher education to special groups of students who cannot participate in the ‘traditional’ face-to-face learning processes.

This study aims to make a contribution in this direction: it evaluates these government policies of adopting digital distance learning in higher education during the Covid-19 period, based on public policy evaluation theory (Adelle and Weiland 2012; Wollmann 2016; Vedung 2017; Bundi and Trein 2022) (outlined in Sect. 2.1), with respect to both their direct outputs, meant as educational resources (technological and human) provided to the students, as well as educational outcomes. So, the main research objectives of this study are:

  1. i)

    to assess the quality of the main educational resources provided to the students during the Covid-19 period: the quality of the e-learning platform and the quality of ‘e-instruction’ (i.e. teaching the distant students by instructors though the e-learning platform);

  2. ii)

    to assess the educational outcomes of the digital distance learning during the Covid-19 period: level of understanding the online lectures as well as of concentration on them, in comparison with the traditional face-to-face teaching, and finally extent of online participation in the online exams.

For this purpose, evaluation data have been collected through a survey of undergraduate students of the Department of Information and Communication Systems Engineering of the University of Aegean, Greece. The statistical analysis of their responses allows drawing interesting and useful conclusions concerning their degree of satisfaction with the technological and human educational resources provided to the students during the Covid-19 period, as well as the level of understanding the online lectures as well as of concentration on them, and finally the extent of their participation in a critical element of the university courses: the examinations, which had been conducted online as well.

In the following Sect. 2 a brief review of representative relevant literature is provided, while in Sect. 3 the method and data of this study are described, followed by the results in Sect. 4 and finally the conclusions in Sect. 5.

2 Literature Review

2.1 Public Policy Evaluation

Since the policy interventions of modern state has become quite costly (consuming large amounts of taxpayers’ money), complex to implement and also have high impacts on the economy and the society, their comprehensive and rational evaluation is an imperative, so there has been extensive research and practical effort in this direction (Adelle and Weiland 2012; Wollmann 2016; Vedung 2017; Bundi and Trein 2022). Public policy evaluation can be defined as ‘careful retrospective assessment of the merit, worth, and value of administration, output, and outcome of government interventions, which is intended to play a role in future, practical action situations’ (Vedung 2017). It is usually conducted after the end of the implementation of a public policy (ex-post evaluation) in order to assess its impact as well as the degree of attainment its objectives, and also identify weaknesses and possible improvements, and in general gain relevant knowledge, which can be useful for future relevant decision-making and for the design and implementation of similar policies in the future. Furthermore, it can be conducted also before the implementation of a public policy (ex-ante evaluation) in order to assess on one hand its costs and on the other hand its impacts, and examine whether the latter are worthy of the former, and also to assess and compare alternative courses of action.

A public policy requires some ‘inputs’ (including usually financial and human resources), which are used by the government agency(ies) responsible for this policy in order to produce some direct ‘outputs’; these outputs affect the target group (e.g., some citizens or firms) of the public policy and produce some first-level ‘outcomes’ (meant as impacts and changes in their situation/behavior), and possibly some second-level elements, etc.; this ‘structure’ of a public policy is shown in Fig. 1; therefore the general methodology of the evaluation of a public policy has to follow this structure, and focus on these three main elements of it (inputs, outputs and outcomes) as well as the relationships among them (Vedung 2017).

Fig. 1.
figure 1

The structure of public policy and its evaluation methodology

Full size image

2.2 Evaluation of Digital Distance Learning in Higher Education

Some research has been conducted for the evaluation of the digital distance learning policies adopted by governments during the Covid-19 period in higher education. It is useful to review the most representative of these studies. Fabriz et al. (2021), based on a survey of 3056 students and 396 instructors from a large German university, investigated whether the dominance of synchronous or asynchronous online teaching and learning in higher education during the Covid-19 period has affected the whole experience of students as well as their performance/results. In addition, it examined how well these two online teaching and learning methods satisfy students’ basic psychological needs for autonomy, competence, and relatedness suggested by self-determination theory. The results suggest that students who were taught mainly through synchronous methods reported more self-centered activities, such as feedback, than students taught mainly through asynchronous method. In contrast, teachers perceived fewer differences between these two teaching methods (synchronous and asynchronous online learning), especially concerning students’ feedback activities.

A qualitative case study, assisted by an online survey, has been conducted by Irfan et al. (2020) aiming to identify the barriers that arise during online learning in the mathematics domain in higher education. The data was collected through an online survey consisting of 27 structured questions concerning basic skills challenges, teaching and learning challenges, and university challenges. Twenty-six Professors from universities in Sumatra, Java, Kalimantan, and Sulawesi who teach mathematics participated in the research. The results of this study reveal that all teachers used a Learning Management System (LMS)-based website as a means of online teaching: The learning management system-based platform is the most widely used (google class and Edmodo), while video conferencing is the second choice (Zoom and Skype). It has been concluded that there have been significant obstacles, such as the limitations of writing mathematical symbols and the limited basic capabilities of the system and multimedia software to support online learning.

Baxter and Hainey (2022) explore views of students of a UK higher education institution concerning distance online delivery. Students were asked about their views on distance learning and the psychological impact it had on students and students’ studies. The research provided students with an opportunity to reflect on whether the practice of providing distance education continues to provide students with a beneficial learning experience. The research adopted a case study methodology using questionnaires; in total, 894 students completed the questionnaire. The survey findings showed that some participants felt that distance learning was beneficial for immediate feedback, motivational support, and encouragement. The negative findings identified consequences for feeling isolated and unmotivated and a preference for face-to-face delivery.

However, further research is required concerning the digital distance learning in the higher education during the Covid-19 period, in different national contexts and thematic disciplines, in order to obtain extensive knowledge about various aspects of it, which can provide a strong basis for reaching higher levels of maturity of it.

3 Method and Data

The method of this study was based on the theory of public policy evaluation outlined in Sect. 2.1, and especially on the general methodology of public policy evaluation shown in Fig. 1. We have focused our evaluation:

  1. a)

    on the outputs that have been provided to the students as part of this government digital distance learning policy during the Covid-19 period: the technological resources (the e-learning platform) provided to them as well as the human resources (the ‘e-instruction’, meant as distant teaching by instructors though the e-learning platform) provided to them;

  2. b)

    and on the educational outcomes of the digital distance learning during the Covid-19 period: the level of understanding the online lectures as well as the level of concentration on them, in comparison with the traditional face-to-face teaching and also the extent of online participation in the online exams (as the exams constitute the final and highly important stage of a university course, and the extent of students’ participation in these online exams is significantly affected by - and is a good measure of - the level of learning they have achieved through the online teaching of theory and labs during the semester - if students feel that they have not gain sufficient knowledge through the online teaching they will probably have lower propensity to participate in the online exams;

    Furthermore, we have examined the relationships between the above policy outputs and outcomes (in order to investigate which of the former affect the latter). Our evaluation method is shown below in Fig. 2.

    Fig. 2.
    figure 2

    Evaluation method

    Full size image

Data were collected through a questionnaire (provided in the Appendix) from students of the Department of Information and Communication Systems Engineering of the University of the Aegean after the first wave of coronavirus. It included questions concerning the degree of satisfaction of students with:

  • the technological resources provided them for digital distance learning: “The quality of the e-learning platforms”, “The response of the technical support to problems/malfunctions of the distance learning platforms” and “The information provided to students about the use of the various distance learning platforms”;

  • the human resources: the online theory teaching, and the online conduct of the laboratories (it should be mentioned that as mentioned above the context of our study was a Department of Information and Communication Systems Engineering, so the laboratories of the courses concerned programming and/or use of sophisticated software, therefore it was possible to be conducted online); and also “The consistency of the teachers and the response to their obligations”, “The encouragement of students by teachers for active participation in the courses” and “The academic secretariat of the Department”;

  • the educational outcomes: the level of understanding the online lectures as well as the ease of concentration on them, in comparison with the traditional face-to-face teaching; and also the extent of online participation in the online exams of the theory and in the laboratories.

Also, the questionnaire included in the beginning questions about some respondent’s demographic characteristics: gender, year of birth, year of study and previous use of video conferencing platforms before the start of this digital distance learning in the Covid-19 period.

We received valid questionnaires from a sample of 269 students. The sample consisted of 73.98% men, and the mean age was 23.21 (Std = 2.60) years, while the average of their year of study was 3.94 (Std = 2.52) years of study. Most of them (81.78%) had used a videoconferencing platform before.

Using these data we tested the above variables for normality using the Kolmogorov-Smirnov and Shapiro-Wilk normality tests; according to the results the hypothesis of normality of distribution is rejected for all variables. As a result, the non-parametric Mann-Whitney U test and the non-parametric Spearman correlation coefficients were used to investigate possible correlations among them.

4 Results

4.1 Descriptive Statistics

In Table 1 we can see the descriptive statistics (average and standard deviation) of the variables that measure students’ degree of satisfaction with the technological and human resources provided to them as part of the digital distance learning policies during the Covid-19 period.

Table 1. Descriptive statistics for the variables measuring students’ satisfaction with the technological and human resources provided to them during the Covid-19 period.
Full size table

We can see that the respondents are neutral to satisfied, however being closer to the latter, with the online theory teaching (Average = 3.65, Std = 1.03) and the online conduct of the laboratories (Average = 3.65, Std = 1.09) during the Covid-19 period. Therefore, students seem to be rather satisfied with the online conduct of the theory teaching and the laboratories, but only to some degree, and definitely not completely satisfied. The reasons for this can be identified by examining the average values with the next six variables shown in the same Table 1. We can see that the respondents are on average satisfied, or between neutral and satisfied, but closer to the latter, with “the response of the technical support to problems/malfunctions of the e-learning platforms” (Average = 4.10, Std = 0.93), “the information provided to the students on the use of the e-learning platforms” (Average = 3.75, Std = 0.96) and “the quality of the e-learning platforms (technical problems)” (Average = 3.68, Std = 0.99), which all concern the technological resources provided to the students as part of the digital distance learning policies during the Covid-19 period. On the contrary there is a lower degree of satisfaction, between neutral and satisfied but closer to the former, with “the encouragement of students by the teachers for active participation in the courses” (Average = 3.31, Std = 0.97), “the consistency of the teachers and the response to their obligations” (Average = 3.26, Std = 1.25) and “the academic secretariat of the department” (Average = 3.13, Std = 1.25), 54.65%), which all concern the human resources provided to the students as part of the digital distance learning policies during the Covid-19 period. These indicate the lack of experience and knowledge: a) on one hand of the teaching staff about effective methods and practices of online conduct of theory teaching and laboratories, as well as ways of motivating and encouraging the remote students to be more engaged, participate and not remain passive; and b) on the other hand of the staff of the academic secretariat about the administration of this online conduct of the courses and the effective communication with the remote students. On the contrary, the technical staff managed to provide a high-quality operation of the e-learning platforms, as well as of technical support of them concerning the management and recovery of their problems/malfunctions.

In Table 2 we can see the descriptive statistics of educational outcomes’ variables, which concern the level of understanding the online lectures as well as of the ease of concentration on them, in comparison with the traditional face-to-face teaching. We can see that on average there is disagreement to neutrality (however closer to the latter) to the statement “I can understand better the online lectures than the traditional face-to-face ones” (Average = 2.82, Std = 0.83); also, there is on average neutrality to the statement “It is easier for me to concentrate on the online lectures than in the traditional face-to-face ones” (Average = 2.99, Std = 1.55). These findings indicate that the participants perceive a slightly lower level of understanding the online lectures in comparison with the traditional face-to-face ones, and a similar level of concentration.

Table 2. Descriptive statistics for the understanding and concentration variables.
Full size table

In Table 3 we can see the descriptive statistics of. We can see a high to very high level of “Participation in online exams of theory” (Average = 4.57, Std = 0.83), as well as “Participation in online exams of laboratories” (Average = 4.22, Std = 1.21). This extensive participation reflects on one hand the need and pressure that the students feel to pass as many courses as possible during these difficult times, and not lag behind in their studies, but on the other hand it also reflects their perception that they had gained sufficient knowledge through the online teaching, so they had a good chance of passing the exams.

Table 3. Descriptive statistics for the participation in online exams’ variables.
Full size table

4.2 Effects of Demographic Characteristics

Next, we examined whether students’ satisfaction levels with online teaching and online laboratories conduct, as well as with overall online courses organization (measured through the average of remaining six variables that measure students’ satisfaction with these important aspects of the organization and technology of the online courses), is affected by gender. For this purpose, the non-parametric Mann-Whitney U-test was performed for a significance level of α = 0.05, and the results are shown in Table 4. It can be concluded that the gender of the respondent does not affect “Satisfaction with online theory teaching” (Z = −1.183, p = 0.237 > 0.05), “Satisfaction with online laboratories’ conduct” (Z = −1.106, p = 0.269 > 0.05) and “Satisfaction with online courses’ organization and technology” (Z = −1.513, p = 0.130 > 0.05).

Table 4. Results of non-parametric Mann-Whitney U-tests – effect of gender on satisfaction with online theory teaching, satisfaction with online laboratories’ conduct, satisfaction with online courses’ organization and technology.
Full size table

Accordingly, in order to investigate whether the above three satisfaction variables are affected by prior use of videoconferencing platforms (before the start of the digital distance learning in the Covid-19 period) we performed similar non-parametric Mann-Whitney U-tests, and their results are shown in Table 5. It can be concluded that the prior use of videoconferencing platforms does not affect “Satisfaction with theory teaching” (Z = −1.443, p = 0.149 > 0.05), “Satisfaction with laboratories” (Z = −1.608, p = 0.108 > 0.05) and “Satisfaction with courses’ organization and technology” (Z = −1.324, p = 0.185 > 0.05) (Table 4), for α = 0.05.

Table 5. Results of non-parametric Mann-Whitney U-tests – Effect of prior use of video conferencing platforms -on satisfaction with online theory teaching, satisfaction with online laboratories’ conduct, satisfaction with online courses’ organization and technology.
Full size table

Finally, we investigated whether these three satisfaction variables are affected by respondent’s age and year of study; for this purpose, we calculated Spearman’s Rho non-parametric correlation coefficients, using the significance level of α = 0.05, and the results are shown in Table 6. We can see that age has a statistically significant weak positive correlation with “Satisfaction with online theory teaching” (Rho = 0.200, p = 0.001 < 0.05) and “Satisfaction with online laboratories’ conduct” (Rho = 0.149, p = 0.017 < 0.05); at the same time, there is no statistically significant correlation with the “Satisfaction with courses’ organization and technology” (Rho = 0.087, p = 0.154 > 0.05). With respect to the year of study, it shows a statistically significant weak positive correlation with “Satisfaction with online theory teaching” (Rho = 0.178, p = 0.004 < 0.05) and “Satisfaction with laboratories” (Rho = 0.139, p = 0.027 < 0.05), while there is no statistically significant correlation with “Satisfaction with online courses’ organization and technology” (Rho = 0.076, p = 0.213 > 0.05). Therefore, it is concluded that as age and year of study increase, satisfaction with online theory teaching and online laboratories’ conduct tends to increase weakly, probably because of the increasing maturity and overall ability of students to cope with the inherent disadvantages of distance education (lack of co-location of learners and instructors); on the contrary, the satisfaction with the organization and technology of the online courses is not affected by age and year of study.

Table 6. Results of non-parametric Spearman’s Rho calculations of age and year of study with satisfaction with online theory teaching, satisfaction with online laboratories’ conduct, satisfaction with online courses’ organization and technology.
Full size table

The same analysis was made for the educational outcomes’ variables that measure the perceived level of understanding of the online lectures as well as of ease of concentration on them, in comparison with the traditional face-to-face teaching. We found than male students have a higher level of both in comparison with female students; also, both have weak statistically significant correlation with age and year of study, but no statistically significant correlation with the prior use of videoconferencing platforms.

4.3 Relationships Between the Policy Outputs and Outcomes

Finally, we investigated the effects of the examined policy outputs (technological and human resources) on the policy outcomes: a) on the average of the perceived level of understanding the online lectures and the perceived level of ease of concentration on them, in comparison with the traditional face-to-face teaching, and b) the average of the degrees of participation in the online theory exams and in the online laboratories’ exams. For this purpose, we calculated Spearman’s Rho non-parametric correlation coefficients of these two average variables with the abovementioned eight policy output variables; the results are shown below in Tables 7 and 8 (the statistically significant values are shown in bold).

Table 7. Spearman’s Rho non-parametric correlation coefficients of the average level of understanding the online courses and concentrating on them with the policy output variables
Full size table

We can see that all the examined policy outputs (technological and human resources), with the only exception of the academic secretariat of the Department) have positive effects on the average of the perceived level of understanding the online lectures and the perceived level of ease of concentration on them, in comparison with the traditional face-to-face teaching (level of educational outcomes). The quality of (degree of students’ satisfaction with) the online theory teaching and the online laboratories conduct have the strongest effects, followed by the level of encouragement of students by teachers for active participation in the courses; this indicates that the quality of the human resources provided to the students during the Covid-19 period (= the ‘e-instruction’, meant as distant teaching by instructors though the e-learning platform) affects most the level of the educational outcomes of these digital education policies in higher education.

Table 8. Spearman’s Rho non-parametric correlation coefficients of the average degree of participation in the theory and laboratories exams variable with the policy output variables
Full size table

We can see that the response of the technical support to problems/malfunctions of the e-learning platforms, the quality of (degree of students’ satisfaction with) the online laboratories conduct and the quality of the e-learning platforms have the strongest positives effects on the extent of students’ participation in the online exams, followed by the quality (degree of students’ satisfaction with) the online theory teaching as well as the information that had been provided to them about the use of the e-learning platforms.

5 Conclusion

The Greek government rapidly after the outbreak of the Covid-19 pandemic, in order to mitigate its spread, adopted a central policy for all universities of shutdown, and at the same time of continuing their educational activities using digital distance learning technologies and methods. In the previous sections has been presented an evaluation of these government policies; it has been based on public policy evaluation theory, which distinguishes between direct policy outputs and policy outcomes. In this direction have been evaluated both the direct outputs of these policies (i.e. the educational resources, both technological and human ones, provided to the students), and its educational outcomes (level of understanding the online lectures as well as of concentration on them, in comparison with the traditional face-to-face teaching, and also extent of online participation in the online exams). Evaluation data have been collected through a survey of 269 undergraduate students of the Department of Information and Communication Systems Engineering of the University of Aegean.

The results of the analysis of the data we collected indicated a moderate to good level of success of these policies with respect to the educational resources provided to the students during the Covid-19 period, which however was not a complete success; this difficult (but absolutely necessary) undertaking worked to some extent, but had also some important weaknesses, which concerned “the encouragement of students by the teachers for active participation in the courses”, “the consistency of the teachers and the response to their obligations” and “the academic secretariat of the department”. These weaknesses reflect the lack of experience and knowledge of the teaching staff about effective methods and practices of online conduct of theory teaching and laboratories, as well as ways of motivating and encouraging the remote students to be more engaged, participate and not remain passive. Also, they reflect the lack of experience and knowledge of the administrative staff of the academic secretariat about the administration of this online conduct of the courses and the effective communication with the remote students.

Therefore, since as mentioned in the Introduction digital distance learning is expected to be used extensively in the future in higher education (e.g. for postgraduate programs and continuous education, for providing shared courses attended by students of several geographically remote departments), it is necessary to provide sufficient training to the teaching staff about the effective conduct of digital distance learning, and especially the motivation, encouragement and engagement of the remote students. Also, it is necessary to provide sufficient training to the administrative staff about the effective administration of this digital distance learning, in which the students are not physically present in the same geographical location.

With respect to the educational outcomes the results indicate a satisfactory level of success of these policies: the participants perceive a slightly lower level of understanding the online lectures in comparison with the traditional face-to-face ones, and a similar level of concentration. Furthermore, there has been extensive participation in the most critical element of these online courses, their online examinations of the theory and the laboratories of the courses, which probably reflects a positive students’ perception about their learning from this digital conduct of the courses: their feeling that they had gained sufficient knowledge through the online teaching for having a good chance of being successful in the exams.

Our study has two main limitations. The first limitation is that it is dealing only with the learners’ perspective, but not with the instructors’ perspective (who are more experienced, so they can provide a more substantial and in-depth evaluation of these digital distance learning policies in higher education during the Covid-19 period, and identification of weaknesses that have to be addressed). So, it is necessary to conduct similar evaluation research in the future from the instructors’ perspective, using both quantitative methods (questionnaire-based) as well as qualitative ones (e.g. based on interviews and focus-groups). The second limitation is that our study has been based on a survey of undergraduate students of a Department of Information and Communication Systems Engineering, whose instructors and students are quite familiar with the use of digital technologies, and also its laboratories can be conducted online. So, it is necessary to conduct similar evaluation research in other types of departments, in which instructors and students have lower familiarity with the use of digital technologies, and also it is not possible to conduct the laboratories online (e.g. this might need more sophisticated simulation approaches).