Keywords

1 Introduction

March 2020 will remain as a global landmark date that established the first global health crisis that would change the world as we knew it and determined a mandatory confinement worldwide as a measure to contain the spread of the COVID19 disease. The situation determined that all education shifted to Emergency Remote Learning/Instruction/Teaching. As Hodges et al. [1] defined, ERT consists as a “temporary shift of instructional delivery to an alternate delivery mode due to crisis circumstances”. The process “involves the use of fully remote teaching solutions for instruction or education that would, otherwise, be delivered face-to-face” and “return[ing] to that format once the crisis or emergency” is over. As such, the main goal of ERT consists of providing temporary access to instruction and instructional support, in synchronous and asynchronous formats, in reliable ways during a particular moment and is essentially a shift of delivery modes, methods, and media which are progressively adjusted to the available settings, resources, and limitations of organizations, teachers and students [2]. As the situation called for immediate actions, HEI focused on ensuring the continuity of the provision of the services for whatever means available, setting up an operational instructional approach. This emergency approach revealed some level of unawareness of the impacts of the situation on the students’ wellbeing and mental health, which can be understood given the urgency of the situation.

As the confinement period extended, people began to realize the potential risks for the wellbeing, namely mental health, of the confinement and other sanitary measures, could have on the population in general and on the HE students in particular. That was a time when a significant number of publications emerged presenting multiple Covid19-related research topics, from all over the world, with a special focus on the mental health of individuals and families [3,4,5,6,7,8,9,10,11]. Another stream of research focused on understanding the overall sentiment of the population through the analysis of emoji posts following news publication especially on social media platforms [12].

In Portugal, the mandatory confinement and the closure of schools occurred from the March 12th, 2020, until the end of the school year, for basic and secondary education cycles and determined the need to implement emergency responses which, in the absence of contingency plans, pushed organizations and teachers to focus on easy-to-implement operational solutions that facilitate the continuing of the delivery of instructional contents and support to students, as it happened all over the world. After a thorough analysis of the situation and being conscious of the multiple potential impacts that could emerge from it, we performed a research that aimed to evaluate the impacts of the confinement situation and remote learning effects on HE students’ experience, exploring multilevel repercussions of the situation. For that purpose, we developed a six-dimension model to assess the student’s perceptions, focusing on the educational/pedagogical, technological, and working conditions, social, family-related, psychological, and financial issues. Results obtained reveal that the most impactful problems focus on the pedagogical and psychological domains [2].

Further ahead, we used the CIPP (Context, Input, Process, and Product) model to establish a 46 variable analysis based on the students’ responses which confirmed the crucial importance of motivation and engagement in online classes as learning enablers or constrainers. These also shape the students’ perception of the role that online classes play in helping them to stay more positive during ERL [13].

The second mandatory confinement in Portugal started on January 21st, 2021, being gradually reopened as soon as the sanitary conditions allowed. On March 15th, the 1st cycle of basic education returned to face-to-face classes; the 2nd cycle of basic education and the secondary cycle returned to in-person classes on April 5th, 2021. HEI partially returned to having in-person classes on April 19th, 2021, but most of the HE institutions decided that only part of the classes returned to fully face-to-face.

This 2nd confinement determined the return, once again, to remote learning/teaching/instruction, and it became imperative that we reassessed the students’ perceptions regarding the context, input, and process elements of the CIPP Model.

As such, the present research aims to follow the evolution of the remote learning situation in Portugal, as by providing the follow up of the situation will allow to confront the results obtained during the 1st confinement (May 2020) and the ones obtained during the 2nd confinement (May 2021), figuring out the potential positive and negative evolution of the multidomain assessment, enabling an in-depth reflection and drawing conclusions about the adaptation curve.

The basic underlying reasoning that shaped the former and present research is that the learning/teaching/instruction process is complex and multidimensional, consisting of interrelations that need to be understood and explained in order to assess the conditions that influence students’ performance and wellbeing. School closures and subsequent confinements due to the COVID-19 outbreak have affected 87% of the world’s students physically, socially, and psychologically [14, 15], yet the consequences, namely social distancing and isolation, the lack of interaction with peers, and with extended family, the stress caused by news about the disease and panic actions or limit situations abundantly broadcasted, and the lack of adequate support from families, from community and HEI, among other reasons, caused a wide range of psychological disturbances on students [16, 17].

At this point, our purpose was to combine methodologically the structure of data collection used in our previous research consisting of a six-dimension model of issues that may impact ERL covering the fields of educational and organizational issues; technological and working conditions; social, family-related; psychological and financial issues framed in the CIPP model.

In the next section, we present some of the basic underpinnings of the CIPP Model while mostly remitting to our previous studies and move on to presenting our findings and discussion of the results.

2 The CIPP Model

In this paper, as in previous research [13], we use the CIPP model – Context, Input, Process, and Product evaluations [18] to reflect on the circumstances and impacting Emergency Remote Learning. We have focused on analyzing critical aspects contained in the CIPP model’s context, input, and process elements, leaving the product element out, according to what is recommended by Hodges [1] for remote learning settings. We make some final remarks regarding the product element as a result of the comparison between the first and second confinements, highlighting future institutional interventions that we consider relevant.

The context assesses needs, problems, assets, and opportunities, as well as relevant contextual conditions and dynamics (e.g., institutional, social, financial, and governmental aspects). The input evaluation focuses on how it should be done and assesses competing strategies, and the work plans and budgets of the selected approach (e.g., technology infrastructure, software, faculty support, faculty professional development, learning resources such as access to libraries). The process evaluation considers if the program is being done and consists of monitoring, documenting, assessing, and reporting on the implementation of plans (e.g., how processes can be adapted to this new reality, quality of teaching, and learning). Finally, the product evaluation sees if the program succeeded and it identifies and assesses costs and outcomes – intended and unintended short term and long term (e.g., course completion rates, aggregated grade analyses, feedback).

3 Methods and Procedures

The adopted methodology consists of comparative survey-based research. We build on previous exploratory research, which was characterized by employing a single data collection method to obtain an initial view of the key issues in the CIPP Model. Provided the scarceness of literature and systematization on the socio-educational impacts of the current confinement period on student’s lives and on how they are responding to implemented ERL solutions, we proposed an evaluation instrument [2], comprised of 67 items organized in six dimensions of issues (educational and organizational, technological and working conditions, social, family-related, psychological, and financial issues) matched with the CIPP Model elements context, input, and process [13].

The survey was disseminated among students enrolled in HEI in Northern Portugal, through social media channels, namely in institutional public pages and open groups in May 2020 (student group 1) and in May 2021 (student group 2). Since the questionnaires were anonymous, it was not possible to track which students have answered both surveys. However, given that the dissemination strategy and channels were the same for both moments, we assume that the likelihood of having students answering both surveys is high. The items on the scale were anchored at 1 = never, 2 = rarely, 3 = frequently, and 4 = always. The survey data were downloaded and transferred into IBM SPSS Statistics 26.0 statistical analysis software package for analysis. Higher scores represent the higher frequency of the item.

4 Results

The sample demographics for both periods are presented in Table 1. The entire sample is composed of 762 answers to the questionnaire, divided into two groups. The first group of students (G1) answered the questionnaire in May 2020 (n = 359), and the second group (G2) answered it in May 2021 (n = 403).

Table 1. Sample demographics (N = 762)
Full size table

Regarding the commonalities among both groups (χ2 test), the majority of respondents are female (>70%), more than 31% received a financial aid scholarship, they have unlimited access to the Internet (≈78%), and more than 94% were attending online classes in May (p = n.s.). There are, however, some statistically significant differences between G1 and G1. Most of the students are aged 18–22 years old in both groups, but students from G2 are, on average, 1.5 years older (t(760) = 4.016; p < .001). Also, in both groups, most are non-working students, but this percentage is 9.3% higher in G2 (χ2(2) = 8.718;p < .05). The percentage of students that were compelled to return home (family residence) during ERL in G2 is about half (21.1%) of that recorded for G1 (41.8%) (χ2(2) = 38.624;p < .001). Similarly, and expectedly, the number of students attending online for the first time has drastically decreased from 91.9% in G1 to 43.2% in G2 (χ2(2) = 201.678;p < .001), although it is still substantial.

In the following sections, we present the sets of unchanged, improved, and worsened issues between the two groups regarding the CIPP model’s context, input, and process elements, considering only the students attending online learning (n = 731). We focus on statistically significant equal/unequal variances.

4.1 Persistent Aspects

We identified seventeen persistent aspects (equal variances) among the two groups (Levene’s test > .05) regarding the context, input, and process elements of the CIPP Model, presented in Table 2.

Table 2. Unchanged issues (n = 731)
Full size table

Regarding the family context, both groups report frequently having a proper workstation with the ideal conditions to participate in online classes in a favorable environment. In both groups, students also report frequently trying to keep a work/study routine while rarely being able to save money on school supplies. There are slight decreases in the averages of G2 regarding the wellbeing aspects; however, in both groups, students report frequent sleep disorders and rarely having more time for hobbies and exercise.

Concerning the remote learning environment, students keep feeling rarely motivated to participate (with a slight decrease in G2) and frequently more tired (with a slight increase in G2) in online classes. The access to a greater amount and diversity of educational resources has a slight increase in G2 but with a moderate frequency in both groups. Students kept rarely making changes to their assessment regime, frequently participating in online evaluation moments, and, on average, there were still some changes in the evaluation components of courses that were less beneficial to them.

In the process elements, there is a slight decrease in G2 concerning the frequency of support/interaction with teachers, but both groups reveal that teachers frequently care about their personal wellbeing and send clear information regarding the functioning of online classes. Likewise, with a slight decrease in G2, both groups report frequently keeping in contact with their school friends/colleagues.

4.2 Improved Aspects

We identified ten aspects in which the variation of averages, even if slight, among the two groups (Levene’s test <.05) allowed us to detect improvements statistically significant in G2, depicted in Table 3.

Table 3. Improved issues (n = 731)
Full size table

The limitations felt in the family environment based on general disturbances and the need to share the workspace, and the computer decreased in G2. Students in this group also report a higher frequency in having the necessary equipment to attend online classes. Regarding their financial context, the students in G2 are slightly less concerned about the worsening of their financial situation, which also results in a less frequent notion that their permanence in higher education is risked due to financial struggles. In previous research, we have detected that students were frequently overwhelmed with work derived from online classes. This has also decreased in G2, despite still rarely being the same as in face-to-face classes. The teachers also shared information about the functioning of online classes more frequently, denoting increased communication in G2. For this group, there was also a decrease in the frequency in the possibility to pursue a continuous assessment method, which students tend to prefer.

Pertaining to social interactions, students in G2 are even less preoccupied about their privacy in online classes - despite several accounts of students unwillingness to turn on their video cameras or microphones during videoconferencing-based classes, namely in Zoom meetings [19, 20] - and, although they miss their school friends frequently, this feeling has also decreased in G2.

4.3 Worsened Aspects

We identified ten aspects in which the variation of averages, even if slight, among the two groups (Levene’s test <.05) allowed us to detect the worsening issues with statistical significance in G2, depicted in Table 4.

Table 4. Worsened issues (n = 731)
Full size table

Referring to the contextual element, in G2, students report an increased frequency of need to buy equipment/devices to participate in online classes. This group also reports a lower frequency of saving money in transportation, which might be linked to an increase in mobility as confinement measures are less strict. In both groups, students indicate almost never receiving psychological and moral support from the school, and this has worsened for G2.

In the input element that refers to the learning environment and learning resources, both groups indicate frequently participating in interactive classes, but the frequency is lower for G2. The potential of online classes to keep students more positive during confinements is rarely encountered in both groups and has also decreased for G2.

In the process element, students identify a less frequent commitment of the school board to their welfare and academic success in G2, despite being nearly frequent for both groups. The second group of students also feels less frequently closer to their teachers. This is also felt in other relationships. Despite rarely feeling a worsening in the relationships with their school friends (both groups), their frequency is visibly increased among G2 students. There is also a decrease, in G2, in the frequency of belief that one’s wellbeing depends on keeping in contact with one’s school friends/colleagues. We consider this to be a worsened aspect because the school environment is primarily a social space. This might, however, result from the maintenance of contacts with school friends, which remains unchanged, as identified in Table 2.

5 Discussion and Conclusion

According to the statistical significance established criteria, we found a set of 17 aspects that remaining the same, the second set of 10 items revealing improvements and a third set of 10 worsened aspects.

While the improvements (Table 3) reside mainly in contextual aspects, such as the family dynamics, working conditions, and financial safety, the worsening (Table 4) is mostly felt in the input and process elements’ issues which concern to the remote learning environment and social interactions with teacher and peers. However, the set of persistent (unchanged) issues (Table 2) reveals a scenario more prone to the perpetuation of the problems/concerns detected in May 2020 than to a continuation of optimistic circumstances. The exception is relying, once more, on the contextual aspects, such as the family dynamics, working conditions, and financial safety, but not on the contextual aspects regarding the students’ wellbeing.

Our analysis reveals the persistence of frequent sleep disorders and rarely having time for hobbies or working out (contextual), together with a persistent lack of motivation to engage in remote learning and persistent exhaustion (input). This, according to MacMahon, Leggett [21], might actually be a two-way problem. Teachers’ ability to notice verbal and nonverbal behavior and interactions of students is reduced in remote learning. These social behaviors give vital insight into student knowledge, engagement, emotions, and motivation. Students, on the other hand, may feel socially isolated, cut off from their friends and classmates, who are crucial in developing good affect, motivation, and learning control.

This also reinforces the global awareness of the current physical, social, and psychological problems being faced by students, as noted by Tang, Xiang [14], and teachers [22, 23]. In fact, among the set of worsened aspects, it is possible to observe a significant reduction of the online classes’ potential to help students stay more positive during confinement, as well as on the classes’ ability to offer participative and interactive moments. This has also resulted in students feeling less close to their teachers and believing that the school board is less committed to their academic success, despite frequently reporting (and persistently) sensing that teachers care about their wellbeing and receiving clear information from the teachers and school regarding the functioning of online classes.

There is no doubt that the social dimension was heavily affected. Although students report a small reduction of the worsening of relationships with peers, the frequency in which this occurs is still relevant, particularly considering the decreased perception that one’s wellbeing depends on social interactions within the school. We believe that it could indicate that students may be adjusting to a lifestyle in which they accommodate to isolation and undervalue social interactions, which, in turn, may worsen their psychological wellbeing and their role in the social ecosystem of the school and society. It may also signify that the notion of wellbeing has been altered, and this type of contact has been replaced with something else or that isolation is becoming increasingly comfortable.

Our analysis has allowed us to understand that the socialization potential of remote learning has been undoubtedly insufficient and to reflect upon several forces pushing its depression: the need to comply with the established curricula leaving little space for pure socialization, the overall lack of teacher’s preparedness to instantly adjust to and manage remote instruction, the lack of institutional readiness and infrastructures, the students reluctancy to engage in video conferencing classes and/or to acquire the necessary autonomy to manage self-paced learning, to name a few. However, it should be noted that, according to our data, the main difficulties reside less on the contextual aspects that are somewhat controllable by the students (family, working conditions, and income) than on the aspects that pertain to the domains in which education institutions are expected to have an active intervention. For instance, one of the most aggravating aspects that we identified reside in the reduction of the psychological and moral support provided to students, either in the form of support emails or psychology appointments.

The lack of guidelines for implementing distance learning, lack of infrastructure, lack of competencies, and security-related problems were some of the challenges met during the pandemic by the vast majority of education institutions. As a consequence, organizational models for remote learning are now being proposed in the literature [24], in order to prevent and minimize future problems. This, however, should not exclude the need to develop additional measures to compensate and minimize the long-term problems and effects that are and will be felt as a consequence of the first and second confinements, as, for instance, there might be a long way in recovering students’ psychological wellbeing, motivation, intrapersonal skills and their trust in the education system.

The authors acknowledge that this work is not without limitations. Although we clearly state that we present a comparative study, since the surveys are anonymous, there is no way to determine how many students have answered both surveys and how their specific circumstances have changed.

As future work we suggest that a similar study takes place at the end of the first semester of the next academic year (2021–2022. At that time, supposedly, classes will be face-to-face again, and it would be interesting to understand if the more problematic situations described in this paper were improved or not.