Keywords

1 Introduction

The evaluation process of School Libraries, developed by the Portuguese School Libraries Network (SLN – RBE, in Portuguese language), has been carried out through the application of a self-evaluation model (MABE). This evaluation seeks to establish a relationship between processes, the impact or value they generate, and the prospects for future development [1]. The model contemplates four domains, which constitute parts of analysis and evaluation in harmony with the main axes of work of the School Library (SL). The instrument used by Teacher Librarians (TL) is based on a set of elements, including the collection of data through questionnaires applied to students of different schooling levels. In Portugal, TL are regular teachers who, in their great majority, have school librarianship qualifications to perform their SL duties.

Therefore, in the framework of an agreement for research purposes between Universidade Aberta (UAb) and the SLN, we examine the responses obtained through the questionnaires carried out in that context, to students from the Portuguese districts of Évora, Leiria, Setúbal and Vila Real, in 2015, 2017 and 2019 (questionnaires were not applied in 2021 due to COVID19 pandemic). The analysis is focused on activities related to MIL, considering that primarily it concerns competencies and skills, but can also be seen as the basis for a democratic society in which all citizens can participate [2]. According to the orientations published by SLN [3], students need to critically engage with media and information, namely in digital settings, and it is of the utmost importance to identify how students evaluate SL’s action and to realize whether the SL is achieving its goals in this area.

This study is based on a mixed methods approach and crosses the four regions (representative of different geographic and social contexts) and the three indicated time periods, verifying the incidences and contrasts in the dimensions studied in the observed universe. We want to understand how students face the contributions of the school library in the development of their skills in information and media literacy by means of a statistical analysis of the answers given to MABE questionnaires. These data are compared with the SL self-evaluation Report of 2019, specifically the strengths and weaknesses pointed out by the TLs. From this analysis it is possible to identify what has been the role of SL in the development of MIL skills, as well as the constraints and improvements in the action of libraries. We think that SLs are right at the centre of this process, and that schools have an important role to play in this context, as MIL is widely considered the answer to today's information crisis. This study is particularly relevant as it analyses unpublished data collected by SLN, especially considering that the SL evaluation model has become an instrument to guide good practices, monitoring and evaluating the different intervention areas, certifying processes and results, and informing decision-making.

2 Conceptual Background

2.1 Relevant National and International Media Literacy Documents

SL are important partners in the educational process by contributing to the development of students’ skills that will be fundamental for their future professional life and for their training and personal development as critical and active citizens. The action of these educational spaces, which integrates multiple literacies in the areas of reading, media, and information, should consider a set of reference documents that may guide the work developed. We focus on some of these documents, according to the objectives of this article.

The guide “Learning with the School Library” [3] is a key document in terms of the orientation of the work that is intended to be carried out and dynamized by the SLN in schools, with teachers and students. It points to the three areas of literacy that constitute important cores of the libraries’ activity: reading, information, and media literacy. In the last two areas, it is intended that media and information literacy practices may “endow the students with knowledge necessary for their creative and informed use” [3].

The directives of the above-mentioned guide are articulated with another document published by the Portuguese Ministry of Education, “The profile of students leaving compulsory education” [4], which distinguishes ten areas of competences. The areas of “Information and Communication”, and its associated competences, which refer to information and media literacy, through the “selection, analysis, production and dissemination of products, experiences and knowledge, in different formats” [4] are the most relevant for the present work.

At the European level, it is worth mentioning the “Digital Competence Framework for Educators” (DigCompEdu). The DigCompEdu framework [5], being oriented to educators at all levels of education, from early childhood to higher education, including general and vocational education, special education, and non-formal learning contexts, proposes 22 competences and a progression model to help educators assess and develop students’ digital competence.

2.2 The SL Evaluation Model (MABE)

Since the establishment of the School Libraries Network, its coordinating office has been developing several actions aimed at consolidating the work of the school library and highlighting its relevance in the context of Portuguese schools. Among the initiatives that have been designed over more than 25 years of deepening the work of SLN, the introduction of a self-assessment model for libraries stands out. This model - MABE – School Library Assessment Model - currently being applied in all schools of the 2nd and 3rd elementary cycles and secondary education and in some schools of the 1st basic cycle has constituted a management benchmark and an instrument for guidance and internal improvement.

After an experimental phase, in 2008–2009, the model was widely applied from 2010–2011 onwards. This self-assessment instrument has already had three versions, the 2010, 2014 and the current one, published in 2018, which “consolidates the objectives set out in the previous edition, reinforcing the adequacy of contents and methodologies to the changes that school libraries have been experiencing.” [6].

It is important to emphasize that the model was conceived with an orientation towards the analysis of the processes that contribute to realize the impact of SL on teaching and learning. The resources and activities carried out at SL are important considering their added value for the educational process and for the development of students’ skills. Concepts such as value, impact and evidence-based practice were at the basis of how the self-assessment model was designed and built [7]. A model such as this allows for a better understanding of how the mission and objectives established for the school library are being achieved, by analysing the practices and actions required for its improvement. In this sense, this self-assessment model is also an example of an action-research process, as it seeks to establish a relationship between processes, the impact or value they generate, and future development prospects [1]. Therefore, SL can, autonomously, carry out an evaluation of their actions and define strategies for improving and developing their practices. This instrument is intended to guide SL's self-assessment, and is based on a set of elements, including data collection, in the form of a questionnaire. MABE contemplates four domains, each divided into two subdomains which constitute parts of analysis and evaluation in harmony with the main SL work axes.

It is the domain A. “Curriculum, literacies and learning”, that points to action in information and ​​media literacy, specifically in the subdomain: A2 - “Training for information and media literacy”. The present reflection was based on this axis of work. It is also important to add that these training activities are normally carried out by TL with the groups of students who go to the library, being included in the SL activities programme. Sometimes there is some collaborative work with classroom teachers. In any case, the fact that this work is always carried out in the library makes its role more evident, namely when students are invited to evaluate the SL’ work.

3 Methodology

In this study, qualitative and quantitative procedures were used, in a perspective of mutual enrichment, and as an attempt to understand more broadly the object of analysis. First, we intended to capture and analyse the ways the SL actions in MIL are seen by students, so we fundamentally favour the perspectives of the research subjects [8]. In a second section of the analysis, we confront the reports of TL to examine the way some evaluation results, in particular data provided by students, are referenced in the final reports.

The analysed data was collected in a questionnaire that is part of the SL evaluation model. As part of this evaluation process, students are surveyed at the end of the school year, every two years. In general, students fill out the questionnaires online, in the school library. Younger students can be helped by a teacher to understand the questions. It is a large-scale survey, applied in all schools belonging to the SLN, to a representative sample of students from different school years (minimum of 20% of students from each school). It is, therefore, a survey capable of capturing the perspectives of the whole country (mainland Portugal) on the questioned topics.

Our analysis focuses on comparing the results of the questionnaires referring to students in the 1st cycle of basic education (from 1st to 4th grade, that is, ages between 6 and 9 years old, but the surveys are only applied to the 3rd and 4th years) and the 2nd and 3rd cycles of basic education (5th to 9th grade).

The students’ questionnaire covers the four domains that make up the assessment model (cf. 2.1) and has 11 questions (question 11 is an open question). The MIL area is present in items from question 7 and question 10, and those are the items to be examined in the present article. In question 7 the answers are Yes/No, and in question 10 a Likert-type evaluation scale is used. We follow a standardized quantitative approach, with a descriptive statistical analysis. The students’ responses to the questionnaire were studied by calculating the averages.

The TL Improvement Reports (2019) were confronted in the items also related to the MIL area. The responses presented in the SWOT analysis and global assessment attributed to the following 3 items of the report were analysed:

  1. (i)

    Development of students’ abilities in the use of technologies in an educational context.

  2. (ii)

    Increase of students’ skills in the personal use and management of information.

  3. (iii)

    Changing students’ attitudes towards the critical use of information and media.

4 Results and Discussion

Considering the imposed limitations of article length, we decided to present the results globally, for the four regions examined. After analysing these global results, the most significant aspects that were identified in the four regions are highlighted, by comparison with the average values achieved. In this way, we seek to highlight the most relevant deviations from these values. It should be noted that the four regions are not homogeneous. For example, they have significant variations in the number of schools and students, and this aspect has an influence on the results taken globally.

The results reveal that there are no significant changes over the three evaluation cycles. The most relevant differences concern the activities that have a higher incidence in each of the levels of schooling analysed.

Thus, it appears that with younger students (1st cycle of basic education) the work of the SL in information literacy focuses mainly on information research, which has the most expressive positive values. By contrast, an activity in IL that is fundamental, as is the case of information evaluation, has low achievement values. This question does not appear in 2019, following an update of the questionnaires that are applied by the SLN. It seems to us, however, that this more specific dimension of IL work should be included. It could be considered that younger students may not have a clear notion of the elements involved in this type of activity, also considering the high percentage of non-responses, but comparing with the answers given by older students, who should have a clearer notion of the difference between the two actions - information research and information evaluation -, it is also verified that about half of the respondents affirm that the SL did not provide them with information evaluation activities. This could be considered a weakness in the library's action, judging by the perception that students have. These weaknesses must be taken into account, as several studies report the difficulties that young people reveal when researching and evaluating information sources [9, 10]. To support critical student inquiry skills, it is necessary, in the first place, to create conditions for carrying out activities in this area, which seems to be lacking in many libraries.

Regarding digital literacy, and from the perspective of students as producers of digital artifacts, it appears that in the 1st cycle library work involves few students in this type of activity, although there is a slight positive evolution in 2019. In the remaining school cycles students’ perception is more positive, but there is still a considerable percentage of students who say they did not participate in activities that put them in an active attitude towards the creative use of digital resources.

One of the areas in which there has been a greater commitment in Portugal, and in which libraries have been involved, concerns the development of skills in the safe use of the internet and social networks [11]. There is a slight appreciation of this type of action in 1st cycle schools, approaching in 2019 the values mentioned by students from 2nd and 3rd cycle schools.

We can compare these data with the answers given by students to another question in the questionnaire (question 10), in which students are asked to evaluate the contribution of the library in several of the learning activities carried out at school (for example, in the development of interest and skills in reading). Item 3 specifically focuses on the area of media and information literacy (Table 2).

In the case of this question, students in both groups evaluate the contribution of the library in this area in a very similar way, with 1st cycle students choosing a slightly higher percentage for the classification “Very good”. But it is also in this group that the most negative evaluations are found regarding the contribution of the library to their learning in MIL, with percentages above 10% located in the “Weak” classification, a value that decreases slightly in 2019 (9%). In comparative terms, it appears that the assessments of students in the 2nd and 3rd cycles are more positive (sum of evaluations “Very Good” and “Good”), which is consistent with the answers given in question 7 (Table 1).

Table 1. Results from answers to question 7.
Full size table
Table 2. Results of answers to question 10, item 3.
Full size table

Looking separately at each of the territories selected for analysis, and although there are significant differences in terms of the number of schools in each of the four regions (especially at the 1st cycle level), we identify some deviations from the global panorama, which are worth mentioning. We will discuss here the results from schools of the 2nd and 3rd cycles, in 2019, since, as mentioned, there are no major fluctuations between the results obtained in the three evaluation cycles. In the two school cycles that we consider in this part of the analysis, students also show greater maturity and analytical capacity, so their opinions will be more solid. Let us then comparatively analyse the four selected regions.

Regarding the parameters verified by question 7, items 1, 2 and 4 (Table 3), Évora (10 schools: 634 students responding) is the region that largely presents the best results, with percentages of answers in the option “Yes” above the overall average. It is the territory that shows a more effective work by the library in the MIL area, including in item 2, which focuses on the development of skills in the use of digital resources to produce various artifacts. Leiria (23 schools: 1709 students) and Vila Real (8 schools: 491 students) have all values below the global average and Setúbal (39 schools: 3114 students) have values below the global average in item 4.

Table 3. Comparative results among the 4 regions (2019).
Full size table

Naturally, given the characteristics of each of these regions and the number of schools examined in each territory, a more focused analysis will be important, which it is not possible to present in this article. For example, of the 10 schools belonging to the region of Évora, a school in a small town in the rural interior stands out, which presents negative values in the 3 questions, in sharp contrast with the general panorama of the district. This type of situation requires investigating the reasons for this discrepancy which, we add, also extends to previous evaluation cycles (2015 and 2017).

On the other hand, a densely populated region such as Setúbal (a suburban district, located south of the capital, Lisbon) and, therefore, with a high number of schools, presents a lot of contrasts, with schools where the number of students involved in activities within the scope of MIL is quite high, and others that reveal low rates, with a significant number of students who do not participate in MIL related activities. It should be noted that this region is still the one with the highest percentage of students who do not answer the questions, above the global average.

The district of Vila Real, located in the north of the country, which in the 3 questions analysed presents values below the average, is curiously the region in which the SL is evaluated more positively by the students, in question 10. 44% of the students rate the contribution of their library to their learning in MIL with “Very good”. Setúbal, which, as we mentioned, is the region where the most contrasts are found, presents the highest percentage of responses that assess the contribution of the library in the MIL area as “Medium”: 37% of students chose this option (global average of the 4 regions: 14%). It was also intended to compare the students’ responses with the Improvement Reports made by teacher librarians. These reports follow a model and are based on the analysis that TL makes of the results obtained in the questionnaires (to students and teachers). TL should also do a SWOT analysis for each of the four areas that are being assessed. We chose to examine the reports of the schools that in 2019, and in each region, had the best and worst evaluation by students. Our aim was to understand how the results of the student questionnaires were reflected in these reports.

In the Évora region, the report of a school (A) located in the capital city of the district (school with the best results in the questionnaire) and a school (B) located in a small town in the countryside (school with the worst results in the questionnaire) were analysed. TL should use a scale of 1 to 4 to do the library self-assessment. In items more directly related to MIL skills, despite the significant differences in the results of the student survey, the self-assessment report of each of these schools presents the same appreciation:

  1. (i)

    evelopment of students' abilities in the use of technologies in an educational context (4 in both school reports).

  2. (ii)

    Increase of students' skills in the use and management of personal and school information (3 in both school reports).

  3. (iii)

    Changing students' attitudes towards the critical use of information and media (3 in both school reports).

The same exam was carried out for the remaining regions, and it was verified in the Leiria district that the two SLs selected according to the same criteria, after all, presented the same evaluation for the MIL area: evaluation of 3 for each of the three items. SL A, with the best scores in the evaluation of students, presents in its report, in the strengths, a detailed description of some actions that the SL develops, highlighting the mention of the use of the Big 6 research guide and the availability of several tutorials and the realization of training for the use of digital tools. As negative points, the SL points out the weakness of the computer equipment. SL B, located in a region with a fishing tradition, has a high rate of non-responses, which negatively influences the results of the assessment made by the students. For example, no student evaluates with “Very Good” the impact of the SL action on their learning in MIL.

In the district of Vila Real, an even more paradoxical situation is found. In fact, in this case, the library whose work is perceived in a less positive way by students (SL B), is the one whose TL assigns the maximum score (4) in the three items related to MIL that are proposed for evaluation in the report. The library with the best appreciation by students (SL A) has a 3–3-4 rating in its report. It is necessary to read the aspects mentioned in the SWOT analysis to better understand the context of each of the libraries. SL B points out several negative aspects, which somewhat contradict the assessment presented: for example, “SL's activity plan includes the organization of a calendar of user training sessions, but this training was not possible at all”, or “Little acceptance in working the framework “Learning with the School Library” in articulation with the teachers” or even, “It lacks a SL’s stable team. The teachers who support the SL are teachers with article 79 of the teaching career statute or mobility due to illness, which sometimes causes difficulties in the management of human resources”. As a positive element, the following generic information stands out: “The material produced by the SL is of interest to students and teachers, supporting them in their search for information.”

Regarding the SL A, the report mentions that the “Learning with the School Library” framework was applied in various activities, including training in the area: “SL promoted a training event, at the beginning of the year, for the teachers and other staff who are part of the SL team, on the SL referential and the area of literacy, the use of technology and new pedagogical and didactic practices”. It appears, however, that the report is cautious in the way it makes a global assessment of the SL action in the MIL area. However, the work carried out, and reported more objectively in the report, is perceived positively by the students.

Finally, in relation to the Setúbal district, which includes locations with very diverse characteristics, we chose SL A with the most positive evaluations from the students in question 10 (84% of students rated the SL’s impact as “Very good”) and two SL with similar evaluations in the lower quadrant (SL B - 22% of students rated the SL’s impact on learning as “Very good” and 10% as “Weak”; and SL C – 19% rated it “Very good” and 10% “Weak”). In the SL B report it is pointed out that the library was without TL for most of the school year, absent for health reasons, adding that it was not possible to develop the usual program of activities in IL: therefore, the evaluation of the students is justified. SL B, located in a small town in a rural region, presents in its report a classification of 3 for the three items in the MIL area. Weak points mention the lack of computers and other digital devices that allow carrying out more actions in MIL. In fact, it appears that this need is transversal to the various reports, and not only those of the SL that present a weaker evaluation. SL C, with the best scores given by students, evaluates its work with a 4. As strong points, it points out, for example, the use of social networks and the participation of parents in some activities; in terms of weaknesses, the report underlines the “Urgent need to equip the library with more computers or tablets for research work and personalized dynamics with students and not always in small groups as is currently the case.”

5 Conclusions

This study is not limited to providing a simple photograph of the existing situations, but aims further, to explore the possible implications for the future. In fact, with the Covid19 pandemic, which caused important changes in the functioning of schools and SL, the SLN interrupted the evaluation of SL, and the questionnaires that are part of this process were not applied. Only in this year, 2023, TL will once again carry out the entire evaluation path. This new data can be compared with previous cycles, namely with the 2019 data, which we analysed in more detail.

Based on the students’ perspectives, after all the central elements of the educational process, the study demonstrated that SL in Portugal are carrying out relevant work in MIL that covers a quite significant number of students. However, the present study also reveals that there are several gaps in the work developed by SL, which require an urgent intervention by the SLN (and the support of the Ministry of Education) in several aspects. It appears that some MIL domains are not being properly worked on. Even in the field of IL, there are many schools in the analysed sample (and, we can assume, that this situation will also be registered in the other regions) that do not develop systematic activities in this area. Thus, many students will be unable to acquire fundamental skills, which we know are also not worked on in the curricular subjects. This situation is also identified in another study [12] based on a questionnaire applied to TL.

Traditionally, since the creation of the SLN, in 1996, the MIL's work has been in the SL. However, there is no obligation for teachers to collaborate with the SL or regular involvement of classes with the SL, in a scheme like library lessons that we find in other national contexts. In Portugal, although there are guidelines from the SLN, the work of the SL is very dependent on the context, on the value that is attributed to the library, on the support of school principals, on the interest of teachers of the various disciplines and on the profile and training of the TL. Furthermore, since 2015 there has been a setback in schools, given that the role of the TL is no longer a full-time job and these teachers also must ensure a class, which has reduced their availability for library work. One TL often has to support libraries in several schools, with the consequent dispersion of efforts, with 1st cycle schools being the most affected, as we can conclude from the analysed data, which show lower values of involvement of students in this cycle of schooling in MIL-related activities.

So, the research results also make it possible to critically problematize the SL's work. One of the critical elements that it was possible to identify is related to the way in which some evaluation results are transferred to the reports, by the TL. Especially in the SWOT analysis that should be presented, some lack of precision was noticed, with the inclusion of very general observations that will be of little help in terms, for example, of the improvements that will have to be implemented.

Based on the results of this study, it is recommended that the SLN provide some training into the proper registration of the elements resulting from the evaluation process that is carried out in the school. It is also necessary to follow up more closely some schools, to better understand the factors that influence the work carried out, especially those that reveal a lower quality work, so that it is possible to find effective improvement strategies. Equity issues are fundamental, and recent data from PIRLS [13] reveal that in Portugal there are still many discrepancies in student outcomes related to their geographical and socio-cultural origin. In this sense, it is also important to overcome some problems of this nature also in Portuguese school libraries.