Keywords

1 Introduction

It is widely acknowledged that digital literacy is a must in current times. Yet, this field has received scant attention despite that teachers must constantly be upgrading their competences to meet the evolving demands of digital education systems [1]. Among all professions, training of digital competences for preservice teachers is considered the lowest of all [2]. Consequently, it is becoming increasingly difficult to ignore that teachers need the necessary set of skills to integrate technology in their daily practices, mainly when students are developing their own digital expertise and learning from early ages, see e.g. [3,4,5]. Trends show us an increasing number of technologies which are waiting for well-trained teachers to be implemented in the 21st century classroom [6]. In the coming section, a review of the literature is presented to scope what has been done in the field of teachers’ digital competences.

1.1 Digital Competence

The term digital competence is as elusive as evolving due to, among others, the rapid social and technological changes. Apart from some confusion about its conceptualization (competence, competency and competencies) [7], no one can deny that they are relevant to the educational field nowadays. Thus, Oberländer et al. [8] refer to digital competencies as “a set of basic knowledge, skills, abilities, and other characteristics that enable people…to efficiently and successfully accomplish their job tasks regarding digital media at work” whereas Ferrari [9] conceptualized digital competence as ‘the set of knowledge, skills, attitudes, abilities, strategies and awareness that are required when using ICT [information and communication technologies] and digital media to perform tasks; solve problems; communicate; manage information; collaborate; create and share content; and build knowledge effectively, efficiently, appropriately, critically, creatively, autonomously, flexibly, ethically, reflectively for work, leisure, participation, learning and socialising’ (p. 30).

Yet, there are few mechanisms to differentiate which digital competences teachers need to focus on. The most common tools are perception surveys and some frameworks intended to capture the nature of indicators in the complex field of digital competences. Although useful, both mechanisms heavily rely on personal opinions and, as such, perceptions regarding one’s own competencies might be prone to subjectivity. Moreover, respondents may be confronted to choose between what they want instead of what they need to develop around their digital competences. In consequence, there are divergent results in early and primary education teachers who are the focus of the current study. Betancourt-Odio et al. [10] reported that 427 teachers from 15 Ibero-american countries found their digital competences ineffective to cope with changing situations (such as those of the Covid-19 pandemic). Even more, when digital competences are compared between levels, early childhood educators seem to have a better performance than their counterparts in primary education [11]. Sometimes student teachers attribute digital skills deficiency to their professors [12] or the outdated university programs [13]. That judgement may be because, in some contexts, teaching the basics of a program (e.g. a word processor) is considered to be enough, denying thus the utilization of other interactive resources that may help in the teaching tasks [14].

Even if digital competencies were identified, teachers usually use them inconsistently throughout the teaching process or with more emphasis in some subjects than others [15] which, again, widen the existing gaps in the field of teachers’ digital competencies. In this regard, some local attempts, e.g. in Slovakia, to improve curricula are usually made, but again, they are based on subjectiveness [16].

On the other hand, there are several frameworks which have tried to standardize digital competences of teachers. Of the most seven used, the European framework for the digital competence of educators or DigCompEdu [17] has been considered the most favored by specialists which, of course, does not disregard the usefulness of other frameworks [18]. In addition to the obvious bias of self-perception, a common disadvantage is that each framework do not always point to the same indicators which has led researchers to take their own pathway through scale validations, see [19].

When applying DigCompEdu, it is usually the case that early childhood educators obtain the lowest scores [20] which contradicts what other studies found, e.g. [2, 7]. But this is not the only contradiction. There are more inconsistencies regarding the digital competences of early childhood educators. Despite acknowledging the importance of digital competences, student teachers in early childhood education believe that little these competences can help to their career and, although they recognize that ICTs may have a motivator effect on children, they do not always want to be trained in digital competences [21]. This perception is similar when other types of surveys are applied to early childhood education students [22]. However, in countries such as Portugal, the level of digital competences in teachers at primary and secondary level seems to be moderate while the pedagogic and student management competences are considered to be low [23, 24].

Another less frequent framework is the Common Framework for Teachers’ Digital Competence (MCCDD, acronym in Spanish) which is supported by the Spain’s National Institute of Educational Technologies and Teacher Training (INTEF). Supporters of this framework assure that it improves the digital competences in teachers mainly when it is combined with active methods [25]. Nevertheless, regardless which framework is taken into account, they all either use self-reporting instruments or are not adapted to countries in Latin America but the specific European contexts.

An additional oversighted component of every training program is the type of assessment used to evaluate digital literacy in early childhood educators. The next section deals with the type of assessment questions used in training programs for digital competences in education.

1.2 Types of Assessment Questions

The type of assessment may play a significant role in the teacher training of digital competences. Given the wide variety of questions to be used during a training program, stakeholders need to choose the best options. Some argue that formative assessment guided by a system can help trainees to better learn [26] as it has a positive influence on students due to its engaging nature [27]. Likewise, Draskovic et al. [28] calls our attention to the fact that students benefit from a more efficient method of performing tasks when using digital assessment items compared to pen and paper. Although, teachers may initially feel a little overloaded, the authors assert that teachers found this type of items easier to administrate and score than those of pen and paper. While there is some discussion about the fact that the type of question does not greatly affect the learning outcomes [29] others give special importance to certain types of questions. In this way, contemporary pedagogical models and their implementation in ICT mediating tools show a tendency towards quantitative and summative evaluation [30]. These researchers found that commonly used learning management systems (LMS) include a variety of quantitative assessment questions, with multiple choice questions being the most common.

On this subject, Mafinejad et al. [31] found that scores for multiple choice questions are significantly related to total skill scores while others, e.g. [32], see that multiple choice questions have the drawback that the student can choose the correct answer by guessing, without following the proper procedure to solve a problem. Where most research seems to agree is on the fact that there is practically no difference between text-based questions and multiple choice questions [33].

As described above, this section has outlined the importance of digital competences in teachers and how these competences are assessed. The evidence presented thus far suggests that discrepancies found among studies in the field of teachers’ digital competences may emerge from the subjective nature of the instruments to assess. As a way to find another alternative outside the realm of self-reporting instruments, we launched a training course to assess digital competences in the very field through learning analytics. Based on the literature review, we hypothesized that assessment of digital competences is also threatened by the type of questions used to assess those digital competences. Accordingly, this study set out to apply a highly reliable algorithm 1) to identify how the type of question affected assessment of digital competences. Then, 2) to determine digital competences of teachers based on their performance.

2 Methodology

2.1 Research Design

This study adopted a quantitative non-experimental cross-sectional design without any manipulation of the variables which were measured at a given time [34]. Based on this, our research followed four differentiated and sequential stages: (a) introduction, which aimed at analyzing previous gaps; (b) methodological design, in which we selected research tools and techniques, study sample, and the data analysis method; (c) results, which showcased figures and tables of findings; finally (d) discussion, to compare and contrast our findings with previous studies.

2.2 Participants

181 early childhood educators from different provinces in Ecuador were enrolled in our course. Their age ranged from 22–59 years old. Of them, the largest subgroup (59%) was made of teachers between 22–38 years old followed by a smaller subgroup (36%) with ages between 39–53 years and finally the smallest subgroup (3%) with an age between 54 -59 years old.

They all came from 12 out of the 24 provinces of Ecuador, being Pichincha the most representative (94%), followed by Azuay, Loja, Santo Domingo (3%) and finally the other provinces (1%). Concerning the type of institution, 32% reported to work in public schools, 29% in private institutions, 34% were not working at that time and 5% worked in charter schools.

Regarding the educational level in which they taught, 36% worked with children from 5–6 years old, 24% with children from 4–5 years old, 22% with children from 3–4 years old, 11% with children from 7 -8 years and 6% work with children between 5–6 years. For this study, the results of the first 59 teachers who finished the course were taken to avoid noise in the results.

2.3 Instruments and Procedures

For data collection, a virtual training course was planned, designed and implemented in an institutional Learning Management System (LMS). This course was called How to teach children through the Internet (in Spanish), and aimed at early childhood education teachers. The course was made of 3 blocks whose contents are detailed below (Table 1):

Table 1. Blocks of the course How to teach children through the Internet.
Full size table

For the purposes of this study, only blocks 2 and 3 were taken into account since block 1 was introductory and focused on capturing the attention of the participants rather than evaluating their technological skills. Hereafter, block 2 will be named Module 1 (Digital Content Creation) and block 3 will be named Module 2 (School Management).

Advertising of the virtual course was carried out through the institutional Facebook and registrations were open for two weeks during the height of the Covid-19 pandemic in 2021. At the end of the registrations, 181 teachers from different provinces of Ecuador were enrolled. Once the course was finished, the records of the grades of each student were retrieved from the online platform database.

2.4 Data Analysis

For data analysis we used a data mining approach. Data mining combines several technologies and theories (e.g. artificial intelligence, mathematical statistics, data visualization) to extract and link useful information to make scientific decisions more efficient [35].

Therefore, analyses were carried out in Python 3 [36] along with Pandas [37] (for tabular data management), numpy [38] (for array management), seaborn [39] (to make graphics) and scikit-learn [40] (for clusterization). Data were cleaned prior to descriptive and multivariate analyses by removing non-relevant information. Additionally, demographic data were coded according to province, city, institution in which teachers worked and the educational level in which they taught.

Data Analysis 1: How the Type of Question Affected Assessment

Data related to assessment were classified according to the type of question (multiple choice, true/false, matching). To determine which questions were easy or difficult for teachers, an analysis was carried out regarding the type of questions. To this end, the average for each participant grouped according to the type of question was obtained by following these steps: 1) data preprocessing, words were all turned to lowercase, spaces were changed to hyphens, special characters and accents were removed. 2) All scores were scaled to avoid extreme values. One of the main purposes of scaling is to make all datapoints be comparable, so we used the MaxAbsScaler method. 3) In order to compare the impact of the three types of questions: MC (multiple choice), T/F (True/False) and M (matching), the mean of the scores grouped by question type was obtained, to create three variables: MC_score, T/F_score and M_score (See Table 2).

Data Analysis 2: Digital Competences of Teachers

To determine the teachers' academic performance in terms of their digital competences, we applied k-means clustering which is an unsupervised classification method that groups individuals into subgroups based on their characteristics [41]. Additionally, the “k-means clustering algorithm is considered one of the most powerful and popular data mining algorithms in the research community” [42] (p. 1) and it has proven its value in grouping large amounts of data in virtual courses [43]. The value of k is fundamental in carrying out an adequate clustering, so the elbow method was used to determine the value of k. To get k-number of groups, it was necessary to compute the square of the distance between the sample points in each group and the center of the cluster to give a series of k values [41] (p. 2). Thus, the optimal number of groups was 3. In addition, the XGBoost regression model was used to determine the attributes that contributed the most to the final score of participants. i.e. we used XGBoost to make a feature selection of the most important characteristics of individuals.

3 Results

3.1 How the Type of Question Affected Assessment

The first reported results are based on the scores grouped by question type: MC (mul-tiple choice), T/F (True/False) and M (matching).

Table 2. Scores grouped by question type
Full size table

Table 2 shows the mean of the scores obtained by the participants with respect to the type of question. It can be seen that each score is slightly different, with TF_score being the highest, followed by MC_score and M_score.

Fig. 1.
figure 1

Scaled scores by question type.

Full size image

Figure 1 and Table 2 show that matching questions have a greater distribution of scores, which tells us that, for students, this type of question could be a little more complicated or that, by having a greater rating range, the information is more variable.

Additionally, the Pearson correlation coefficient was used to determine a linear relationship between age, module 1, module 2, final score, MC score and M_score.

Table 3. Relationship between attributes of the scores grouped by type of question
Full size table

Table 3 indicates that there is a slight negative correlation (-0.2063) between age and module 1 (Digital Content Creation), which suggests that the older the age the lower the score in module 1 (Digital Content Creation). Conversely, a slight positive correlation (0.1711) between age and module 2 (School Management) which makes sense as the older the age the higher the score in module 2 (School Management). Likewise, a marginal positive correlation between module 1 (Creation of Digital Content) and module 2 (School Management) (0.3042) meaning that the higher the score in module 1 (Creation of Digital Content), the higher the score in module 2 (School Management).

3.2 Digital Competences of Teachers

As there were multiple digital competences assessed throughout the two modules of the course, a rigorous procedure that objectively grouped teachers according to the mastery of those competences was necessary. Therefore, the k-means algorithm was used as explained in Sect. 2.4. From such analyses, we derived 3 groups of teachers with similar characteristics.

Henceforth, these 3 groups will be called performance groups, which present the following characteristics according to the type of institution and educational level to which teachers belong (Figs. 2 and 3).

Fig. 2.
figure 2

Teacher groups according to the type of institution.

Full size image
Fig. 3.
figure 3

Teacher groups according to the educational level where they teach.

Full size image

Group 0: It is mostly made of teachers who work in public institutions and teach at first grade level (children 5–6 years old).

Group 1: It is mostly made of teachers who are not currently working but teach at first grade level (children 5–6 years old).

Group 2: It is mostly made of teachers who work in private institutions and teach at early level 1 (children 3–4 years old).

Performance Groups According to Their Final Scores

Table 4. Final scores (module 1 + module 2)
Full size table

Table 4 shows that group 0 (public teachers-teaching children 5–6 years old) obtained the lowest mean (51.08 points) and approximately 50% of the participants obtained a score of 52.00 points out of 60. Group 1 (teachers who are not currently working - teach children 5–6 years old) obtained the highest mean (56.98 points) and approximately 50% of them obtained a score of 56.67 points out of 60. Group 2 (private teachers - teaching children 3–4 years old) obtained an average of 52.15 and approximately 50% of the participants obtained a score of 52.33 points out of 60.

Fig. 4.
figure 4

Performance groups in Module 1

Full size image
Fig. 5.
figure 5

Performance groups in Module 2

Full size image

The maximum score for each module was 30 points. So, Fig. 4 shows that, in Module 1 (Digital Content Creation), the scores of group 0 (public teachers- teaching children of 5–6 years) are more dispersed compared to group 1 (teachers who are not working- teach to children aged 5–6 years) which are more compact. This means that the level of knowledge/proficiency in group 0 is more heterogeneous which could imply a wider variety in terms of training in contrast to group 1 which has a more homogeneous level of digital competences.

On the other hand, Fig. 5 shows that, in Module 2 (School Management), the grades of group 0 (public teachers- teaching children of 5–6 years) are more scattered than those of group 2 (private teachers - teaching children 3–4 years old). This means that group 0 showed greater heterogeneity in terms of mastering the school management skills, which implies that this group may need a wider variety of training courses in the near future.

Fig. 6.
figure 6

Graphical comparison of the 3 groups regarding final score and modules 1 and 2

Full size image

Figure 6 reveals that group 1 (teachers who are not currently working and teach children aged 5–6 years) in green performed better academically than the other groups in both module 1 (Digital Content Creation, left) and module 2 (School Management, center) as well as in the final result of the course (right).

Fig. 7.
figure 7

Final scores of the 3 groups in relation to modules 1 and 2

Full size image

Figure 7 shows comparatively, in greater detail, the academic performance in both modules (Digital Content Creation and School Management). Group 0 (in blue) had a higher score in module 1 (Digital Content Creation) but their performance is low in module 2 (School Management) whereas group 1 (in orange) obtained the highest grade in the two modules and group 2 (in green) obtained a low grade in module 1 (Digital Content Creation) but their score went up in module 2 (School Management).

4 Discussion and Conclusion

Assessment of digital competences in teachers is often carried out through self-reporting instruments which may usually induce biased or contradictory outcomes as reported by the literature (see Sect. 1.1). However, our study set out a different approach by applying machine learning techniques on the digital competence behaviors of early childhood educators who took an online training course about how to teach children through the Internet. The course in itself was intended to develop and assess digital competencies in participants at the end of it. Yet we used learning analytics to unveil further details.

The results of this study have shown that groups of teachers sharing similar characteristics also share some common digital competences. The k-means algorithm grouped all participating teachers in three clusters: group 0 (public teachers-teaching children of 5–6 years), group 1 (teachers who are not working-teach to children aged 5–6 years) and group 2 (private teachers - teaching children 3–4 years old). Of all these three groups of teachers, group 1 showed the highest performance in the competences of the two modules of the course: Module 1 (Digital Content Creation) and Module 2 (School Management). While group 0 scored second place in Module 1 (Digital Content Creation) and group 2 obtained second place in Module 2 (School Management). That a group of teachers who are not currently working had obtained the highest performance may indicate that teacher training should not take place during periods of high workload.

It was also found that age seems to be associated with digital competencies i.e. the younger the teachers the higher their digital competencies which, in turn, support those initiatives to include such skills in the curricula of pre-service teachers. Comparing these findings with those of other studies, e.g. [44,45,46,47], confirm age as predictor of the mastery of digital competences. Regarding the level where teachers work (which seems to be another predictor of teacher competencies), our research showed consistency with past studies, e.g. [48, 49].

The second major finding was that the type of question does influence the outcome of assessment, in this case, of teacher digital competences. Matching questions are the ones with the greatest dispersion in their data, followed by the Multiple Choice and T/F questions, which shows that the Matching questions and Multiple-Choice questions are more complex. The max and min score in the T/F questions do not have any variation, which reveals that these questions have much less complexity. This indicates that matching and multiple choice questions seem to be most suitable as they can cover a wider range of knowledge in terms of content as stated by previous studies, e.g. [50, 51], and can discourage students to guess the correct answer as contrarily stated by [32].

In conclusion, teacher training should be organized according to diagnoses that reveal the most important variables to be intervened. This research shows that the type of question used in training courses is key, as well as the level of education to which it is directed and, to a lesser degree, the age of the teachers. In other words, teacher training on technological teacher competencies must be adjusted to the type of institution and educational level in which teachers have worked recently, considering the existing weaknesses. An implication of this is the possibility for institutions to design digital literacy training programs for their teachers according to these specifics.

Taken together, these results suggest that there are much more objective ways to assess teacher digital competences than sole self-reporting instruments. As with all such studies, there are limitations that offer opportunities for further research. Further work containing a larger sample size is required in order to reach more generalizable results.