Keywords

1 Introduction

Czechia is preparing to reform the provision of informatics by beginning to teach algorithms and programming in primary schools. This new teaching approach is very different from the current concept of teaching informatics at primary/lower secondary level. Contrarily, a similar reform had already been introduced in Slovakia several years ago, first in 2008. The breadth and depth of informatics and its content is currently set out in the Innovated National Curriculum [1]. It requires schools to teach a minimum of 1 lessons of informatics a week in two years at primary levelFootnote 1 and 1 lesson a week in four years at lower secondary level. Hence, informatics focuses not only on the teaching of digital literacy but also on the teaching of computational thinking and computer science basics from the earliest age groups.

At the moment, the informatics section of the Czech national Curriculum for primary and lower secondary schools completely fails to take into account computational thinking, including algorithms and programming [2]. Table 1 shows that schools are required to teach a minimum of one lesson per week throughout the primary level [2]. However, schools can decide to increase this number by using extra lessons they have at their disposal.

Table 1. Differences between informatics in school in Slovakia and Czechia, together with respective Bebras categories
Full size table

According to the curricular documents mentioned above, there are clearly great differences in the teaching of computational thinking at primary/lower secondary schools in the two countries. What unifies both countries in this respect is their participation in the Bebras contest, which aims to support computational thinking at all levels of education. Both countries organize the contest in several age categories, ranging from primary school pupils to high school students in their final year, for primary and lower secondary school categories see Table 1. In the past, the two countries have worked together to compare results from the contest [3, 4]. The authors faced problems with implementation and incoherency of data, having compared data from the school years 2007/8 and 2008/9. Finland, Lithuania and Sweden also attempted to carry out an international comparison of results from this contest [5]. Their findings indicate that there was a slight mismatch between the difficulty level of the tasks used in the contest and students’ actual abilities, as some tasks were too difficult. Their results also show that there is no difference in performance between boys and girls in this age group. A German research [6] reveals that there is no significant difference between the performance of girls and boys in younger age group in this contest. However, the differences increase dramatically with the age of the contestants [6].

In the run-up to a reform in Czechia, it would be of benefit to ascertain to what extent and in which areas the competences of Slovakian primary pupils differ from the competences of Czech pupils in the same year group and if the differences in gender performance could be caused by informatics education. We assume the Czech and Slovakian versions of the Bebras contest could provide a suitable platform for such research, measuring pupils’ results achieved in comparable tasks. The two countries have a very similar culture, both having very similar sets of values, a similar education system and school curriculum. It can therefore be expected that school teaching will be the real cause of particular differences in pupils’ competences. Another reason for selecting this platform is the large number of participants.

2 Research Aims

Our main research aim is thus to ascertain whether there are differences in Czech and Slovakian primary pupils’ achievement in the Bebras contest and, if so, in which areas of informatics. The impact of informatics teaching on Slovakian pupils as compared to Czech pupils, who are not taught informatics to such an extent, will be of interest to us. We are aware that this research problem should have a clear answer that the teaching of informatics positively influences contestants’ achievement in Slovakia. However, this was not unanimously confirmed by some of the results of the pilot study examining this issue. For that reason, we decided to investigate this in more detail.

As stated above, there is no significant difference in boys’ and girls’ overall achievement in the Bebras contest in this age group. However, among older pupils, boys are more successful than girls. Therefore, our additional research aim is to ascertain whether informatics education influences the difference in boys’ and girls’ achievement in some task group from the Bebras contest. It is possible that informatics teaching has a significant positive impact on achievement of one gender while the other gender is not influenced to such an extent. The impact of informatics teaching on differences in Slovakian girls’ and boys’ achievements as compared to differences in Czech girls’ and boys’ achievements will be of interest to us.

Two hypotheses were created and tested using analysis of gathered data:

  1. 1.

    Slovakian Year 4 pupils’ achievement in identical tasks in the Bebras contest is statistically significantly higher than that of Czech pupils of the same age in identical tasks in the Bebras contest.

  2. 2.

    Differences in Year 4 boys’ and girls’ achievement in identical tasks are the same in both countries.

3 The Bebras Contest

Both Czechia and Slovakia take part in the international Bebras contest, aimed at informatics and computational thinking [7]. This contest is designed for primary/lower secondary pupils and high school students and is organised into several age categoriesFootnote 2. The contest is called Bobřík informatiky in Czechia and iBobor in Slovakia. Both countries base their contests on the common Bebras core but they differ in partial decisions. Take the list of tasks used in the contest for example - despite a database of contest tasks being available to be accessed by all countries, each country is entitled to use its own subset of tasks [3].

3.1 Primary Level Contest Categories

Category Mini is designated for pupils in year 4 and 5 in the Czech version of the Bebras contest and in the Slovakian version this includes Benjamin and Little Beavers categories. Year groups which each category is designated for can be found in Table 1. Regarding our research aim, it appears to be most convenient to compare the results of pupils in the Mini (CZ) and Little Beavers (SK) categories. These categories focus on the competences of primary pupils (as opposed to the Benjamins category, which could include tasks verifying the competence of lower secondary pupils) and have been running in the contest for several years (category Mini (CZ) since 2012 and Little Beavers (SK) since 2011). The following part provides a comparison of the characteristics which the categories Mini (CZ) and Little Beavers (SK) have in common and those they differ in.

3.2 A Comparison of Categories Mini (CZ) and Little Beavers (SK)

Category Mini (CZ) and category Little Beavers (SK) have the same 30 min time limit for solving 12 tasks. There are three types of tasks in both categories: (a) multiple choice with four possible answers, (b) short-answer tasks (e.g. the contestant enters a certain number), (c) interactive tasks, where the contestant e.g. might drag cards with pictures to designated positions.

Tasks in both categories are divided into three levels of difficulty - easy, medium and hard. Contestants receive a certain number of points for a correct answer, whereas points are subtracted for an incorrect answer. The contestant’s score remains unchanged if he leaves the question blank [8, 9].

There are differences between the contests as well, though. One of them concerns interactive tasks. In the Slovakian category Little Beavers, interactive tasks often have a lower success rate. Interactivity offers pupils more possible answers, thus making it more difficult. The conception of interactive tasks in the Czech category Mini is to use them more as a help and they may even comprise answer checks, increasing their success rate as opposed to the Slovakian ones. Another difference is also linked to answering. Although contestants in both countries have the right to decide whether to answer a question or not, the way of leaving a question blank differs. In the category Mini (CZ), the contestant can check a special box Leave blank, whereas he can cancel his choice by pressing a Delete answer button in the category Little Beavers (SK). Results indicate that this is a significant difference and Czech contestants decide to leave a question blank much more often than Slovakian contestants. Another difference between the categories is in the way contestants register for the contest. In the Mini (CZ) category, contestants register for the contest themselves just before the contest is about to start by using a so-called school code provided by the contest’s school coordinator. This code is common for all contestants in one school. In the Little Beavers (SK) category, the school coordinator registers a contestant for the contest. The selection of contestants is left to the coordinator and often depends on the school’s capacity possibilities. If the school does not have a sufficient number of computers available, coordinators must select pupils who will compete.

4 Research Method

4.1 Choice of Comparable Tasks in the Mini and Little Beavers Categories

Tasks that were mutually comparable or the same were selected for research from the Czech Mini category and the Slovakian Little Beavers category. The selection of such tasks was carried out qualitatively. As each task is named differently in the Czech and Slovakian version, task content had to be compared. Comparison was done by means of qualitative coding [10] of the wording of the assigned task. Tasks were analysed by each researcher separately then jointly discussed. We focused on task type, wording of answer choices including incorrect answers (so-called distractors) and accompanying graphics. Tasks were not compared only within one year of the contest but throughout the duration of the contest from 2012 to 2017. By doing this, 15 tasks were identified as seeming to be fully comparable and being used in both the Czech Mini category and the Slovakian Little Beavers category. Along with them, another 9 tasks were identified as seeming to be comparable to a certain extent, differing only slightly (for example different distractors or different question). Particular care needs to be taken in interpreting comparisons of contestants’ achievement and for that reason these 9 tasks will not be used for further analysis. Only the 15 identical tasks will be explored.

4.2 Data Processing

The Bebras contest already has its own task categorization, but we have formed our own categorization during the last year, based on tasks from Slovakian contest, to suit our research [11]. In the first phase of this research, i.e. during the qualitative phase, tasks were placed into individual categories from our own categorization. A brief description of the categories follows:

  • Digital literacy tasks – focusing on verifying knowledge and skill that prove a good software or hardware understanding;

  • Logical tasks subdivided into:

    • graph tasks – pupils work with a graph structure (net, binary tree, etc.), carrying out complicated or less complicated operations on it;

    • statement tasks – by examining statements, pupils have make judgements as to which answer is correct. Statements may be in the form of text or picture.

  • Algorithmic tasks – pupils observe a procedure, algorithm or set of instructions to guide them to find the result of operating with objects or information. It is usually a matter of dynamic action.

  • Programming tasks – pupils either create or interpret a program in the form of simple commands, cards or icons.

Table 2. Number of Year 4 girls and boys in Slovak and Czech competition for each analysed year
Full size table

4.3 Sample of Participants

Having identified appropriate tasks, the second part of the research was approached. This was of a quantitative character. The sample of research participants was made up of contestants from the Mini(CZ) and Little Beavers(SK) categories, i.e. Year 4 primary school pupils. This is the only primary year group that competed in the same category in both Slovakia and Czechia. From the Czech Mini category, all contestants who had not been explicitly excluded by the school coordinator of the contest (for example due to cheating) and had registered themselves as Year 4 pupils became participants. It can be said that this is a case of nonprobability sampling, more specifically convenience sampling [12]. In the Slovakian Little Beavers category, it was all the contestants from Year 4 who had responded to at least one task. This was close to double the number of participants than in the Czech contest. Numbers of contestants are given in Table 2.

4.4 Data Analysis

Data needed for the analysis were gained from databases. Although both countries use their own database, differences between them are not significant for data analysis. Table 2 indicates the considerable differences in numbers of Year 4 pupils, i.e. our research participants. For that reason, we searched for a statistical method that would take the differing sample sizes into account and prove to be suitable for comparing the two distinct groups. For each task there was created contingency table with two nominal variables, country (SK or CZ) and answer (correct, incorrect, none) and we seek to find out if the results in the task is dependent on the participants’ country. Therefore, we decided to use the Chi square test of independence [13, 15] for each task which we identified as fully comparable. While chi-square test provides little information about the nature of the association [15], Pearson residuals were also computed. See example in Table 3Footnote 3. If there turned out to be significant differences in the zero response option between Slovakian and Czech Year 4 pupils, we decided to analyse only the number of correct and incorrect responses, excluding pupils who had not completed the task from the analysis.

Table 3. Example of country differences statistical analysis in task 2016-SK-10
Full size table

Our null hypothesis for each of the 15 observed tasks was: There is no difference in the response given for the chosen task between Year 4 pupils at Czech and Slovakian primary schools. This was set at level \(\alpha =0.025\), providing \(\chi ^2\) was smaller than 5.02 (according to statistical tables for \((2-1)(2-1) = 2\) degrees of freedom) [14]. In other instances, we dismissed the hypothesis, because test showed a strong evidence of an association (i.e. results of contestants are dependent on a country they come from).

In the analysis of differences by gender, we examined the differences separately for each country, which means there were two nominal variables – gender (boys, girls) and answer (correct, incorrect, none) and the same method as above was used. We investigated if answer is independent of gender of Year 4 pupils separately for each of the identical tasks. The Pearson residuals [15] were used to enable us to identify the level of significance the differences occurred at. If the score was between 1.96 and 2.93, the difference was at level \(\alpha = 0.05\). If it was between 2.93 and 3.3, it was at level \(\alpha = 0.01\). If it was higher than 3.3, the significance of the difference was at level \(\alpha = 0.001\).

5 Results

Thanks to appropriate statistical methods, we were able to work with an unequal number of participants, Slovakia having involved many more Year 4 primary school pupils than Czechia. Presumably, this is related to the compulsory teaching of computational thinking in Slovakian primary schools. Despite such a disproportion, pupils from both countries achieved the same results in several tasks, even considering their decision not to answer a question, something which Czech pupils would have been more likely to do otherwise. In several tasks, however, statistically significant differences between countries in the number of correct and incorrect answers were disproved, but there were statistically significant differences in non-response. All these tasks shall be considered as tasks with the same result in both countries.Footnote 4

5.1 Differences Between Countries

The results of the analysis show that there is no statistically significant difference between Czech and Slovakian contestants’ answers in 5 tasks. On the contrary, there is a statistically significant difference in 10 tasks (Czech pupils having done better in 1 of them). It follows that Slovakian pupils’ achievement in most tasks is statistically significantly higher than that of Czech pupils. To better understand what categories of tasks these statistically significant differences came up in (or did not come up in), the tasks were split into categories from our task categorization, see Fig. 1.

Fig. 1.
figure 1

Numbers of tasks by category and existence of statistically significant difference in answers

Full size image

As Slovakian Year 4 primary school pupils (as opposed to Czech pupils in the same year group) had informatics lessons, the observed results can lead us to assume that the teaching of informatics in Slovakian primary schools: (1) is likely to influence digital literacy competences, algorithmic competences and programming abilities of Year 4 primary school pupils, (2) has a possible influence on pupils’ statement logic ability, (3) is unlikely to have influenced pupils’ graph logic abilities.

5.2 Differences by Gender

The quantitative analysis described in Sect. 4.4 enabled us to observe the statistically significant difference to determine whether girls or boys had achieved better results in each particular task. If a common characteristic can be identified in certain tasks in this respect, it is stated. The analysis shows that the outcome of 10 tasks out of 15 was the same in both countries, as far as gender is concerned – i.e. either boys achieved better results in both countries; or girls did; or boys and girls achieved the same results in both countries. It follows that differences in boys’ and girls’ achievement in most tasks are the same in both countries. More precisely it can be stated that:

  • In both countries:

    • boys achieved better results in two tasks;

    • girls achieved better results in four tasks (those four tasks being from the Statement logic category);

    • girls and boys achieved the same results in four tasks.

  • boys and girls achieved the same results in 5 tasks in one country, while either girls or boys achieved better results in the other country (each option occurred).

In addition to the above mentioned findings, the following facts were also observed: (1) there were more significant differences between boys and girls in Slovakia, (2) tasks where pupils achieved the same results in both countries had differences in results according to gender, (3) tasks in which either Slovakian pupils or Czech pupils did better had the same outcome as far as gender is concerned.

Observed result did not show some trends or tendencies that gender performance is dependent on task category (apart from 4 aforementioned girls’ tasks). Ten tasks with same results can lead us to assume that the teaching of informatics in Slovakian primary schools does not influence the differences in boys’ and girls’ competencies.

6 Discussion

The qualitative analysis revealed several factors among the Slovakian and Czech tasks that may have led to very similar tasks differing in difficulty. These include different wording of tasks, different distractors, different accompanying graphics and different ways of formulating answers. Despite a task leading to the same outcome, different wording or highlighting a certain part of the task may cause differences in how pupils understand a task. Different distractors may confuse several pupils if they contain answers that pupils are often misled by. If the distractors do not reflect such behaviour, the correct answer can often be revealed by way of trial and error. The use of varying task types is also related to this, where multiple choice tasks might be set in one country, but interactive tasks may be set or short answers required in the other. In some short-answer tasks, there were cases where one of the most frequent wrong answers was not even included in the four available options. The most visible differences were in tasks with accompanying graphics. The Crispy Cake task (see Fig. 2) is one example where signposts providing links to the next ingredient in the recipe made the Slovakian version more user-friendly than the Czech one, which only displayed pictures with a different background.

Fig. 2.
figure 2

Comparison of graphics for the same task in the Slovakian version (on the left) and the Czech version (on the right)

Full size image

The research participants themselves are another factor which could have influenced the results of our research. Bearing in mind that computational thinking is not to be taught at primary schools as part of the Czech National Curriculum for informatics, it could be assumed that primary school pupils often enter the contest in Czechia due to their enthusiastic teachers. This might be a lower secondary informatics teacher (who does not usually teach at primary level and therefore does not directly influence primary pupils’ abilities) or it could be the primary teachers themselves whose enthusiasm leads to pupil involvement. To distinguish these two cases, we traced how many Czech schools took part in only the Mini category of the Bebras contest (which is intended for primary school pupils). According to the findings for years 2012 to 2017, there are always around a quarter of schools which only took part in the Mini category; the remaining three quarters of schools that took part in the Mini category also competed in the Benjamin and Cadet categories, which are intended for lower secondary pupils. On the other hand, teachers in Slovakia can, due to capacity problems, enroll only pupils with better school results or high intrinsic motivation.

Pupils’ achievement in the statement logic category may have been influenced by mathematics education in both countries, while in Czechia there are more schools that use different, non-traditional ways of teaching mathematics.

Our research could be considered to have been limited by the low number of quantitatively analysed tasks, of which there were only 15. The low number of analysed tasks was a result of efforts to eliminate the influence of differing task difficulty, which has been discussed above. For that reason, only tasks that were fully comparable were included in our quantitative research. To generalize the results, more task should have been studied.

7 Conclusion

The aim of our article was to ascertain whether and in which areas of informatics there are differences in Czech and Slovakian primary school pupils’ achievement in the Bebras contest. We compared the success rate of Slovakian and Czech pupils in those contest tasks that were identical in both countries. 15 such tasks were found in 2012 to 2017 contests. Due to differences in targeted age groups in primary school contest categories in the two countries, we narrowed the analysis of results down to Year 4 primary school pupils. Our own form of categorisation was then carried out on the analysed tasks [12]. Slovakian Year 4 pupils are more successful in digital literacy and algorithmic tasks, slightly more successful in statement logic and programming tasks. In tasks involving graph logics, there is no difference between Czech and Slovakian Year 4 pupils. The question here is to what extent graph logics should be included in an appropriate scope and form in the primary school informatics curriculum, this area having turned out to be the weakest of the areas researched in both countries.

If we are to focus on difference by gender, 10 tasks out of 15 had the same outcome in both countries, i.g. either boys did better in both countries; or girls did; or boys and girls achieved the same results in both countries. Slovakian pupils’ results showed significant differences in the number of right and wrong answers, whereas these differences were mainly in right answers in Czech pupils’ results.

We find the recommended improvements for the creators of contest tasks useful. They are to look more closely at the wording of tasks in Slovakian or Czech and harmonize the wording of tasks in both countries. It would also be possible to prevent double translation from English by arranging to cooperate on the translation, Czech and Slovakian syntax being much more similar as compared to English.

At the very end, let us state that the teaching of informatics in Slovakia is perceived positively by us, both from the point of view of achievement in the contest and in terms of the number of contestants. However, we believe that Czech pupils will also manage to achieve similar or even better results after the informatics teaching reform has been implemented.