Keywords

1 Introduction

In the school year 2016/2017 we participated in the 10th year of the competition iBoborFootnote 1 in Slovakia. It is an international competition named Bebras which originated in Lihuania and it is focused on informatic tasks [1]. In Slovakia iBobor competition has been organized since the school year 2007/2008. We included the category named Little Beaver (at the official website of competitionFootnote 2 named Primary) into competition iBobor in the school year 2011/2012 for the first time [2]. It was the first year when the fourth grade pupils had already had Elementary Informatics during primary school. This compulsory school subject has been taught since the school year 2009/2010 from the second to the fourth grade for one school period (45 min) per week. The new educational reform introduced in 2014 renamed aforementioned subject to Informatics [3] and placed it in the third and the fourth grade at primary school. In the first year, there were 7,727 competitors in category Little Beavers. Since then, the number of participants has gradually increased, and this year it was 15,486. The category Little Beavers is intended for the third and the fourth grade pupils at primary schools. However, teachers may also register younger pupils into this category. Due to the age and cognitive development of the primary school pupils, the organizers have decided to reduce the number of tasks to 12 for this category along with the reduction of time to solve the tasks down to 30 min, while other categories include 15 tasks each with 40 min to find the solution. Tasks in this category typically contain less text information and more pictures. The tasks mentioned are in the form of stories or they focus on real-life situations that should be well known to pupils - so it tries to eliminate abstraction [2].

2 Research Problem

Over the past few years, we have gained a great deal of data that can help us to create a picture of the state of pupils’ knowledge. In this article, we focus on the Little Beavers category and the results of pupils in this category over the last 5 years. Tasks in this competition usually come from an international database, yet about 50% of the used tasks in this category have been created by Slovak authors (Fig. 1). The reason may be that not all countries have a category for the primary education in the competition. Within the international database [4], the tasks are assigned to one or more of the following areas:

  • Information comprehension,

  • Algorithmic thinking,

  • Using computer systems,

  • Structures, patterns and arrangements,

  • Puzzles,

  • Social, ethical, cultural, international, and legal issues.

Fig. 1.
figure 1

Authorship of tasks in last five year of iBobor competition in category Little Beavers

Full size image

However, the tasks used in Slovakia in the category Little Beavers are designed according to the State Educational Program (hereinafter referred to as SEP) [3], which is divided into five thematic areas:

  • representations and tools (work with graphics, work with text, work with stories, multimedia work, information, structures);

  • communication and collaboration (working with a web site, searching the web, working with communication tools);

  • algorithmic problem solving (problem analysis, interactive solution compilation, solution by command sequence, interpretation of solution writing, finding and correcting errors);

  • software and hardware (work with files and folders, work in the operating system, computer and add-ons, work on a computer network and on the Internet),

  • information society (safety and risks, digital technologies in the company, legality of use).

The tasks categories used in the international database overlap only to a certain extent with the tasks originating from thematic areas of the SEP. Therefore, we are not able to uniquely include the tasks created for competition in the context of informatics in Slovakia. In 2009 Kalaš and Tomcsányiová proposed new categorisation for Bebras task [5] consisting of four categories: digital literacy, programing, problem solving and data handling, but categories are broad and overlapping as one task can fit into one or two categories.

In [6] the new two-dimensional categorisation system for tasks in Bebras contest has been introduced. It combines computational thinking skills with informatics concepts. Each task can be placed in only one informatics area, but in up to three computational thinking areas.

One of the objectives of this article was to investigate the success of pupils depending on different phenomena (e.g., conceptual complexity of the task, type of the task, age and gender of the contestant, etc.) and we therefore created our own categorisation to examine these dependencies. The base of it is similar to [5], but we tried to make unambiguous categories and define subcategories whenever possible.

There are not so many researches aimed at Little Beavers category, as many countries don’t have this category, but overal we can say, that in most countries boys and girls has similar performance in lower categories [5, 7, 9]. Boys tend to be more successful in tasks aimed at spatial thinking, creating strategy and in harder tasks, and girls perform better in tasks with colorful pictures and social themes [8]. Contestants are more likely to guess the correct answer than to use no respond answer, and boys do it more often (and even more successful) [7]. In lower categories the proposed difficulty of tasks usually differ from real difficulty, and pupils in this categories tend to underestimate the complexity of the tasks [9]. Pupils in primary schools have problems with reading long texts, they cannot focus on task for a long time, and need to have unambiguous texts and pictures [2, 9].

3 Methodology

The aim of our work was to better understand the results of contestants, and also find out correlation between assignments of individual tasks and their results. In our research, we have used qualitative and quantitative research methods. When analyzing the task assignment, we used grounded theory with the systematic design [10]. We coded 60 tasks from the Little Beavers category over the last 5 years, and, based on them, we subsequently created new categorisation. We tried to preserve the data triangulation and the objectivity of the results by individual authors’ coding and subsequent joint processing based on inter-coder agreement [11]. The created categories are described in Sect. 4. As in the competition, all tasks are evaluated by points based on difficulty, using three groups: easy, medium and hard, with 4 tasks being in each group. The proposed difficulty does not always correspond to the pupils’ results, so for our research, we have redistributed the analysed tasks into these three categories, depending on the real difficulty. (We sorted tasks based on their results, first four with the best average score were labeled as easy, next four as medium and four with the lowest score were hard.)

Next, we used qualitative methods to analyse data from the Slovak competition database, where the following information are kept about each contestant: gender, grade, school’s ID, the chosen answer and the time they needed to solve the contest. From this database, we first exported information about pupils competing in the category Little Beavers, which we further processed using inferential statistics for different groups - based on gender, or grade. We have also used correlation designs [10], examining relations and dependencies between the tasks results and the category in which they were placed.

4 New Categorisation

As it was mentioned above, we have created four new categories where all types of tasks used in Slovak competition in category Little Beavers can be included. When creating these categories, we focused on analysing the tasks’ text, or more specifically their form and content, and the type of tasks (which area of knowledge is tested). We have further analysed which information is available in text and which is needed to be analysed by pupils, and also how they come up with the answer - if they choose it or they need to create it. Example for each category is shown in the Fig. 2.

Fig. 2.
figure 2

Examples of tasks for each category from iBobor in school year 2016/17 in Slovakia. Volume - digital literacy task. Shelf Sort - logic task (statement). Bebras painting - algorithmic task (rules are given, answer is created). Robot 1 - programming oriented task (program creation).

Full size image

4.1 Digital Literacy Tasks

Tasks oriented towards digital literacy are those that focus on verifying the skills in using some (relatively general) software or hardware. We have divided them into two subcategories:

  1. i.

    basic - if the task is focused only on one type of skill or knowledge,

  2. ii.

    combined- if the task contains some more difficult concept or combines basic digital skills with algorithms, pattern recognition or rule identification.

This type of tasks are not usually used in Bebras in other countries and they are not very supported by the international community. However, we have found out that pupils improve between the third and the fourth grade, so we have come to the conclusion that this area of informatics takes a big part in Slovak education. In other countries like for example England, digital literacy is a part of different subjects, not only informatics or computing.

4.2 Logic Tasks

By logic tasks we mean tasks where the steps or algorithm to be followed are not clearly defined. The pupils use exclusion, matching or deduction to solve the tasks. We divided them into three subcategories:

  1. i.

    statement - Statement logic tasks are tasks in which pupils work with logic statement usually given in text, sometimes in a graphical form. Using these statements they deduce the answer.

  2. ii.

    graphs - In the graph tasks, pupils work with the data structure represented by a graph. Typically, it is a tree, a graph (e.g. a square network) or even a linked list or a diagram. More difficult tasks also include some graph algorithm - for example minimal path finding.

  3. iii.

    others - This is a minimal set of tasks that do not belong completely to either of the two previous categories. It can be a mixture of the two, or it can be a task demonstrating a completely different information representation. This category also includes logic puzzles.

4.3 Algorithmic Tasks

These are the tasks in which a pupil needs to follow a procedure or an algorithm or a set of instructions or rules to work with some objects or information to get an answer. We have identified two additional subcategories for these tasks based on the tasks’ rules and the way of response.

  1. i.

    rules

    1. a.

      are given - Pupils have to work with exactly defined conditions, rules, an algorithm or steps to follow and it is obviously stated for them how to solve the task.

    2. b.

      need to be identified - Pupils are not explicitly told the algorithm or conditions for solving the problem, so they need to identify them by themselves in the text.

  2. ii.

    answer

    1. a.

      is created - In this categorisation we do not focus on the way the answer is given in the competition (multi-choice or interactive), but on the way how pupils come up with the answer. They can create the answer from a scratch or simulate the whole algorithm, so that the answer is created.

    2. b.

      is identified - Pupils are already given a pre-prepared answer and they either need to finish it, or they need to identify the right answer which is not needed to be created from a scratch.

4.4 Programming-Oriented Tasks

Tasks which clearly include some execution or a creation of a program or a sequence of steps in a form of text, icons, pictures or combination of the previous are included in this category. Although these tasks might be quite similar to the algorithmic tasks, they specifically need to fall into one of the following subcategories:

  1. i.

    creation of a program - In this category pupils had to create or complete a sequence of steps or a program.

  2. ii.

    interpreting the program - In this category pupils need to interpret or simulate a program or a sequence of steps written in text to find the correct answer.

5 Results

First of all, we analysed data of the last five years of Little Beavers category for grade and gender groups. The competition is more attractive to boys than to girls and the average score of girls and boys is similar, with girls being slightly better in four out of five years (see Fig. 3). Girls achieved better results in tasks where it was not necessary to create or discover an algorithm or use some strategy, but to perform some sequence of steps or commands or to evaluate some states and statements.

Fig. 3.
figure 3

The last five years of iBobor competition in category Little Beavers - a. number of contestants, b. overall performance for boys and girls (maximum score is 96 points, minimum score is 0 points

Full size image

There were more fourth graders than third graders in competition and on average 2% of all pupils in category were second graders. Interestingly, the second graders, who entered the contest despite the fact that they were in the category with older pupils, achieved on average very similar results as third graders (see Fig. 4) and they even outperformed third graders and sometimes even fourth graders in some tasks. The reason may be that teachers only enrolled the most clever second graders, but there were often entire classes enrolled in the competition in the third and fourth grades.

Fig. 4.
figure 4

The last five years of iBobor competition in category Little Beavers - a. number of contestants in each grade, b. overall average performance for each grade (maximum score is 96 points, minimum score is 0 points

Full size image

Next, based on our new categorisation and data analysis, we were looking for some correlations and dependencies. We took a look at the group of pupils who contested in the school year 2015/16 as third-graders and again in 2016/17 as fourth-graders and compared them to the corresponding grade score average. We found that our group improved mostly in digital literacy based tasks. There could be many reasons why, but it is very likely that this confirms our belief that informatics teachers mostly include digital literacy in their lessons and therefore pupils are most advanced in developing user skills. The same pupils were slightly better than other contestants in the logic and algorithmic task categories and in the programming-oriented tasks they reached approximately the same level.

When comparing the results of all five years, we wanted to discover some correlation that would help us to estimate the difficulty of the task before its application in the competition. However, such correlation and dependencies could be ambiguous, as Vaníček [12] declares, although at least some correlation had been confirmed. These claims are divided into four groups according to the new categorisation, see below. For each group we have created a scatter plot to see if some gender correlation could be found (see Fig. 5).

Fig. 5.
figure 5

Correlation of gender performance for each group of tasks. Green line represents the same score for boys and girls. Point below the line means that girls performed better in representing task, point above the line means that boys were better. (Color figure online)

Full size image

5.1 Digital Literacy Tasks

  • The differences between boys and girls in this group were minimal.

  • If an algorithmic or logical problem is combined with testing user skills, tasks are more difficult for pupils - they are not easy.

5.2 Logic Tasks

  • If the task falls into the logic statement group and task text is in form text-image with clearly defined rules, task is easy for pupils.

  • This also applies to image-image tasks that have clear rules but they may not contain another algorithm-complex problem.

  • If the task has a small finite number of solutions, it turns out to be easy.

  • If the task had more than one correct solution or contains more complicated rules, it is medium difficult. They are mostly in image-image form.

  • If pupils need to create answer while the task contains more complicated rules consisting of a number of conditions that need to be identified, the task turns out to be hard.

5.3 Algorithmic Tasks

  • If rules are clearly defined in the task, it will be hard or medium difficult for pupils. This may be caused by the fact that authors, seeing that the task contains a more complex algorithmic problem, try not to make it more complicated by identifying rules.

  • On the other side, if the pupils had to identify the rules in the task, it usually had a high success rate, i.e. about 70%. In these tasks, pupils usually did not have to create a strategy to solve them.

5.4 Programming-Oriented Tasks

  • If task contains a programming concept – such as a variable, a cycle, an obstacle movement or conditions – and program commands are represented by images or pupils need to interpret text commands, task are difficult.

6 Discussion and Conclusion

In this paper we have described new categorisation of the tasks for Little Beavers group in iBobor (Bebras) contest. We have used this categorisation to analyse the last five years of the competition in Slovakia. We have found some correlations and dependencies, mentioned in Sect. 5, although we cannot definitely tell if some causalities really exist. So in the next competition iBobor, in the school year 2017/18, we plan to use these dependencies to categorise tasks in the difficulty groups, and, based on the pupils’ results, we would be able to better validate our statements. Also, we would like to use our categorisation on tasks used in other countries in similar age category to compare results of slovak pupils with their peers in other countries.

Pupils’ good performance in logic tasks can be influenced by their math lessons or even other subjects, so it would be helpful to do qualitative research aimed at primary school teachers (which in Slovakia use to teach all subjects in their class) and talk with them about tasks in Little Beaver category and possible interdisciplinary relations. So we would find correlation not only with informatics, but get a whole picture.

We were looking at the difficulty of the tasks using recategorisation as follows - first four tasks with the best results were easy, next four medium and the last four (with the lowest score) were hard. In further analysis we would like to recategorise tasks’ difficulty based on the percentage results, so the categories could contain more or less than 4 tasks, and then use this recategorisation to find dependencies between our proposed tasks categories and the task’s difficulty.

In gender-based analysis, we have found out that girls and boys have overall the same score, but some differencies between the tasks exist. Girls usually perform better in easier tasks, while boys are better in the hardest tasks. Yet to find out in which categories girls or boys are better, we need to create more specific categorisation and analyse the data.

In [12] interactive tasks showed up as easy, but in our data some of them were hard, so in the following research we will try to extend our categorisation to answer type used in the contest with subcategories for interactive tasks. All the proposed improvements could help us better understand what causes the difficulties in the contest and also allow us to create more suitable tasks for Little Beavers category.