Keywords

1 Introduction

Bebras is an international informatics and CT challenge born in Lithuania in 2004. In 2019, it attracted about three million pupils (from primary and secondary education) from all over the worldFootnote 1. The challenge is based on tasks that pupils have to solve  [17]. They are created by an international community made of Bebras enthusiasts from participating countries. They then improve and polish the proposed tasks once a year during an international workshop  [14].

In addition to a large amount of work done on tasks, Bebras community members are also active in conducting research related to various aspects of the Bebras challenge. During the past five years, research about the design of activities based on Bebras materials and tasks to serve computer science education is being led. These activities include the development of classroom activities and workshops, the creation of games, and the design of tests to evaluate CT skills.

The research presented in this paper has two goals. The first one is to review the literature to find the existing research about activities based on Bebras materials and tasks, built for CS education. Based on the review, the second goal is to identify research directions for the development of relevant activities, thanks to a survey conducted among Bebras community members.

Instead of the term “computer science”, many countries use other words such as computing or informatics, especially in multilingual Europe. The terms “informatics”, “computing” and “computer science” are used interchangeably in this paper, to refer to the same subject, that is, the entire discipline.

The remainder of the paper is organised as follows. Section 2 reviews the literature on Bebras based activities design. Section 3 presents the research methodology. Section 4 draws up the detailed results of the survey. Section 5 discusses the results of the presented research and gives several recommendations. Section 6 concludes the paper with further work and future perspectives.

2 Literature Review

The literature review highlighted five main categories of activities, created based on Bebras tasks, and that are used to support CS education.

The first kind of activity is the creation of task books or textbooks to use in classrooms with pupils. Several countries do create annual books with tasks used in their national challenge. In addition to the task itself, they are typically enriched with detailed explanations about the correct answer and about what they have to do with informatics. The goal of these books is, in the end, to provide teachers materials to delve into computer science concepts with their pupils. No specific research has been carried out about the design of such books.

The second category is related to task creation activities. Teachers can learn about informatics concepts in two ways: by creating tasks and by analysing and solving them and explaining why it is informatics  [10]. Therefore, creating tasks is an interesting activity for people willing to learn informatics concepts, should it be pupils or their teachers. One approach is to ask pupils to create Bebras-like problems during an activity in classrooms  [19]. These problems are evaluated by teachers according to three criteria: quality, originality, and understandability. The materials used by teachers to help their pupils creating tasks are existing Bebras tasks and a booklet for beginners in informatics.

The third category is about the development of games based on Bebras tasks and materials. Games provide learners with the most interactive experience and increased motivation thanks to elements such as goals, scoreboards, competitions, etc. compared to other kinds of resources  [7]. They are one important kind of resource to teach and learn CS concepts  [8]. A quick search in the Google Play Store reveals several game apps, either made of a collection of tasks to solve or as standalone games built from a single Bebras task. In the literature review, two different kinds of game-related activities have been found: tangible games and game design process workshops.

The first kind of activity related to games is the creation of tangible ones, such as a card game for high school students to discover algorithms and CS concepts unplugged  [9, 12]. The content of the cards is inspired by Bebras tasks, shortened to only contain the task formulation and a short explanation about why it is informatics. They are used to start the problem-solving process leading to the learning of new computer science concepts  [11].

Another approach involving games is the development of classroom activities where pupils are asked to imagine and design games based on Bebras tasks. For example, an experiment asked groups of pupils to think about and design a tangible game based on Bebras tasks they first solved and understood  [3]. In this latter experiment, the 6th-grade pupils who created the games also had to train 3rd-grade pupils to play the games they designed. This activity is an example of situated learning whose content is built based on Bebras tasks.

The fourth category consists of tests to assess computational thinking (CT) skills. The goal is to measure CT abilities or skills using Bebras tasks, which is possible by analysing the answers given by pupils according to some research  [1, 16]. These tests have also been shown to be complementary to others such as CTt and Dr. Scratch  [20]. Based on these observations, tests used to assess acquired CT skills either targeted to pupils  [5, 6, 18] or teachers  [2] have been developed in different contexts. The goal of these various researches is to assess quantitatively whether the targeted public has acquired CT abilities.

The fifth category consists of activities developed to train pupils’ CT skills  [22] or to build pieces of training for teachers with workshops, for example  [13]. The activities from the two last categories show the close relation between Bebras tasks and CT skills, even if it is not always clear whether some tasks do contribute to a better understanding of CT skills or not. Bebras tasks can surely be used as part of a larger activity meant to work on and train CT skills.

According to this literature review, there is an interest in building activities based on Bebras tasks to serve CS education. Although research in this direction is quite recent, five categories of developed activities have been identified. The four main characteristics of Bebras tasks, highlighted in the literature, that make them relevant to develop such activities are the following:

  • The close relation between Bebras tasks and CT abilities make them suitable to design activities to acquire CT skills and to measure their mastery level.

  • Bebras tasks typically having visual graphics and fun stories make them suitable to work with younger pupils, fostering soft skills such as their imagination and creativity and making these tasks relevant for game development.

  • Both the task solving and task creation processes can be used to design activities related to CS education, either on their own or as part of larger activities, typically as workshops or other in-classroom activities.

  • Metadata associated with Bebras tasks (answer explanation, ‘It’s Informatics’ section, category and difficulty level by age groups, etc.) make them useful for various kinds of activity that can be targeted to several age groups.

Finally, Bebras tasks are also used to teach some concepts of CS-related to other fields, such as mathematics. The literature review revealed research highlighting the fact that Bebras tasks can be used to teach the concept of graphs and their properties to younger pupils  [4, 15, 21].

3 Research Methodology

This mixed-methods research uses qualitative and quantitative data collection methods. A literature review of designed Bebras based activities is used as a qualitative one. As a quantitative data collection instrument, a questionnaire survey has been designed based on the results of the literature review.

A web-based questionnaire has been used and data collection was performed from July 5 to July 18, 2020. Participants of this study consisted of 24 representatives from different countries belonging to the Bebras community (out of 67 countries). Countries are volunteers to do any additional activity in addition to run the annual Bebras challenge. Respondents were the only ones who volunteered to take part in the survey. The questionnaire consists of 23 questions: 18 open-ended questions, two five-level Likert-scale questions, and three multiple-choices questions. Data collected from both the literature review and the survey have been used to provide answers to the goals addressed in this paper.

4 Survey Results

Table 1 shows the list of participating countries, including 14 from the European region and ten non-European countries. They have been divided into three groups: (I) old-timer countries, (II) experienced countries, and (III) newcomers, depending on how long they are involved with the Bebras community.

Table 1. Period at which surveyed countries started to be involved with Bebras.
Full size table

The first set of questions of the survey is about annual brochures, with tasks and solutions, of national Bebras challenges. These brochures are sometimes used as the base to build textbooks. The vast majority of surveyed countries (22 of 24) prepare such brochures. They aim to provide more details on the Bebras challenge as a way to learn informatics and CT. As reported by the representatives, more attention is paid to teachers, mainly to explain the correct answer and the relation with CT, for them to explain it to their pupils. For some countries, brochures are dedicated to students to help them understand the task and the correct answer. They also are a resource for pupils that did not get the opportunity to participate in the challenge or want to train off-contest. To write the brochures, 71% of representatives need more information than what can be found in the original task. Almost 42% of countries create additional information: improvements (graphics, explanations), lesson ideas, or other technical information (authors and contributors). Some countries also add mappings to their schools’ curriculum or educational goals (see Appendix A).

The next set of questions is about the use of eleven identified activities from the five main categories identified in the literature review, for different target groups. Table 2 shows the mapping between the activities and the categories.

Table 2. List of activities organised following the five main categories.
Full size table

Figure 1 summarises the relevance of the activities for their country and following their personal opinion. Data analysis reveals that the most relevant activities for countries are A7 and A9 (average score of 4.4) and the less relevant are A5 and A11 (average 3.7). According to personal opinions, the most relevant is A1 (average 4.1) and the less relevant is A11 (average 3.5). Figure 2 shows the activities’ relevance grouped by countries’ experience with the Bebras community. For old-timer countries, the most relevant is A7 (average 4.5) and the less relevant are A3 and A11 (average 3.4). For experienced ones, the most relevant is A10 (average 4.4) and the less relevant is A1 (average 3.6). For newcomers, the most relevant is A9 (average 4.6) and the less relevant is A8 (average 3.7).

Fig. 1.
figure 1

Relevance of activities for the country and according to the personal opinion.

Full size image
Fig. 2.
figure 2

Relevance of activities by country experience with Bebras.

Full size image

Grouping the results according to the five main categories results in the following orders, starting with the most relevant one:

  • CT skills training, games, task creation, textbooks, tests (for countries),

  • textbooks, CT skills training, games, tests, task creation (personal opinion).

Looking at the results from the countries’ experience perspective gives the following relevance order for the five main categories:

  • CT skills training, tests, games, textbooks, task creation (newcomers),

  • task creation, games, CT skills training, tests, textbooks (experienced),

  • textbooks, CT skills training, task creation, games, tests (old-timer).

The next questions of the survey are specific to the five activity categories. Respondents were asked to describe in detail the activities using Bebras tasks that are run in their country. They were asked to give the goal, continuity (one time or periodic), organiser (lecture, teacher, researcher, etc.), and whether the activity is an extra activity out of school or scheduled at school. The results are grouped following the five main categories and detailed in Appendix B:

Textbooks for schools based on Bebras tasks are written in 25% of countries. In South Korea, textbooks’ content is aligned with the national curriculum. In Lithuania, Bebras-like tasks are included in textbooks for primary schools, but not for secondary schools because their textbooks are focused on special topics. Switzerland is a champion with 13 books, those for primary schools using the challenges approach to introduce/train a topic. Their annual brochures serve as a kind of textbook since they are designed to be used by pupils and their teachers. Turkey also wrote a book targeted to higher education students.

Task creation activities can be workshops for classrooms with pupils or teacher training activities. In Hungary, a CT course exists for university students where they create and solve Bebras-like tasks. Also, part of a teacher training university course is based on Bebras tasks and activities with robotics. In South Korea, they both focus on problem-solving and making, in the teaching training process. Lithuania started with in-service teacher workshops and is now working on pre-service ones, in particular for task creation. Switzerland also organises similar workshops, to prepare teachers to create Bebras tasks for their pupils.

Games for unplugged activities are developed in almost 16% of countries. In Australia, they are a resource for learning rather than explicit games. They created downloadable and printable cards to encourage pupils to develop CT skills as well as teamwork, collaboration, and critical thinking. Hungary provides two types of challenges. In the shorter version, pupils have to solve four Bebras tasks to obtain a code to open a treasure box. The longer version is a challenge game with ten stations where different activities with physical game-parts like board games, robots, etc. are organised. Lithuania created three different sets of “Bebras cards” for pupils from 7 to 13 years old, based on Bebras tasks. Switzerland designed postcard-sized descriptions of activities and riddles based on Bebras tasks, for primary school teachers. Ireland is preparing dynamic instances of Bebras tasks, with difficulty level based on player’s ability and South Korea has a plan to realise an online game.

Assessment tools to evaluate pupils’, teachers’ or high school students’ CT skills are in the beginning stage. Hungary created a test for first-year students and a course to improve CT skills. There are also some attempts from PhD students in Austria to do this. Turkey is also currently developing such an assessment tool, focused on the design of a self-efficacy perception scale for CT.

CT skills training is organised in classrooms for pupils in 29% of countries. India does this during school visits. South Korea ran an offline Bebras camp. In Indonesia, the training is conducted by universities collaborating with the National Bebras Organiser. In Canada, the Centre for Education in Mathematics and Computing runs workshops in classrooms, from grades 7 to 12.

Training targeted to teachers is organised in 50% of countries. South Korea organises teacher training sessions, introducing Bebras tasks in the pre-teacher training course. In Lithuania, there are workshops and seminars for teachers, in different places. They like hands-on activities when something to play and to discuss is provided. South Africa runs in-service two-hour workshops for teachers, especially in non-urban/rural communities. Turkey wrote two books and prepared online training for the Ministry of National Education. Ireland brings a “Computational Thinking Obstacle Course” to schools where pupils solve Bebras tasks in small groups. They then discuss their strategies and researchers explain alternative strategies if needed. They also host workshops for primary and secondary teachers, both in- and pre-service, to learn about CT using Bebras tasks and help them creating lesson plans. These activities take place at school if several teachers from the same school request it, or at university. Australia organises webinars to discuss CT skills, how relevant they are to the Australian Curriculum, and how Bebras can be used in the classroom to teach and learn these skills. Bebras unplugged and CT in Action activities, which draw on the CT skills but in contexts outside of Bebras, are also offered. These sessions are open to primary and secondary teachers, should they be pre-service or substitute teachers. In Switzerland, workshops were offered at the central CS teachers meetings once or twice a year, when asked for by the meeting offerers. In the context of pilot projects to introduce CS education in primary schools, they also organised a few workshops for teachers to make them discover the contest and other resources of Bebras, like the annual brochure.

The survey also revealed that task solving workshops are also organised for public targets (parents, communities, sponsors, journalists, etc.) Ireland contributes to a “pub quiz” for parents and teachers from a local school, supplying one round of questions using Bebras tasks, once a year. South Korea also has a future project for public targets.

The next question reveals that 54% of countries have events where Bebras-based tasks are used. In Ireland, they have summer camps for 13–18 years old, the “DojoMor” computer science event (Scratch, Raspberry Pi, Web, audio and video tutorials), and the “Science Week”, a science and engineering showcase for the public. In Austria, Bebras tasks are used during the “Abenteuer Informatik Workshops” at TU Wien or Teacher Days. Some year, Belgium organises a second round of the Bebras challenge which takes place during a “Computer Science Day” where other activities are organised for the pupils (a CS Escape Game, unplugged activities workshops, conferences, robot programming activities, etc.). South Korea organises government-led online software coding parties, offline software festivals, and more. Participants solve tasks online and receive certificates, or participate in offline camps to collaborate and solve tasks. Lithuania has had a summer campus together with Kangaroo for many years, it means mathematics and informatics together. Bebras tasks are also used in national education exhibitions and for kids’ TV shows (as part of a quiz). In Portugal, “TreeTree2” organises a Summer Academy for young and gifted students, where Bebras is part of their CS introduction. In South Africa, Bebras is introduced as part of the “Programming and Application Olympiad” workshops. Switzerland uses Bebras tasks in central CS teachers meetings and “Schweizer Tag für Informatik Unterricht”, an annual half-day teacher training. Syria uses them in training sessions for the kids and adolescents’ programming marathon.

Bebras based activities are not limited to CS and CT. Syria uses them for mathematics activities. Bebras unplugged in Australia is mostly marketed to Digital Technology/STEM teachers, but with an emphasis on Critical Thinking, which can be integrated by schools in other disciplines. They encourage teachers from a broad range of disciplines to attend.

5 Discussions and Recommendations

Following the purpose of the study, the literature about activities serving CS education and based on Bebras tasks has been reviewed. Five main categories of activities have been identified. Adding the results from the survey lead to several observations detailed below.

Textbooks for schools are created and used in 25% of the surveyed countries. This category is also very important to representatives according to their personal opinion about each activity. Also, this category is emphasised by old-timer countries, otherwise than for experienced countries. In Switzerland (involved in Bebras since 2010), there will be 20 available textbooks (for a spiral curriculum) based on Bebras tasks. Lithuania (involved since 2004) uses Bebras-like tasks in primary school textbooks.

Researchers and educators see Bebras tasks as a way of engagement in computer science and CT. Materials based on Bebras tasks could find a place in curriculums. But more data should be provided on curriculum issues, in the original Bebras tasks, such as the related CT concepts or in which other disciplines/contexts they can be integrated. Also, no research on the design of textbooks based on Bebras materials and tasks seems to exist.

Task creation activities were mainly highlighted by experienced countries of the Bebras community. The opinion of newcomers is on the opposite, with the lowest scores as well for the relevance for the country as for the personal opinion. Such activities are not easy to create and organise, because participants in workshops should first understand the principles of CS and CT. In countries, this kind of activity is more applicable for in- and pre-service teachers to help them acquire not only problem-solving but also problem making skills.

More methodology about how to create good Bebras tasks should be provided. Existing research about this is mainly targeted to the Bebras community members but should be extended to reach teachers. Such a methodology would also be useful for new countries that are interested in starting the Bebras challenge in their country. They have indeed to provide task proposals for the international workshop.

Games development is the second most relevant category of activities for countries. Games are indeed a good way to motivate people to learn, especially for younger ones. But in practice, there are very few examples of such activities developed. Only 16% of countries are involved in games creation (such as card games) for unplugged activities. Ireland and Hungary would find physical versions of selected Bebras tasks very engaging for young pupils, such as made of wood/plastic/magnets and using marbles/water or board games. Also, Ireland is preparing dynamic instances of Bebras tasks.

Bebras tasks being tightly related to CT skills, games targeted at learning these skills should be developed, should it be physical tangible ones or virtual ones. Then, studies about the effectiveness and efficiency of these games should be conducted. Of course, developing games takes time, so they may be designed to be easily translated so that to reach as many people as possible.

Assessment tools are the second most relevant category for newcomer countries while other countries do not pay attention to them. This is the most scientific research consuming activity. As found in the literature review, Bebras tasks can be a complementary part of other tests to evaluate CT abilities. More research should be done on how to design such assessment tools and how they can be linked to curriculums. Identifying which psychometric methods could be used to build such assessment tools would also be a promising research direction.

CT skills training activities are almost the most popular and relevant category for all countries, especially for newcomers. Workshops for classrooms with pupils are organised in 29% of countries and workshops for teacher training in 50% of them. Countries’ representatives highlighted two concrete activities as the most relevant ones: workshops for pupils and teachers. This is probably because they are easy to directly organise from Bebras tasks and because they are perceived as the most effective kind of event to promote the Bebras challenge. Only part of in- and pre-service teachers are familiar with CS and CT, which may also explain the success of these workshops. They are very important, especially for primary school teachers. Solving Bebras tasks helps them understand more about CS and CT. Workshops targeted at the public were decided as least relevant. Although workshops are organised in many countries, very few research about how to design and structure them does exist. Another possible research direction would be to study how effective they are to make pupils and teachers learn CS and CT. This is of course related to the fourth category of activities.

Suggestions about what should be further developed were also asked in the survey. A first element raised by South Korea is the possibility to run tracks and sessions at academic conferences. They also think that Bebras could be linked to the International Olympiad in Informatics and, at a local scale, with the Korean Education Broadcasting. Singapore highlighted the fact that more online training materials based on Bebras tasks should be made available. Explaining to students how CT is helping them for programming would also be valuable. According to Lithuania, it would be good to systemise Bebras tasks that teachers can use to teach concrete topics. For example, it would be good to have a set of tasks to introduce algorithms on graphs. Switzerland thinks about how they could implement a database with all the necessary information in an easily usable form for teachers and educators. Making such a database free to use may help teachers to pick individual tasks or even to develop their own task sets. Of course, issues may arise concerning the maintenance and upgrade of such a database system.

6 Conclusion

Results from the qualitative research revealed that there are five main categories of activities based on international Bebras challenge tasks in the scientific literature (textbooks, task creation, games, assessments, task solving). Results from the quantitative study supplement the meaning of these categories and show that countries are developing concrete activities from them. Countries have been grouped into three groups according to their experience with Bebras. For old-timer countries, the most relevant activity is task-solving workshops for classrooms with pupils. For experienced countries, it is teacher training activities on task creation for higher CS/CT skills. For newcomers, it is teacher training activities on task solving. The survey also put in light that the Bebras community is very active, but there is still room for more collaboration and the production of easily shareable and translatable materials.

This paper also shows a gap between the number of organised activities and the percentage of them subject to objective research. This observation highlights a big potential for research about Bebras based activities serving CS education. Systemising Bebras tasks for wider use in CS/CT education also needs further investigation and is one of the directions for future work.