Activities with Educational Robotics: Research Model and Tools for Evaluation of Progress

Abstract

The use of robots in the learning process has been popular since S. Papert developed his LOGO Turtle idea and argued that students can construct their own knowledge, test their constructive solutions and be motivated to learn if they use robotics in the learning process. Today, the idea of using elements of robotics in the learning process is no longer new and innovative but there are still elements that can be developed and issues that should be discussed. In this chapter, the authors provide the research model and five research tools (structured observation protocol, evaluation of the possible risks of early school leaving to be filled in by teachers before and after activities, students’ questionnaires to be filled in before and after activities) for evaluating the outcomes of organized after-school robotics activities. The research model and tools were tested and approbated with students who are at risk of early school leaving and students who participate in robotics activities to develop computational thinking.

Introduction

The use of robots in the world is no longer a novelty. There are some who think that the origins of robotics came with the work of the Czech author Karel Capek’s RUR, or Rossum’s Universal Robots, 1921, where the word ‘robot’ is first mentioned, which means the term describing devices that do the work (Niku 2011). Others say that robotics is beginning its success with Papert’s ideas that children need to learn computational thinking while working with computers and later developing the LOGO turtle (1983). Three major directions in robotics could be distinguished at this time:

• Industrial robots

• Educational robots

• Assistive robots

This chapter will analyse the use of educational robotics (ER) and the possible evaluation of results. In general, it is clear that robots have become a part of our daily lives, but to prepare the new generation for robots, to create new innovations, to create new software and to be prepared for the challenges posed by robotization, we need to develop computational thinking, but also to raise awareness of the side effects of robotization of various processes, such as interactions between robotics and inclusive education (Bargagna et al. 2018; Catlin and Blamires 2018; Daniela and Lytras 2018; Jung et al. 2019; Ronsivalle et al. 2018), the possibilities of using robotics to reduce early school leaving risks (Alimisis 2014; Daniela and Strods 2018; Karampinis 2018; Karkazis et al. 2018; Moro et al. 2018) and meeting the risks of robotization, which are mentioned in the European Civil Law Rules in Robotics (Nevejans 2017).

Theoretical Background

The use of ER in the learning process has been popular since the time Papert came up with the idea that children themselves should be allowed to work with robots to promote their computational thinking and defined this direction as constructionism (1984). Currently, the ER learning process is widely used at various levels of education − starting from preschool (Bers 2008; Cejka et al. 2006; Kazakoff and Bers 2012, 2014) to higher education (Danahy et al. 2014; Sünderhauf et al. 2018; Sünderhauf et al. 2016) – in various fields and in various dimensions: science, technology, engineering and mathematics (STEM) (Williams et al. 2007), engineering (Ariza et al. 2017; Zaldivar et al. 2013) and other branches of STEM (Eck et al. 2014; Witherspoon et al. 2018). The relationship between science and engineering practice (Bell et al. 2012; Li et al. 2016) is described, but the relationship between them should be clearly seen by the students as well and can be purposefully developed by supporting the development of computational thinking (Bocconi et al. 2016) by developing such competencies as asking questions and defining problems; developing and using models; planning and carrying out investigations; analysing and interpreting data; using mathematics and algorithmical thinking; constructing explanations and designing solutions; engaging in argument from evidence; and obtaining, evaluating, and communicating information.

The issues of the use of ER have been actively thought about and studied over the past decades, with a particular focus on mastering STEM as a whole. Many researchers are discussing what we understand today by integrated STEM education, which dates back to 1990 in the USA. However, there is still a lot of focus on improving science and mathematics as isolated disciplines (Kelley and Knowles 2016), with less emphasis on teaching technology and engineering (Castro et al. 2018). Nowadays, with the advancement of technology, it is essential to promote the development of both computational thinking (Bocconi et al. 2016) and thinking about the possibilities of various activities with technology, to foster mutual cooperation between students in order to strengthen the formation of social links that are essential to making the individual feel accepted in society.

Despite the recent emergence of more and more initiatives to use robotics creatively, to use different robots to develop certain competencies, to promote the development of an inclusive society and to strengthen the development of STEM competencies, there are authors who point out that in the early stages, children are offered too few activities to help them develop their STEM competencies in their broader sense; more activities are focused on natural sciences through analysing plants, animals, natural conditions and so on. Sullivan and Bers state that these aspects are important in fostering children’s comprehensive knowledge, point out the importance of preparing children for the world as it is today as well as preparing for the progress that humans have created and therefore emphasize the need to promote the development of the competencies required for technology collaboration (2016). There are many examples where the acquisition of STEM is associated with robotics activities initially organized as after-school activities (Smyrnova-Trybulska et al. 2017), but which later try to include them in the compulsory curriculum (Ntemngwa and Oliver 2018).

The Context

The research model and tools that are offered to the reader are originally designed to meet the needs of the project ‘Robotics-based learning interventions for preventing school failure and early school leaving’, where Italy, Latvia and Greece jointly implemented a project whose main objective was to reduce early school leaving (ESL) risks using ER. The focus of this RoboESL project was more on the involvement of pupils at risk of ESL, so it is important to remember that they are pupils who often have low learning motivation, have lost the desire to actively engage in the compulsory learning process because their learning needs are not being satisfied and have a relatively negative attitude towards teachers. As a result of the project, evidence was provided that purposefully organized robotics activities reduce the risks of ESL, and original lesson descriptions for working with LEGO Mindstorm robots were developed. The pupils involved in the project activities were selected according to the methodology developed for the project (Daniela 2016), where teachers had to assess the risks of social exclusion of pupils, and those students who were at the highest risk of social exclusion and whose parents agreed that their children would engage in such activities took part in the project. Robotic activities were organized as after-school activities after a compulsory school day once a week. Teachers working with pupils represented a variety of subjects − ICT teachers, mathematics, physics, English, philosophy, home economics and so on. The participation of teachers in project activities was voluntary, where their benefit was the opportunity to participate in an international project, participate in training organized by the project team, and use innovative teaching methods in their work.

The evaluation tools were re-examined in the context of organized after-school activities with LEGO Mindstorm educational robotics. The curriculum for after-school activities was developed by integrating elements of robotics, science and maths. Students aged 11–13 years old were offered the chance to work with the developed curriculum and were motivated by teachers to develop different LEGO robotics constructions by themselves. This kind of activity is like an intermediate step between the compulsory process in school, where students have to complete certain tasks for which they receive assessment, and leisure time activities, where children have full freedom in choosing the activities they want to do at a given moment.

Students worked on an integrated curriculum that included science, mathematics and programming. The total number of classes in the programme during the school year is 56 lessons, each 120 min long. The programme can be used as a deepening and extension of the school curriculum. In these classes, students engage voluntarily, but the learning process is structured and designed to achieve specific learning goals according to the programme developed.

The course included elements of engineering design process:

• 1. Identify the need or problem.

• 2. Find the information to create a robot.

• 3. Design the idea and plan the activities.

• 4. Prototype the program.

• 5. Test and evaluate the program.

• 6. Provide feedback. Feedback can be asked for and/or given at any point during engineering design.

• 7. Redesign the product to improve it.

In these robotics lessons, pupils were mostly given a problem, to which they need to find a solution, except for the final lesson block − creating their own project.

Research Model and Tools for Evaluation of Educational Robotics Activities

In this section the authors offer a research model, tools and methodology on how student activities and the outcomes reached can be evaluated during and after educational robotics activities. Developed tools are designed to evaluate students’ attitude, motivation, collaboration, problem-solving skills and learning activity, so they can also be used in other contexts.

Before the research is started, data are collected and further analysed and it is imperative to have the parents’ permission for the researchers to obtain and analyse the results of the study. It is also important to observe the principles of data protection in data collection and use the obtained results only to improve the learning process and reduce problems that can cause risks in order to promote the development of certain competencies as well as to allow students to express their views on organized activities.

The first tool provides (see Appendix 1) the evaluation instrument, which consists of several parts. In part one there are a few questions which can be used to detect the learning subjects that are problematic for particular students and the learning subjects in which these students achieve their best results. This information can be used to evaluate intervention activities to find out whether they are effective or whether additional intervention activities should be added to support the student. Next, information is asked regarding the number of lessons that students are skipping to find out whether there is a problem with truancy, and these data can be analysed after specific activities are provided to conclude whether interventional activities are successful or not. Moreover, there is a section where teachers assess students’ difficulties that can influence the learning process. This part of the evaluation tool can be used separately to detect risks of ESL in order to plan the activities that are aimed at reducing such risks. The subsequent parts of the questionnaire consist of statements where teachers have to evaluate students’ attitude, motivation, behaviour and problem-solving skills on a Likert scale of 1−5, where 1 = never, 2 = rarely, 3 = sometimes, 4 = often and 5 = always.

The second tool (see Appendix 2) is the questionnaire for students, which should be filled in before they start robotics activities. This tool is divided into several parts: first, students give the information about themselves; second, they are asked to evaluate the statements about learning and achievements on a scale of 1−5, where 1 = completely disagree, 2 = rarely agree, 3 = sometimes agree, 4 = mostly agree and 5 = completely agree. This is followed by the part where students are asked to evaluate the statements about missed lessons on a scale of 1−5, where 1 = always, 2 = often, 3 = sometimes, 4 = rarely and 5 = never. These questions are aimed at finding out the reasons why they are skipping lessons and provide the opportunity to focus on problems that are emerging for the students in order to find the best possible solution. The last part of this questionnaire is to find out the students’ opinion about the learning subjects that they like or don’t like. This information, together with the information given by the teachers about the evaluation grades in particular subjects, can help to provide an understanding of the linkage between students’ attitude and learning outcomes.

The third tool (see Appendix 3) offers a structured observation protocol, which is used by teachers to evaluate students’ outcomes during ER activities according to the criteria, which have to be evaluated on a Likert scale as 0 = can’t be observed, 1 = low level, 2 = can be observed almost in all situations and 3 = does more than expected. This tool can be used quite frequently to see the changes in students’ outcomes.

The fourth tool is a questionnaire that can be used after the activities (see Appendix 4), whereby teachers fill in the information about changes in students’ attitude, motivation and problem-solving skills. These statements should be evaluated on a scale of 1−5, where 1 = no changes at all, 2 = some signs of improvement observed occasionally/rarely, 3 = some signs of improvement observed sometimes, 4 = signs of improvement observed in most situations and 5 = strong improvement observed in all situations. Statements about the student’s behaviour should be evaluated on a scale of 1−5, where 1 = no changes at all, 2 = some signs of positive improvement observed occasionally/rarely, 3 = some signs of positive improvement observed sometimes, 4 = signs of positive improvement observed in most situations and 5 = strong positive improvement observed in all situations. This tool can be used after a period of intervention activities to find out whether there are improvements, but the authors suggest that the first minor changes can be observed after at least 3 months of activities because the risks of ESL develop over a longer period of time and reducing them is not a fast process.

The fifth tool (see Appendix 5) is the questionnaire for students which should be filled in after participation in robotics activities to find out whether there are any changes in students’ attitude to learning. The tool is organized into several parts where the first part is to get the information about the students; the second part is to get to know the students’ opinions about robotics activities to give teachers the opportunity to understand whether there are any changes needed in organizing such activities. The next part is organized as substatements about ‘learning with robots’ and the statement ‘Activities with robots helped me to improve my:..’ where students had to evaluate these substatements in a scale of 1−5, where 1 = completely disagree, 2 = rarely agree, 3 = sometimes agree, 4 = mostly agree and 5 = completely agree. The last question in this questionnaire included the part where students were asked to name three learning subjects where their learning outcomes improved after participation in robotics activities.

The research model is summarized in Fig.10.1.

Fig. 10.1

Research model

Discussion

The proposed research model and all suggested evaluation tools have been tested in several contexts in Latvia, Italy and Greece. The results are summarized in several articles that have already been published (Daniela 2016; Daniela and Strods 2016; Daniela et al. 2017), and one chapter, ‘Educational Robotics for Reducing Early School Leaving from the Perspective of Sustainable Education’, is included in this book where results are analysed from the perspective of sustainable development − by Daniela and Strods. All these results confirmed that the research model and tools can be used to work with students and evaluate their progress.

These tools have also been tested in another context where 11–13-year-old students participated in organized after-school robotics activities to develop their computational thinking.

The proposed research model can be replicated, and the developed tools can be used in different contexts to evaluate outcomes such as learning motivation, improved attitude to learning and improved behaviour and problem-solving skills. To extend the usability of this set of tools, authors continue to work on the development of specific tools to evaluate knowledge improvement during robotics activities.

Appendices

Appendix 1

EVALUATION INSTRUMENT for detecting students who are at risk of ESL

EVALUATION INSTRUMENT

General information about student

Student ____________________________________________ (code of the student)

____ (age)

Subject/s you teach for particular student _________________________________

☐ subject teacher ☐ class coordinator/class teacher ☐ robotics teacher

Average evaluation grade for the 1st term for all subjects _______

Three learning subjects with LOWEST grade for the 1st term (starting from the lowest):	average mark	Three learning subjects with HIGHEST grade for the 1st term (starting from the highest):	average grade
1.		1.
2.		2.
3.		3.

Missed lessons 3 months before starting participation in activities	Number of missed lessons
1. Excused by doctor
2. Excused for other reasons (parents etc.)
3. Without an excuse

Difficulties which influence learning Mark with ‘X’ if the statement characterizes the student
Has learning difficulties connected with reading
Has learning difficulties connected with calculation
Has difficulties in understanding graphs and schemes
Has other different learning difficulties
Has other special needs
Has attention concentration problems
Attitude to learning Please evaluate these statements about the student on a scale of 1−5, where 1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always
Completes homework
Cooperates with teachers in a positive way
Cooperates with classmates during lessons in a positive way
Is ready for work in lessons
Understands the connection between learning and achievements
Is ready to do extra assignments to improve achievements
Obeys behavioural rules in classroom
Is ready to join out-of-class/school activities together with other classmates
Is involved in sport/art activities not connected with learning at school
Knows that the learning is important for him/her
Problem-solving skills Please evaluate these statements about the student on a scale of 1−5, where 1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always
Solves the learning problems by himself/herself
Asks for help from teachers
Solves conflicts in a calm way
Motivation Please evaluate these statements about the student on a scale of 1−5, where 1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always
Is motivated to learn the subject you teach
Is motivated to understand his/her mistakes to correct them
Is motivated to improve achievements
Is motivated to overcome difficulties in learning
Has an aim and works to achieve it
Observed problems Please evaluate these statements about the student on a scale of 1−5, where 1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always
Is late for the beginning of lessons
Has problematic behaviour during recess (break)
Is aggressive to other students
Is aggressive to teachers
Uses rude language with classmates
Uses rude language with teachers
Refuses doing assignments during the lessons
In situation of conflict reacts aggressively

Appendix 2

Hello! You are going to learn how to work with Robots! Congratulations! It is fun! Before starting learning, can you answer some questions? Your responses are anonymous!

• 1. I am:

  ☐ boy ☐ girl

  I am ______years old (your age)

  2. Learning and achievements Please evaluate these statements on a scale of 1−5, where 1 = completely disagree, 2 = rarely agree, 3 = sometimes agree, 4 = mostly agree, 5 = completely agree
  3.1. Learning is fun
  3.2. My achievements depend on my learning
  3.3. I do all the homework
  3.4. I like to cooperate with my classmates in lessons
  3.5. I like to work individually to do assignments
  3.6. I like to do extra assignments
  3.7. I like it when there are different activities in lessons
  3.8. I like it when I can do something active in lessons
  3.9. I like to solve learning problems by myself
  3.10. I like to look for extra information needed for learning
  3. If you miss lessons it happens because Please evaluate these statements on a scale of 1−5, where 1 = always, 2 = often, 3 = sometimes, 4 = rarely, 5 = never
  4.1. I was sick or had an appointment with the doctor
  4.2. I had to participate in another activity – sports, art, etc.
  4.3. I had to help my parents
  4.4. It was an unpredictable situation
  4.5. I had to work
  4.6. I had to babysit my sister/brother
  4.7. I did not want to go to school
  4.8. I overslept
  4.9. I did not want to meet my classmates
  4.10. I do not like learning

• 5. Please write here the three learning subjects you like the most

  • 5.1. _______________________________________

  • 5.2. _______________________________________

  • 5.3. _______________________________________

• 6. Please write here the three learning subjects you like the least

  • 6.1. _______________________________________

  • 6.2. _______________________________________

  • 6.3. _______________________________________

Appendix 3

Protocol of observation

Teacher_________________________

Legend: 0 = can’t be observed, 1 = low level, 2 = can be observed almost in all situations, 3 = can be observed during all projects, 4 = does more than expected

Appendix 4

Dear Teacher,

Please assess the changes in the attitude of the student who has participated in the activities. Use the same code for the student as was used before the activities. This survey is very important. It will take approximately 5 min to fill in the questionnaire. Thank you in advance for your time.

Student _______________________________________________ (code)

Gender_____

Subject/s you teach _________________________________________________

Attitude to learning (statements are the same as in the first questionnaire, but evaluation is based on changes in the student’s attitude) Please evaluate these statements about the student’s attitude on a scale of 1−5, where: 1 = no changes at all 2 = some signs of improvement observed occasionally/rarely 3 = some signs of improvement observed sometimes 4 = signs of improvement observed in most situations 5 = strong improvement observed in all situations
Preparation of homework
Cooperation with teachers in a positive way
Cooperation with classmates during lessons in a positive way
Readiness for work in lessons
Understanding of the connection between learning and achievements
Readiness to do extra assignments to improve achievements
Following of the behavioural rules in the classroom
Readiness to join out-of-class/school activities together with other classmates
Readiness to join activities led by other classmates
Readiness to reach learning aims
Motivation (statements are the same as in the first questionnaire, but evaluation is based on changes in the student’s motivation) Please evaluate these statements about the student’s motivation on a scale of 1−5, where: 1 = no changes at all 2 = some signs of improvement observed occasionally/rarely 3 = some signs of improvement observed sometimes 4 = signs of improvement observed in most situations 5 = strong improvement observed in all situations
Motivation to learn the subject you teach
Motivation to understand his/her mistakes to correct them
Motivation to improve achievements
Motivation to overcome difficulties in learning
Readiness to work hard to achieve the aim
Observed problems (statements are the same as in the first questionnaire, but evaluation is based on changes in the student’s behaviour) Please evaluate these statements about the student’s behaviour on a scale of 1-5, where: 1 = no changes at all 2 = some signs of positive improvement observed occasionally/rarely 3 = some signs of positive improvement observed sometimes 4 = signs of positive improvement observed in most situations 5 = strong positive improvement observed in all situations
Being late for the beginning of lessons
Problematic behaviour during recess (break)
Aggressiveness to other students
Aggressiveness to teachers
Using rude language with classmates
Using rude language with teachers
Refuses to do assignments during the lessons
Aggressive reaction in situations of conflict
Problem-solving skills Please evaluate these statements about the student on a scale of 1−5, where 1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always
Solves the learning problems by himself/herself
Asks for help from teachers
Solves the conflicts in a calm way

Thank you!

Appendix 5

You had a wonderful opportunity to learn how to work with robotics. We hope you enjoyed that! Can you answer some questions about your experience? Your responses are anonymous!

• 1. I am:

  ☐ boy ☐ girl

• 2. I am ______years old (your age)

• 3. Which robotics activities did you like most? Please name at least three of them

____________________________________________

____________________________________________

____________________________________________

____________________________________________

• 4. Which robotics activities were most challenging? Please name them

  ____________________________________________

  ____________________________________________

  ____________________________________________

  ____________________________________________

  5. Learning with robots Please evaluate these statements on a scale of 1−5, where 1 = completely disagree, 2 = rarely agree, 3 = sometimes agree, 4 = mostly agree, 5 = completely agree
  5.1. Learning by using robots was fun
  5.2. I have learned how to program robots
  5.3. I liked to work in groups to do assignments with robots
  5.4. I liked to make calculations for programming
  5.5. I can use this knowledge in other activities
  5.6. I liked to solve problems with programming by myself
  5.7. I liked that others helped me to solve problems with programming
  5.8. I liked to look for extra information needed for using robots
  5.9. Other outcome (please name it)
  6. Activities with robots helped me to improve my: Please evaluate these statements on a scale of 1−5, where 1 = completely disagree, 2 = rarely agree, 3 = somehow agree, 4 = mostly agree, 5 = completely agree
  6.1. understanding of maths
  6.2. understanding of physics
  6.3. understanding of informatics and technologies
  6.4 attitude to learning
  6.5. cooperation skills with my classmates
  6.6. cooperation skills with teachers
  6.7. other outcome (please name it)

• 7. Please write here three learning subjects where your learning outcomes improved

  • 7.1. _______________________________________

  • 7.2. _______________________________________

  • 7.3. _______________________________________