Introduction

Educational robotics (ER) is a powerful and flexible teaching and learning tool which engages students in activities of robot construction and control using specific programming tools (Alimisis 2009). In a typical ER activity, students work in groups to address complex problems, being guided by a teacher’s worksheets. Through iterative design and testing, students get immediate feedback on their actions and learn how to deal with challenging situations in a real-world context.

Current research has focused on and consistently explored the learning benefits emerging from ER activities relevant mainly to transversal skills such as metacognition, problem solving, programming, and collaboration (e.g., Bers 2007; Castledine and Chalmers 2011; Atmatzidou and Demetriadis 2012). However, regarding the impact of ER activities on students’ skills, relevant studies do not converge, ascertain studies report skill improvement (e.g., Castledine and Chalmers 2011), while others, by contrast, conclude that the identified benefits—if any—do not reach the level of statistical significance (e.g., Turner and Hill 2007). Most important, studies do not always present in detail the degree of teacher guidance required to support skill development and do not usually analyze the impact of guidance as an independent factor.

The present study posits that a key factor in students’ skill development is the degree of guidance provided by teachers for the implementation of activities. Varying the degree of external guidance certainly means that skills will be developed in different degrees and also that this development might be more or less identifiable by research instruments and methods. We see this issue as strongly relevant to the “strong vs. minimal” guidance argumentation in the context of constructivist learning activities (Kirschner et al. 2006). Thus, the current study introduces teacher guidance as an independent variable and explores its impact on students’ development of MC and PS skills in the context of ER activities. Two cohorts of students (high and elementary school) participated in guided ER activities using Lego NXT robotics tools. Students from each cohort were distributed in two conditions with varied teacher guidance: one with a high coercion approach, which prompts students to follow specific MC and PS strategies and answer in writing (“strong”), and one with a low coercion approach, which does not oblige students to follow specific strategies and answer in writing (“minimal”). Data were recorded with various instruments, and their analysis indicated that, while students’ skills were improved in both conditions, the improvement reached the level of statistical significance only in the “strong” guidance condition.

Overall, in the following, we present: first, an introductory background section on ER learning activities and the development of MC and PS skills, concluding with the research questions of the study; next, the method details and results; and finally, a discussion on the interpretation of the findings.

Background

ER is being introduced in many learning contexts as an innovative learning tool (Alimisis 2014; Eguchi 2014) that changes classrooms by supporting students to (a) enhance their higher-order thinking skills, (b) create multiple representations of their understanding, (c) communicate and collaborate with each other, and (d) develop their learning, while solving authentic complex problems (Blanchard et al. 2010; Çalik et al. 2014, 2015). Drawing mainly on the theoretical perspective of constructionism (a flavor of classical constructivism that suggests promoting children learning through meaningful construction of artifacts (Papert 1991)), ER activities encourage students to become active learners, construct their own new knowledge, and develop essential mental skills by acting as researchers (Gura 2007) and learning through play (Hussain et al. 2006; Atmatzidou et al. 2008). ER is a typical problem-based learning activity revolving around the investigation and resolution of a complex real-world problem. Building and programming a robot to do even a simple mission can be an intriguing task for the students’ creativity and PS ability, facilitating the construction of students’ own learning (Druin and Hendler 2000).

A literature review about robotics in schools has led to the conclusion that, while ER has an enormous potential as a learning tool, there is limited empirical evidence to prove the impact on the K-12 curriculum (Benitti 2012). Indeed, many studies report a positive ER impact on the development of various skills of the students, such as critical thinking (Ricca et al. 2006; Blanchard et al. 2010), problem solving (Turner and Hill 2007; Castledine and Chalmers 2011; Bers et al. 2014), computational thinking (Bers et al. 2014; Leonard et al. 2016; Atmatzidou and Demetriadis 2016), collaboration (Atmatzidou and Demetriadis 2012), metacognition (Ishii et al. 2006; Lin and Liu 2011; La Paglia et al. 2011), and programming (Bers 2007; Atmatzidou et al. 2008; Alimisis 2009). However, other researchers point out that the development of PS skills in ER activities is not obvious (Hussain et al. 2006; Benitti 2012), while others still report failure to identify any statistically significant influence on the development of MC skills (McWhorter 2008; Gaudiello and Zibetti 2013). As the development of students’ MC and PS skills is a common theme transcending the ER literature, we further explore this issue in the following section.

Metacognitive and Problem-Solving Skills in ER Activities

MC and PS skills are different, but strongly interrelated. The development of both these types of skills in ER activities has been the focus of many studies (most commonly exploring the PS skills; less often the MC skills). MC is defined as “thinking about thinking” (Miller et al. 1970; Flavell 1979) and includes two main types of skills: knowledge of knowledge and regulation of knowledge (Brown 1978). Over the past years, MC has been recognized as one of the most relevant predictors of accomplishing complex learning tasks (Van der Stel and Veenman 2010), as well as a powerful strategy for improving student learning (Chin and Brown 2000). A metacognitive thinker knows when and how he learns best, applies strategies to overcome obstacles to learning (Flavell 1979), and regulates the solution processes (Schoenfeld 1992).

PS is considered as the most important cognitive activity in which students are asked to apply knowledge and monitor behavior in order to solve problems (Jonassen 2000). Learning to solve problems helps students learn how to monitor their understanding, recognize when they have a gap in knowledge (Chi and Bassok 1989), and understand why the content is being learned and how it is applicable (Barrows 1996). However, many researchers have shown that, despite the instructions provided, students still have difficulty in solving problems (Lorenzo 2005), and others argue that it is important to make PS strategies part of the students’ thinking processes (Fülöp 2015).

In the process of building and enhancing PS skills, metacognition is a key element (Jacobse and Harskamp 2012; Huang et al. 2014), and, as Du Toit and Kotze (2009) have mentioned, learners with superior MC abilities are better problem-solvers. Many studies argue that MC processes help students overcome obstacles that arise during the PS task (Stillman and Galbraith 1998; Pugalee 2001) and also improve their performance in PS (Goos and Galbraith 1996; Kramarski and Mevarech 1997). At the same time, PS activities provide ideal opportunities for students to reflect on and analyze their thinking (Panaoura and Philippou 2003; Du Toit and Kotze 2009) and to enhance their MC strategies (Martin et al. 1998; Siegel 2012).

Research reports positive outcomes demonstrating evidence on the potential of ER to improve students’ MC and PS abilities, as shown in the overview Table 1. Many researchers have shown that ER activities enhanced a variety of MC, cognitive, and PS skills (Table 1: studies 3–7, 9, 10, and 12); some have even reported significant positive learning impact (Table 1: studies 8 and 11). However, although there are positive outcomes according to the literature, a more in-depth analysis is needed on how to support the students’ development of MC skills (Table 1: study 10). Moreover, most of the studies have a small sample and many researchers mention that further research is needed on a larger sample (Table 1: studies 7 and 11). Another important observation in the literature is that certain studies implemented strong guidance to support students’ MC and PS skills development (Table 1: studies 1, 2, 5, and 12), while others report a rather minimal guidance (Table 1: studies 3–4 and 6–11).

Table 1 Context of the studies and major findings
Full size table

Table 1 summarizes the basic aspects of the above studies on how to support MC and PS skills, and Table 2 classifies them by important findings and methodology. “Minimal” and “strong” guidance studies are classified by the degree of the teacher’s instructional interventions to support the students’ development of MC and PS skills. Specifically, minimal guidance is a low coercion approach for the MC and PS training of students based on verbal prompting, but not compelling students to externalize their reflections in writing, whereas strong guidance is a high coercion approach based on obliging students to follow specific strategies for MC and PS training and requiring them to externalize in writing their reflections on how they apply the training strategies.

Table 2 Classification of studies by important findings and methodology
Full size table

Strong Vs. Minimal Guidance: the Role and Impact of Teacher Guidance

Different suggestions have been made in literature concerning the role and the impact of guidance—no, minimal or strong—on the student learning process. Researchers indicate that minimally guided instruction is less effective and less efficient than instructional approaches that place a strong emphasis on guidance of the student learning process (Anewalt 2002; Papadopoulos et al. 2011). In particular, Kirschner et al. (2006) argue that, although unguided or minimally guided instructional approaches are very popular and intuitively appealing, they are less effective and efficient for novices than guided instructional approaches, because they ignore the structures that constitute human cognitive architecture (Schmidt et al. 2007). However, other researchers argue that strong assignments are expected to induce increased course workload, which might intimidate students and educators alike (Anewalt 2002). Others still report that problem-based learning is itself an instructional approach that allows for flexible adaptation of guidance, and therefore, no further guidance is required (Schmidt et al. 2007), and also that students learn best when instructed with no or minimal guidance, because they must discover or construct some or all of the essential information for themselves, rather than having all the essential information presented to them and being asked to practice using it. These differences encourage deeper research which will determine with greater precision the level of guidance required by learners through the systematic design of instructional experiments (Sweller et al. 2007).

Significance of the Study

From the above, we conclude that (a) the educational community expects ER activities to help students develop their MC and PS skills (e.g., Ishii et al. 2006; Lin and Liu 2011); (b) relevant studies have explored the potential of ER to advance students’ MC abilities and PS skills; however, no definite answers have been provided, so researchers encourage further investigation (e.g., Benitti 2012; Gaudiello and Zibetti 2013); (c) the debate on the required level of guidance (minimal vs. strong) in constructivist learning settings seems to be quite relevant in the context of ER. The significance of the study lies in the fact that it provides further research evidence on the impact that a varied degree of teacher guidance may have on students’ skill development. The outcomes of the study are of interest for the teacher who needs to take informed decisions when implementing some guidance strategy in the context of ER activities (or other similar constructivist learning approaches).

Research Questions

Within this context, the research questions that this study poses are as follows:

  1. i.

    Is strong support decisive or are skills developed equally well if support remains at minimal level?

  2. ii.

    How does the degree of guidance support students’ metacognitive and problem-solving skills develop across age groups and genders in ER activities?

Method

Participants

For the purpose of our study, we conducted two consecutive robotic training seminars in public schools in Greece. In total, 52 students of two different school levels (Elementary and High) participated in the study. Specifically:

  • In the first seminar, there took part 22 students (10 boys and 12 girls) of 10th grade (ages 15–16) of a High School in Thessaloniki.

  • In the second seminar, there participated 30 students (19 boys and 11 girls) of 6th grade (ages 11–12) of an Elementary School in Kilkis.

The seminars were organized and supervised by the main researchers (authors of this work), whereas trained postgraduate students (“trainers”) assisted with the practicalities of the activity. Both the main researchers and the trained postgraduate students were present at the sessions of the two seminars and followed the same instructions thus ensuring equivalence of the training provided. In both seminars, the Lego Mindstorms NXT 2.0 educational robotics tool was used.

Metacognitive and Problem-Solving Guidance Protocol

In this study, we proposed a guidance protocol that refers to the teachers’ instructional interventions for supporting the development of the students’ MC and PS skills in ER activities. Below, we present our proposed MC and PS Guidance Protocol:

  • Prompting: the instructor, based on the Schoenfeld model (1992), had the role of facilitator and consultant, providing support in the form of hints, prompts, feedback etc. During the activities, she would often prompt the students with the following questions: What exactly are you doing? Can you describe it? Why are you doing it? How does it help you?

  • Students were prompted to externalize their thinking through think-aloud protocols.

  • Students were guided to externalize in writing their reflections on how they applied the MC and PS strategies.

The guidance was based on steps and questions that helped students understand the problem, and plan, implement, and evaluate the solution according to Polya’s methodology (1945). We also handed out questions based on Gama’s (2004) model MIRA, in order to guide the students’ thinking and to improve their MC skills. We continuously motivated students to systematically engage in writing tasks since, as many researchers argue, writing down thoughts and arguments can trigger additional cognitive activity that results in better learning outcomes (Menary 2007; Papadopoulos et al. 2011).

Below, we analyze the protocol of MC and PS guidance with the questions and prompts that guided students to think about thinking during solving problems (Table 3):

Table 3 Metacognitive and problem-solving guidance protocol
Full size table

Design

The study implemented a quasi-experimental 1 × 2 design, with one factor: level of MC/PS guidance and two conditions: “minimal” vs. “strong” guidance. The specific differences between the two conditions are described below.

  1. (a)

    The “minimal” guidance group followed a low coercion approach for the MC and PS training of the students. In every session, the students were given a worksheet with activities of increasing difficulty. The instructor, as facilitator and consultant, provided support in the form of hints, prompts, feedback etc. During the activities, she would often prompt the students with questions based on the Schoenfeld model (1992) such as: What exactly are you doing? Can you describe it? Why are you doing it? She also prompted but did not compel students to follow strategies. More specifically, she:

    • prompted students verbally, rather than in writing and analytically, and without the strict guidance of our proposed MC and PS Guidance Protocol, to externalize representations of MC and PS training procedures;

    • did not oblige students to externalize their reflections in writing;

    • did not prompt students to analyze their thinking on the process of the solution.

  2. (b)

    The “strong” guidance group followed a high coercion approach for the MC and PS training of the students. The students were guided by worksheets which were structured so as to lead them gradually to solve the problems and prompted them to follow specific MC and PS strategies, based on our proposed Guidance Protocol. The strong guidance was based on questions that guided the students’ thinking: (i) to understand the problem; (ii) to design, implement, and evaluate the solution based on the proposed protocol; and (iii) to choose strategies that would improve their MC and PS skills.

Students were also required to externalize in writing their reflections on how they applied the MC and PS strategies. That is, during solving problems, the students were asked to answer in writing specific prompts that made them reflect on their MC and PS abilities. In order to moderate the workload induced by the constant writing, which poses a threat to students’ motivation for engagement, and also to prevent them from getting tired of our interventions, the strong guidance faded out in the last activity of each session.

To summarize, the differences between the “strong” and the “minimal” guidance group lay in the MC and PS training. Although both groups studied in teams, learned the same Lego NXT-G commands (blocks) and procedures (myblocks), practiced the same programming problems, took part in the final challenge, and were helped by the instructor, only the “strong” group had deliberately strong guidance for the MC and PS training.

Procedure

Participants were divided into two groups, the “minimal” guidance group and the “strong” guidance group. In both groups, students worked collaboratively in groups of 3 or 4 members. Students were assigned certain roles (analyzer, programmer, coordinator/editor, and evaluator) which were meant to help them collaborate more efficiently.

In both seminars, there were in total seven sessions. The first six sessions lasted two and a half hours each and the final challenge lasted 3 h (total 18 h). In the first six sessions, the students were trained and became familiar with the basic blocks and abilities of the robot, completing realistic-authentic problems which were meaningful and interesting to them and which excited and encouraged them to participate (Rusk et al. 2008). In the seventh and final session, a complex problem was handed out as a final challenge. Moreover, in the first three sessions, we used the jigsaw method as a collaboration script (Barkley et al. 2014). According to this method, each group member works with a group of experts and specializes in one area of the object being taught, and then each expert member returns to their original group and transfers the new knowledge obtained in the previous step.

  • The sessions were as follows:

  • 1st session: There was a short presentation about the robot and its abilities. The students became familiar with the Lego Mindstorms NXT robot, the Lego NXT-G programming environment, and their basic functions. After that, they were handed out the Student Profile Questionnaire and the pre-MAI questionnaire which had to be filled individually. Then, the students began the activities, learning how to use the motors, the touch sensor, the sound sensor, and the ultrasonic sensor.

  • 2nd session: Students became familiar with basic programming concepts. They learned about the conditional structure, the structure of loop, the operation of light sensor, and some basic features of Lego NXT, such as displaying images on the screen of the robot, the conversion of numbers to text, and the wait block.

  • 3rd session: Students became acquainted with the variables and the basic arithmetic operators.

  • 4th session: Students became familiar with the use of the lamp block, parallel programming, and creation of reusable subprograms.

  • 5th and 6th sessions: Students were handed out activities for familiarization with and integration of concepts in complex authentic problems with graduated difficulty, such as a car that follows the rules of traffic, a guitar that plays music, an alarm etc.

  • 7th session: In the final challenge, the groups were given activities which were demanding, yet within their reach, and clear rules were set for grading and appointing the winner. During the given time, each group had to prepare their solution to the given scenario. Upon completion, we analyzed the strengths and weaknesses in the solutions and in the strategies adopted by each group.

After the completion of each seminar, four instruments were used to capture the students’ level of MC and PS skills and also their views regarding the educational robotic training experience. These were as follows: (a) a think-aloud protocol implementation, (b) a post-MAI questionnaire, (c) a student opinion questionnaire, and finally, (d) a semi-structured interview. Overall, the structure and the various data collection instruments of the seminars are presented below in Figs. 1 and 2.

Fig. 1
figure 1

Structure of the seminars

Full size image
Fig. 2
figure 2

Data collection instruments in the seminars

Full size image

Measures, Instruments, and Data Analysis

The instruments that we used to collect evaluation data (and respective measures) are as follows:

  • Student Profile Questionnaire: An individual questionnaire was administered at the beginning of each seminar. The student profile questionnaire recorded demographic data (such as student gender), the students’ background in computer use (e.g., frequency of computer use, computer experience, knowledge about programming), and experience with robotics.

  • Two intermediate Metacognitive Awareness Inventory (MAI) questionnaires (pre-MAI and post-MAI): Pre-MAI was handed out to students at the beginning and post-MAI at the end of each seminar in order to investigate the development of the students’ metacognitive awareness. Before giving the pre- and the post-MAI, we made clear to the students the context in which the questions were addressed and we specifically mentioned that they had to answer truly by noting down their thoughts on how they studied, solved problems, and organized their activities. As pre-MAI is a measurement reflecting students’ MC skills early in the seminars, we use it as a covariate in our statistical analysis.

The MAI (Schraw and Dennison 1994) was developed to measure metacognitive awareness (both knowledge about cognition and regulation of cognition). Students responded to these items by indicating degrees of agreement with each statement on a 5-point Likert scale (1 = “strongly disagree,” 2 = “disagree,” 3 = “uncertain,” 4 = “agree,” 5 = “strongly agree”). Internal consistency (reliability) statistics range from r = .90 to r = .95 (Dennison 1997; Sperling et al. 2004; Nosratinia et al. 2014). In our study, the inter-rater reliability was high for the pre-test (intraclass correlation coefficient = .856), and the post-test (ICC = .924) MAI questionnaire. The reliability and validity of the MAI instrument have been documented by the results of several studies in the literature (e.g., Panaoura and Philippou 2003; Sperling et al. 2004; Akin et al. 2007; Nosratinia et al. 2014).

  • Think-aloud protocol: After the training, the students were asked individually to describe aloud the process they would follow to solve a certain robot programming task. They were prompted to externalize their thoughts on the process and the MC and PS strategies that they used in their solution. We gave five different tasks sharing the same main characteristics, such as movement, repetition, and variable, but with differences in surface characteristics and in the wording of the task, to prevent the students from transferring the solutions among them. We consider the think-aloud measurement as an indicator of the students’ level of PS and MC skills development after training. The assessment of the students’ answers in that protocol was based on a graded criterion instrument (rubric) using a 4-point Likert scale (1 = “unsatisfactory,” 2 = “quite satisfactory,” 3 = “satisfactory,” 4 = “excellent”). In our study, the reliability was high for the think-aloud measurement (Cronbach’s alpha = .904). The problem given to the students has been constructed by domain experts in order to provide the opportunity to students to express their thinking regarding both their problem solving and their metacognitive strategy.

  • Student Opinion Questionnaires: A student opinion questionnaire was handed out to be filled individually after the completion of the training. The instrument recorded the students’ subjective views and opinions regarding the outcomes of the learning experience on the following key aspects: (a) developing PS and MC skills, (b) understanding basic programming concepts, (c) the students’ collaboration in groups, and (d) likes and dislikes relevant to the educational robotics activity. Responses in the attitude questionnaire were expressed in a Likert scale, from 1 to 5 (1 = “strongly disagree or no,” 2 = “rather disagree or rather no,” 3 = “uncertain or undecided,” 4 = “rather agree or rather yes,” 5 = “strongly agree or yes”).

  • Semi-structured interview: After the end of the think-aloud activity, a semi-structured interview recorded the students’ opinion on key aspects of the activity (as described above in the Student Opinion Questionnaire section). We used content analysis on collected data to identify the views that dominated in the statements of the students.

  • Observations: During the activities, both the supervising researcher and the trainers monitored the students’ work by taking notes on structured observation sheets. Once the session was complete, they discussed extensively on their observations and expanded their notes.

Results

MAI Questionnaire

  1. (a)

    Table 4 presents the results from administering the pre- and post-MAI instruments in student cohorts of Elementary and High School level and also as a “total” (merging together students in the same guidance condition independent of their age group). In our analysis, we first applied normality (Shapiro-Wilks) and variance (Levene) controls on available data, with the results indicating statistical non-significance in most cases suggesting that sample data come from normal distributions and populations with the same variance, therefore being appropriate for parametric test analysis. t test was applied in most cases to pairwise control for significance the score of the various student groups. In Table 4, the last column (“Statistics”) displays row-wise the “paired t test” results comparing between the pre-MAI and post-MAI student scores, for guidance condition and school level. Additionally column-wise, the outcomes of “t test independent samples” controls are presented, comparing between the scores of different guidance condition groups at the same school level. Only in one case (high-minimal, pre-MAI, N = 10), the sample violated the normality criterion (t = 0.831, p = 0.035) and the non-parametric Wilcoxon and Mann-Whitney controls were applied in this case, respectively. Furthermore, a one-way ANCOVA test was conducted to compare between the post-MAI scores across the “minimal” and “strong” condition (with pre-MAI as a covariate). Worth noticing is that ANCOVA is also implemented in the case of the “High” school level where the “High-minimal” sample appears not to conform to the normality criterion. This is because ANCOVA in general is considered as robust statistical control for most distributions under most circumstances outperforming respective non-parametric controls, for example “Mann-Whitney” (see, e.g., Vickers 2005). Additionally, ANCOVA has the advantage of providing a measure for the impact of the treatment, namely the “effect size.”

Table 4 MAI scores (pre- and post-MAI) for the total population and for each school level
Full size table
  1. (b)

    Table 5 presents statistical controls applied to the students’ pre- and post-MAI scores between Elementary and High School level (mostly t test except for the “High-minimal, pre-MAI” group where non-parametric “Mann-Whitney” is applied).

Table 5 Comparing pre- and post-MAI scores between Elementary and High School
Full size table
  1. (c)

    Table 6 presents statistical controls applied to the students’ MAI scores across different gender groups.

Table 6 Comparing MAI scores between gender groups in total population
Full size table

Think-Aloud Protocol

  1. (d)

    Table 7 presents statistical controls applied to the students’ think-aloud scores focusing on the MC and PS strategies that they used in their solution for the total population and for each school level.

Table 7 Evaluation on think-aloud: impact on students’ MC and PS skills
Full size table

(e)Table 8 presents statistical controls applied to the students’ think-aloud scores across different gender groups.

Table 8 Comparing think-aloud scores between gender groups in total population
Full size table

Students Opinion Questionnaires and Semi-Structured Interview

Data from Students Opinion Questionnaires and Semi-structured interviews (content analysis) helped us understand the students’ opinions regarding the overall activity. The key findings can be summarized as follows:

  1. (i)

    The students’ subjective impression was that they acquired certain MC skills. The students reported an improvement of their MC skills. In addition, as is obvious from Table 9, this improvement was more significant for the students of the “strong” group in both school levels. They reported that, when they studied, they stopped and read the parts they did not understand (“minimal”: M = 4.17, SD = .96, “strong”: M = 4.68, SD = .48). They also said that they tried to find out keywords or to make plans during an activity (“minimal”: M = 2.96, SD = 1.46, “strong”: M = 3.96, SD = .92), and that they stopped studying from time to time to conclude what they had learnt (“minimal”: M = 2.96, SD = 1.57, “strong”: M = 3.93, SD = 1.12). Finally, during procedures, they tried to compare what they already knew with the new information (“minimal”: M = 2.88, SD = 1.12, “strong”: M = 4.29, SD = .98)

  2. (ii)

    Regarding the PS skills, the students stated that they increased these skills through ER activities (Table 9). The “strong” group students showed a statistically more significant improvement in PS skills in comparison to the “minimal” group students. They also reported that it was very useful for them to realize that, when solving problems, they could use a thinking process that could also be applied to other topics, such as mathematics.

  3. (iii)

    The “strong” group students understood the value of guidance. They reported that the guidelines, which led them gradually to solve the problems and prompted them to follow specific strategies, were very important for the acquisition and development of MC and PS skills. Also, they believed that, even though the requirement to externalize in writing their reflections on the solution process was very tiring and unpleasant, it was useful as it led them to a deeper understanding. Moreover, they pointed out the importance of the fading of guidance in the last activity of every session, because that allowed them to take control of activities and avoid the boring strict guidance. They also suggested that they would prefer the fade-out method to be applied to more activities in the last sessions.

  4. (iv)

    The students of both groups, “minimal” (M = 4.25, SD = .68) and “strong” (M = 4.50, SD = .69), stated that they became familiar with basic programming constructs. In particular, High School students mentioned that their engagement with robots helped them to better understand basic programming concepts, such as iteration and selection, which had not been clarified during their previous experiences with logo. Similarly, it is worth noting that Elementary School students found it very interesting that they started to learn how to program.

  5. (v)

    Regarding collaboration, the students of both school levels pointed out that there was a satisfactory assignment of roles during the sessions (“minimal”: M = 3.37, SD = .89, “strong”: M = 4.46, SD = .53) and also, they stated that the jigsaw script motivated them to enhance their collaboration, since they felt their contribution was important to the team. Moreover, students reported that they were helped to learn faster the new information (“minimal”: M = 3.63, SD = 1.28, “strong”: M = 4.68, SD = .48), and that kind of collaboration helped everyone to participate in the procedure (“minimal”: M = 3.83, SD = 1.37, “strong”: M = 4.36, SD = .83). As a High School student stated: “I liked working both with the main and the individual groups, and I found my participation particularly useful to my team.” And also, as an Elementary School student stated: “I liked the fact that role assignment meant I could work with my classmates, while in other cases it was difficult to do so.”

  6. (vi)

    Finally, the students found that the robotics experience was an innovative and attractive way of learning (“minimal”: M = 4.83, SD = .38, “strong”: M = 4.96, SD = .19), reporting that they would like to engage in more challenging tasks.

Table 9 Students’ opinion questionnaires
Full size table

Discussion

The current work investigates whether the appropriate guidance during solving authentic problems in educational robotic activities can improve students’ MC and PS skills. The study provides evidence from evaluation instruments administered during the activity, thus offering a picture of how MC and PS skills develop as students’ work progresses. The participants were divided into two groups, the “minimal” guidance group, which follows a low coercion approach for the MC and PS of the students, and the “strong” guidance group, which follows a high coercion approach based on our proposed metacognitive and problem-solving guidance protocol. The students’ MC and PS skills were evaluated by means of different assessment instruments based on the following: (a) the students’ subjective views (MAI questionnaire, Student Opinion Questionnaires), (b) the students’ verbalized thoughts about a specific problem with think-aloud protocol, and finally, (c) the researchers’ observations and qualitative data from the students’ semi-structured interviews which helped triangulate data and understand their meaning more deeply.

Metacognitive Skills

A first observation is that students develop MC skills at the end of their training independently of both school level and degree of guidance (comparing pre- and post-MAI scores between Elementary and High School, and “minimal” and “strong” guidance groups, in Tables 4 and 5). This is in line with studies emphasizing that training with robotics kits may help to develop students’ awareness and metacognitive abilities (La Paglia et al. 2011).

Furthermore, we see that although both groups improve their MC skills at the end of the seminar, only for the “strong” group did this improvement reach a level of statistical significance (Table 4, Fig. 3). From Table 4, we observe that the “strong” group improved the MC skills in a statistically more important degree than the “minimal” group. And also, focusing on each school level, we can see that at the end of the seminars, both “strong” groups of Elementary and High school reached the same level of MC skills (Elementary: M = 4.32 (SD = .28), High: M = 4.31 (SD = .44)), but only the “strong” in the Elementary School reached a statistically significant improvement. We can assume that this is because the High School students had a high score in pre-MAI (Table 4: High: M = 4.05 (SD = .34)), which means that the older students had a high metacognition level at the beginning of the seminars. These results showed that the degree of guidance had a large and significant overall effect on the development of the students’ skills, F(1, 49) = 19.728, p < .001, η2 = .287. The effect size (η2) showed that 29% of the variance in the development of the students’ skills is due to the level of guidance.

Fig. 3
figure 3

Means of pre- and post-MAI (strong vs. minimal guidance)

Full size image

Keeping this in mind, we explore the data in Table 7. According to the evaluation of think-aloud protocol scores of the students’ oral answers in the description of a problem’s solution, we observe that, in the total population and in each school level, the “strong” groups show a statistically significantly higher score than the “minimal” groups. Knowing that the only difference between the two groups is the level of guidance, we can conclude that the appropriate guidance, which prompts students to follow specific MC and PS strategies and to answer in writing, leads to a higher development of these skills. Moreover, we observe that, even if both groups began from the same level (High: M = 1.40, SD = .36, Elementary: M = 1.30, SD = .37), the “strong” group in High school improved the MC skills to a higher degree (M = 2.65, SD = .61) than the same group of Elementary School (M = 1.86, SD = .40), and this difference was actually statistically significant. One possible explanation might be that the older students have higher awareness of the knowledge of their knowledge.

Moving on to Table 9, we observe that the students’ subjective impression showed that they acquired MC skills. The total population increased the MC skills, but as is obvious from Table 9, only the “strong” groups showed significant improvement in both school levels.

From all the instruments, that we used to investigate the development of the students’ MC awareness (MAI, think-aloud protocol, Student Opinion Questionnaires), we conclude that the students improve MC skills, but only the “strong” groups show a statistically significant improvement. These findings are in line with two recent studies, the first showing that students’ performances in metacognitive tasks were statistically significantly improved while they “played” with the robot (Keren and Fridin 2014), and the second arguing that using writing practices can be a useful tool for helping students improve metacognition and engage in a higher level of learning through robotics (Huang et al. 2014).

Finally, we focus on the analysis of scores between gender groups according to the evaluation of MAI and think-aloud scores (Tables 6 and 8). A key conclusion here is that boys and girls reach the same MC skills level, with girls showing a higher increase which, however, is not significantly different from that of boys.

Problem-Solving Skills

Reflecting further on the data of Table 7, which present the think-aloud scores of the students’ answers, we observe that independently of school level, the students developed PS skills at the end of the sessions. The students in the “strong” group achieved a higher level of these skills and even the difference between the two groups is statistically significant.

Keeping this in mind, we see from the students’ subjective impression (Table 9) that the “strong” groups had a statistically significant more positive impression regarding the acquisition of PS skills, in comparison to the “minimal” groups. Additionally, the students reported that becoming familiar with and applying a specific thinking process in solving problems was very helpful, not only in robotics, but in other subjects too, such as mathematics and physics. As a student of the High School stated, “I found the methodology that Ι followed useful and used it in other courses such as physics, mathematics and chemistry.” The students felt that the most useful strategies were the separation of the requested data and the segmentation of the problem into smaller pieces.

Moreover, by reflecting on the researchers’ observations, we report that (a) in the beginning, the students faced difficulties in the problem-solving process and in using corresponding strategies. However, at the end of the training, interesting solutions were noticed in the students’ answers. Especially the “minimal” group students often asked for further clarification during the problem-solving process in the activities. By contrast, students in the “strong” group assimilated the strategies more easily and used them in the activities often without any intervention from the trainers. This corroborates the findings in Table 7, where the “strong” group seems to reflect their thoughts on the process solution significantly better than the “minimal” group. (b) Older adolescents (High School) gave more integrated and enhanced solutions than the younger students (Elementary School), and this is perhaps related to the cognitive development level of the High School students. This further supports the data in Table 7, where the older students reach higher scores than the younger ones.

These findings show that ER activities: (a) when conducted with an appropriate protocol that guides students’ thinking to apply PS strategies and to provide written answers, can improve students’ PS skills. This improvement may be due to the improvement of MC skills, and not only to the PS training, as Du Toit and Kotze (2009) argue that the students’ MC abilities help them to be better problem solvers; (b) developed PS skills at the end of the sessions independently of the level of guidance, which means that ER is a learning tool which engages students to develop their PS skills. This is in line with the views of Lo Ting-kau (1992), who has pointed out that the use of robot materials in the classroom may provide a rich environment for problem solving.

Next, moving on to Table 8 (think-aloud scores), we see that evaluating students’ PS skills orally showed that the development of PS skills happens in the same way for both genders. Similarly, Lai’s study (1993) showed that girls and boys increased their higher-order thinking score by the same amount during the robotics activities.

Robotics, Collaboration, and Programming

Finally, some more interesting evidence emerges in measures. As we concluded from the semi-structured interviews and the researchers’ observations, the students were really enthusiastic about robotics during the sessions.

Moving on to collaboration in the High School, from the observations, it was found that there was a satisfactory assignment of roles during the sessions. The participation of students was enhanced by using the jigsaw method, since they felt that their contribution was important to the team. In the Elementary School, it was also observed that the distribution of roles helped students to collaborate. It took some time until the youngest students became familiar with their roles and followed the jigsaw method, but with the support of the coaches, even the most reluctant children felt confident when contributing to the team.

Finally, as far as programming is concerned, the results of the content analysis of the semi-structured interviews in the High School show that the students, with their engagement with robots, felt that they understood much better basic programming concepts such as iteration and selection, which had not been clarified during their previous experiences with logo. Similarly, in the Elementary School, it is worth noting that students found it very interesting that they started to learn how to program and they also said that it seemed very useful for them to realize that they could use a thinking process when solving problems that could also be applied to other topics such as mathematics.

Study Limitations and Future Research

One key limitation in our study is the small sample size. However, we present our study in line with relevant studies (see Table 1, e.g., Huang et al. 2014; Keren and Fridin 2014) with group sizes comparable to ours, and it is our intention to replicate the study with larger samples in the future. Another point of interest is the impact that the length of the seminars (training time) might have on student skill development. Applying the skill guidance protocol for a longer period will provide the opportunity to investigate the skill development and retention in relation to the training time. Moreover, we suggest that the fade-out technique deserves further exploration to provide evidence on how to avoid the negative consequences of the continuous and long-lasting strict guidance, without decreasing its learning benefits. Future educational research might as well explore the suggested “strong guidance” approach in relation to the development of still other type of student skills such as collaborative and/or computational thinking skills. Moreover, as the proposed metacognitive and problem-solving guidance protocol is not applicable only to robotics but to a broader context of problem-solving based learning, it would be interesting to see the results of its implementation in other domains.

Conclusion

Overall, this study provides evidence that the ER activities, through the appropriate guidance, which means following specific prompting and responding in writing, improve the students’ skills in a statistically significant degree. And as is obvious, ER can be a vehicle for the development of metacognitive and problem-solving skills in students of Elementary and High School grades.

Reflecting further on the results, we report the most important of them as follows: (a) Students independently of their age developed their MC and PS skills through the robotics activities. This conclusion is in line with studies emphasizing that training with robotics kits may help to develop students’ awareness and metacognitive abilities (La Paglia et al. 2011). (b) The guidance whereby students, independently of their age, are prompted to follow specific strategies and are required to externalize in writing their reflections on how they solve a problem can be a useful tool that helps students to improve the MC and PS skills in a statistically significant degree. (c) Both girls and boys reach the same level of MC and PS skills, with girls appearing to increase these skills more than boys, but with no significant difference. Finally, (d) robotics is an attractive and effective way of learning. Students were really enthusiastic during the sessions and the assignment of roles helped them to collaborate. Their engagement with robots helped them to better understand basic programming concepts.

Based on this research, we argue that ER is a powerful teaching and learning tool, whose learning benefits are maximized through an appropriate guidance framework. In this context, we recommend dividing students into groups, each member of which should be assigned a distinct role and guided through the worksheets to follow detailed instructions. Next, students should be led gradually to solve the problems following specific MC and PS strategies and should be prompted to respond in writing. One needs to pay special attention to the fading-out of the strong guidance in order to reduce the students’ workload and allow them to take control of the strategies they follow during problem solving. This guidance framework could help setting up a metacognitively powerful learning environment.