Keywords

1 Introduction

One of the main goals of science and engineering education is to guide students in becoming proficient problem solvers. Metacognitive and self-regulating abilities are needed to become a skilled problem solver in addition to sufficient content and procedural knowledge in different disciplines [11]. These abilities help students to regulate their solving process and to guide their decisions on which approach to follow when solving a problem [19]. This study investigates whether the development of metacognition can be stimulated through reflection on the solving process. Moreover, we consider whether there are preliminary indications that discipline-specific reflection contributes more strongly to the development of metacognitive abilities than generic reflection. We focus on the use of metacognitive knowledge and skills when solving physics problems, in particular problems in Newtonian mechanics.

1.1 Metacognition

Based on the well-known framework of Flavell [4], metacognition can be described as “students’ knowledge about their processes of cognition and the ability to control and monitor those processes as a function of the feedback received via outcomes of learning” [6]. Metacognitive abilities consist of two major components: metacognitive knowledge and metacognitive skills or control [6, 7, 23, 26]. Metacognitive knowledge refers to the knowledge and beliefs students have about their own cognition, the cognitive strategies they use and how these strategies interact with a cognitive task. Moreover, Flavell recognizes three subcategories in metacognitive knowledge: knowledge about persons, tasks, and strategies [4, 23, 26]. In the context of problem solving, the latter is the most important one: it includes knowledge about when, why, and how certain strategies can be applied to achieve certain goals [4, 26]. Metacognitive skills refer to the strategies students use to plan, monitor, control, and evaluate their cognitive activities to ensure effective learning [4, 26].

The notion of metacognition is closely related to self-regulated learning. There is an extensive literature on these two notions and the relationship between them. According to Schraw’s model metacognition is, besides cognition and motivation, one of the components of self-regulated learning [20]. Since considering students’ motivation is not the aim of this study, we look through the lens of metacognition in this contribution.

1.2 Role of Metacognition in Problem Solving

Problem solving is a complex process. Many generic and discipline-specific problem solving models exist. In the context of physics problem solving, the logical problem solving model of the University of Minnesota has been developed to help students improve their understanding of physics problem solving [6].

To become skilled problem solvers students need a solid basis of content and procedural knowledge in different disciplines. Indeed, a profound conceptual understanding of the problem and knowledge of the relevant procedures is a prerequisite to be able to solve a problem [8, 11, 13]. This knowledge and these skills, however, do not help students to make decisions on which actions to undertake when solving a non-routine problem. Here students’ metacognitive abilities play an important role, since they help students to guide their decisions on which approach to follow [19] and to monitor their progress [1, 7]. The research of Schoenfeld [19] showed that experienced problem solvers spend relatively more time on metacognitive processes, such as analysing the problem and reflecting on the solution process, than novice problem solvers. Comparatively, novice problem solvers spend most of their time on cognitive processes, such as finding a solution plan and calculating. The research of, e.g., Rozencwajg [18] confirmed that students are more successful in problem solving when they show a higher level of metacognitive abilities.

Berardi-Coletta et al. showed that transfer of learning to new problems is more likely to take place if students acquire information on the solution of a problem via metacognitive processes [1]. Similarly, the study of Kapa showed that training students in metacognition with a focus on the product and the process phase of problem solving (i.e., on the use of metacognitive activities after and during the solving process) is beneficial for near and far transfer (i.e., for transfer to similar and dissimilar problems) [9].

The results above show that fostering students’ metacognitive abilities can contribute to the education of successful problem solvers. Ample research has confirmed that metacognitive abilities positively affect learning outcomes, hence, academic achievement [10, 22, 27].

1.3 Metacognition and Educational Technology

Numerous interventions have been developed to foster students’ self-regulating and metacognitive abilities, many of them use some kind of technology and/or learning analytics [24, 25]. The majority of studies on mobile-learning showed that mobile-learning can enhance self-regulated learning [15]. The research of Garcia Rodicio et al. showed, however, that only offering a minimal or intermediate support system does not improve students’ learning [5]. Students need a broad support, at least if they have to study complex materials with little prior-knowledge on the subject. In the context of our study, we expect students to need a broad support system to improve their conceptual understanding and metacognitive abilities for problem solving as physics and Newtonian mechanics in particular are known to be conceptually challenging.

2 Motivation and Research Questions

As discussed before, many technology-based interventions have been developed to foster students’ self-regulating and metacognitive abilities [24]. Tormey et al. [21] combined a learning diary with a learning analytics dashboard in order to stimulate self-regulating abilities in relation to problem solving. The result is an online application called the Learning Companion, which is carefully grounded in theory. The learning diary provides a predefined list of generic reflection questions related to problem solving. Students can reflect on their solution and the solving process by answering these generic questions after solving any problem. Ample research has indicated, however, that it is more effective to teach self-regulating and metacognitive abilities in a discipline-specific context, rather than covered in a discipline-agnostic package [7, 19, 23]. Therefore, we believe that the concept of the Learning Companion can be augmented by supplementing or replacing the generic reflection questions by discipline-specific, and even topic-specific reflection questions. This forms the motivation to develop the Disciplinary Learning Companion (DLC).

In this contribution, we discuss the concept of the DLC and its implementation in a first-year course on Newtonian mechanics for students in bioscience engineering at KU Leuven (Belgium). The aim of our study is to investigate the possible impact of the DLC on students’ metacognitive abilities, academic achievement, and conceptual understanding. The research question are:

  • RQ1. How is students’ interaction with the DLC related to their metacognitive abilities as measured by a validated questionnaire?

  • RQ2. How is students’ interaction with the DLC related to their academic achievement on a final exam?

  • RQ3. How is students’ interaction with a particular physics topic in the DLC related to their performance on the corresponding exam problem and their conceptual understanding of this exam problem?

3 Concept of the Disciplinary Learning Companion

The idea of the Disciplinary Learning Companion (DLC) is to foster students’ metacognitive abilities for problem solving by triggering reflection on the solving process. The self-reflection is elicited by discipline-specific or even topic-specific reflection questions and personalized feedback. The DLC consists of reflection modules, where each reflection module discusses one particular problem. In this study, we focus on problems in Newtonian mechanics. The reflection questions are structured according to five problem solving dimensions that can also be linked to the logical problem solving model of the University of Minnesota [6]: (1) strategy plan, focusing on setting up a well-considered and complete strategy plan to tackle the problem; (2) concepts, focusing on identifying the relevant discipline-specific concepts needed to solve the problem; (3) mathematical model, focusing on translating the relevant physical laws and concepts into a set of equations; (4) computations, focusing on the necessary computations to solve the mathematical model obtained; and (5) interpretation, focusing on interpreting and evaluating the answer obtained. Each reflection module counts 10–15 reflection questions, such that we expect students to work on it for about 20–30 min.

The reflection questions are multiple-choice questionsFootnote 1, where the answer options include answers based on common student difficulties. For each reflection question, students receive feedback based on their answer. The feedback explains why the answer is (in)correct, what reasoning could lead to the correct answer, why particular solving strategies could be more suited or efficient, and suggests specific actions (e.g., “Would you solve the problem differently with this new information. If yes, try to do so.”). Once students have gone through all topic-specific reflection questions, they can download a model solution to the problem.

To stimulate transfer of acquired concepts and strategies to future problem solving, the last question of the reflection module instructs students to write down a point of attention, i.e., something they have learned by solving the problem or completing the reflection module and want to take with them to future problem solving. This can be, for example, a concept they did not fully understand yet, a common strategy or how they can write something down. This is an open question such that they are stimulated to reflect on what they have learned themselves. However, as students are not always able to come up with a useful point of attention themselves, a list with suggestions, structured according the five problem solving dimensions described above, is provided afterwards.

4 Methods

This section discusses the participants, course design and the three different data sources in our study.

4.1 Participants and Course Design

Since we believe that the development of metacognitive abilities can help students in the transition from secondary to higher education, we implemented the DLC in an introductory physics course for first-year students in bio-engineering science at KU Leuven (Belgium) (\(N \sim 350\)). This course mainly dealt with Newtonian mechanics and consisted of two 1,5 h weekly lectures and one 2 h problem solving session for each chapter. The professor of the course provided the lectures, while teaching assistants guided the problem solving sessions. For each of the nine Newtonian mechanics topics, we selected an additional problem for which we developed a reflection module. Students were instructed to solve this problem and reflect on their solving process using the reflection module after the problem solving session. The goal of this task was to help students process the concepts and strategies discussed in the problem solving session and to prepare for the next problem solving sessions. The task was not graded nor mandatory, but the students were stimulated by the professor as well as the teaching assistants to engage in the reflection modules as part of the learning activities for the course.

4.2 Students’ Interaction with the DLC

To measure students’ interaction with the DLC, we checked how many reflection modules they completed and to what extent. In the context of the study, we only wanted to include “valid” participations to the reflection modules. Hence, we only considered participations to the reflection modules that were completed timely (i.e., at the latest 2 weeks after the next problem solving session), almost fully (i.e., \({\ge }80\%\) of topic-specific reflection questions are answered), and carefully (i.e., \({\ge }10\) min spent to complete the module).

Table 1. Overview of the grouping of students for the three research questions and the number of students in each group. For RQ1 and RQ2 students were divided in four groups depending on how many reflection modules of the DLC they completed (interaction groups). For RQ3 the groups were combined two-by-two, but then split depending on whether students did or did not complete the particular reflection module on angular momentum (angular momentum interaction groups).
Full size table

For RQ1 and RQ2, the students were divided in four “interaction groups” based on the total number of reflection modules completed: 0, 1–3, 4–6, or 7–9 reflection modules. For RQ3, we studied the exam problem on the particular topic of angular momentum in more detail. Therefore, we took into account whether students did or did not complete the reflection module about angular momentum in addition to the total number of reflection modules completed (0–3 or 4–9 modules). In this way, we obtained four “angular momentum interaction groups”. In the analysis of students’ metacognitive abilities (RQ1), we could only include students that participated in both the pre- and the posttest of the MSLQ (\(N=165\)). In the analysis of students’ performance on the exam (RQ2 & RQ3), we included all students that participated to the final exam (\(N=316\)). Table 1 presents the number of students in each (angular momentum) interaction group. Note that for RQ3 only 5 students were in the second group, which made this group too small to include in this part of the analysis. The performance of these students on the exam problem was very diverse, ranging from a zero score to a score of 8/10, such that no trend could be observed in the data of this small group.

4.3 Metacognitive Abilities

To investigate whether there is a relationship between students’ interaction with the DLC and their metacognitive abilities (RQ1), we administered a pretest during the first lecture of the course and a posttest during a lecture at the end of the semester. These tests were based on (the second part of) the Motivated Strategies for Learning Questionnaire (MSLQ) developed by Pintrich and De Groot [16, 17]. In this self-reported questionnaire, students have to assess 50 statements about learning strategies on a 7-point Likert scale (coded 1–7). We translated the statements to Dutch, which is the native language at our university, and made them more concrete for the context of the course.Footnote 2 The 50 statements are originally categorized into 9 subscales: rehearsal, elaboration, organization, critical thinking, metacognitive self-regulation, time and study environment, effort regulation, peer learning, and help seeking. Using a confirmatory factor analysis, we investigated whether the same subscales applied to our adapted questionnaire as interpreted by the participants to the pre- and posttest. Some adjustments to the categorization of the statements were made (see Footnote 2). Among others, the subscales “time and study environment” and “effort regulation” were merged.

4.4 Academic Achievement and Conceptual Understanding

We studied students’ performance on the exam on a quantitative and on a more qualitative level. We considered students’ final exam score for the course and studied their solution to one problem of the exam in more detail. For the latter, we defined a concept score assessing students’ conceptual understanding of this problem. A coding frame was developed that considers whether students recognized the relevant concepts and whether they applied these concepts correctly. In total, five concepts that should be applied to answer the two subquestions of the problem were identified. Each of these concepts was worth one point if recognized as relevant and worth another point if also applied correctly, resulting in a concept score between 0 and 10. For each relevant concept, some criteria were set up to decide when students did or did not recognize it and did or did not apply it correctly. In total, a sample of 25 student solutions to the exam problem was scored by three independent raters and discussed in two rounds. After refining the criteria another sample of five student solutions was rated. Interrater reliability was tested for the two subquestions and yielded a Cohen’s kappa of .88 and .79, respectively, which can be seen as a substantial agreement.

4.5 Data Analysis

Students’ data were linked to each other by using student numbers. Students were asked to fill in their student number in the reflection modules, pretest, and posttest. Moreover, students signed an informed consent before voluntarily participating to the pretest. Filling in the student number was not mandatory to be able to use the reflection modules. However, for each reflection module only a small fraction of students (\({\le }8\%\)) did not provide a (valid) student number. The data were pseudonymised after linking them to each other by replacing the student numbers with unique codes.

5 Results

This section presents the observed relationships and trends in the data obtained in the study. For the analysis, students were grouped based on their interaction with the DLC as discussed in Sect. 4.2 and shown in Table 1.

5.1 Interaction with DLC vs. Metacognitive Abilities (RQ1)

For each subscale of the MSLQ, the pre- and postscore and the normalized change between both tests were compared for the four interaction groups. Kruskal-Wallis rank sum tests showed that there were almost no significant differences between the four interaction groups, except for the postscore on the subscale “time and study environment and effort regulation” (\(\chi ^2 = 10.90, p = .01, df = 3\)). Post-hoc pairwise Wilcoxon tests showed which pairs of groups scored significantly different (see Fig. 1). This result indicates that students using the DLC more frequently reported that they spend their studying time more effectively and are more committed to reaching their goals.

Fig. 1.
figure 1

Relation between interaction with the DLC measured as the number of reflection modules completed and the postscore on “time and study environment and effort regulation”. The table shows the results of the post-hoc pairwise Wilcoxon tests.

Full size image

5.2 Interaction with DLC vs. Academic Achievement (RQ2)

We measured students’ academic achievement quantitatively by their final exam score. When comparing the final exam score for the four interaction groups, we noted an increasing trend. A Kruskal-Wallis rank sum test indicated that there were significant differences between the four groups (\(\chi ^2 = 48.84, p < .001, df = 3\)). Further analysis using pairwise Wilcoxon tests showed that there was a significant difference between each pair of groups (see Fig. 2). This result indicates that students using the DLC more frequently obtained a higher final exam score.

Fig. 2.
figure 2

Relation between interaction with the DLC measured as the number of reflection modules completed and the final exam score. The table shows the results of the post-hoc pairwise Wilcoxon tests.

Full size image

5.3 Interaction with Module on Angular Momentum vs. Performance and Conceptual Understanding for Corresponding Exam Problem (RQ3)

We further investigated the relationship between students’ interaction with the DLC and academic achievement by studying the role of the topic-specificity of the reflection modules in this relationship. To this end, we investigated students’ performance on one particular problem of the exam, the problem on angular momentum, and their participation to the corresponding reflection module. The score on the exam problem on angular momentum was compared for the three angular momentum interaction groups. A Kruskal-Wallis rank sum test confirmed that there were significant differences between the three groups (\(\chi ^2 = 15.37, p < .001, df = 2\)). Post-hoc pairwise Wilcoxon tests showed that there was a significant difference between the group of students that completed 4–9 modules including the module about angular momentum and both groups of students that did not complete the module about angular momentum (Fig. 3). Note that there was no significant difference between the two groups of students that did not complete the module about angular momentum, but that completed a different number of reflection modules in total. These results suggest that the particular topic of the reflection modules plays a role in the relationship between interaction with the DLC and academic achievement.

Fig. 3.
figure 3

Interaction with DLC and module on angular momentum vs. score exam problem on angular momentum. The table shows the results of the post-hoc pairwise Wilcoxon tests.

Full size image

To measure students’ academic achievement on a more qualitative level, we considered students’ conceptual understanding of one of the exam problems, again the problem on angular momentum. Students’ conceptual understanding was assessed by the concept score as defined in Sect. 4.4. A Kruskal-Wallis rank sum test indicated that there were significant differences between the three angular momentum interaction groups (\(\chi ^2 = 25.39, p < .001, df = 2\)). Post-hoc pairwise Wilcoxon tests showed that there was a significant difference between the group of students that completed 4–9 modules including the module about angular momentum and both groups of students that did not complete the module about angular momentum (Fig. 4). Note that again there was no significant difference between the two groups of students that did not complete the module about angular momentum, but that completed a different number of reflection modules in total. This means that the topic of the reflection modules is important in the development of students’ conceptual understanding, as would be expected.

Fig. 4.
figure 4

Interaction with DLC and module on angular momentum vs. concept score exam problem on angular momentum. The table shows the results of the post-hoc pairwise Wilcoxon tests.

Full size image

6 Discussion and Future Work

We studied the interplay between students’ interaction with the DLC and their metacognitive abilities, academic achievement and conceptual understanding of one of the exam problems. We would like to emphasize that the design of the study did not allow us to draw any causal conclusions. We did not work with a control and experimental group, since this was not feasible in the context of this study due to practical and ethical reasons.

Concerning RQ1, the results obtained show almost no relationships between students’ interaction with the DLC and their metacognitive abilities as measured by the self-reported questionnaire. This could indicate that the reflection modules are not effective for improving students’ metacognitive abilities. However, research [22] already suggested that self-reported questionnaires, such as the MSLQ, are not very accurate measurements of metacognitive abilities. It seems that students are often not able to accurately assess their own use of metacognitive strategies. Moreover, the items in the MSLQ address very general learning strategies, such as how to organize, summarize, and memorize learning materials, how to study for a test, or how to concentrate while studying. By contrast, the DLC focuses specifically on metacognitive strategies for problem solving. Hence, the link between the abilities elicited by the DLC and the items in the questionnaire might be missing or unclear for the students. Therefore, we believe it is still possible that the DLC fosters students’ metacognitive abilities for problem solving. Even stronger, if future qualitative research would indicate that the DLC triggers metacognitive activities, then learning traces from the DLC could be valuable microanalytic measures of metacognitive activity. To this end, systematic observations, think-aloud protocols [22], or existing microanalytic measures [2] could be used, which is very challenging in the context of a large-scale administration.

In answer to RQ2, the results reveal a positive relationship between students’ interaction with the tool and their academic achievement (Sect. 5.2). This corresponds with findings in the literature that metacognitive training results in improved academic achievement [3, 14] and that metacognitively oriented ICT-based interventions and mobile-learning enhance learning outcomes [15, 24]. However, when interpreting this result, we must take into account that students using the DLC more frequently might also be more motivated and spend more time on studying in general than other students. Therefore, we can not conclude that there is a causal relationship between interaction with the DLC and academic achievement. Moreover, we can wonder whether the improvement in academic achievement can be explained by the fact that students have been reflecting on their solving process or by the additional topic-specific feedback that was offered in the reflection modules.

Concerning RQ3, the results indicate that students’ performance on a particular exam problem is related to their participation to the corresponding reflection module. As would be expected, also students’ conceptual understanding of a problem involving a particular concept depends on the topic of the reflection modules that they completed (Sect. 5.3). From these results, we could conclude that the reflection modules only help students to improve their conceptual understanding of physics concepts and therefore also their academic achievement, but not necessarily their metacognitive abilities. We could, however, also argue that these results indicate that reflection on problem solving strategies should be linked to a certain context, as was suggested in the literature [7, 19, 23]. Transfer of learned strategies and metacognitive abilities to new contexts might be difficult for students.

This discussion above shows that there is need for future research, since we do not completely understand yet how the DLC contributes to students’ metacognition and academic achievement. It seems that the improvement in students’ academic achievement by using the DLC can be explained by a combination of improvement in metacognitive abilities and conceptual understanding. As suggested by Verschaffel et al. [24], further research is necessary to disentangle the mediating effect of metacognition on learning outcomes from other possible mediating factors, in this case improvement in conceptual understanding. We need to better understand how students interact with the tool and which reflection questions might trigger metacognitive activities. To this end, we will organize think-aloud interviews in the context of the same course. During individual think-aloud interviews, the students will be instructed to solve a new problem and reflect on the solving process using the corresponding reflection module afterwards, while making their thinking process explicit by talking aloud. We will use an observation protocol for metacognitive activities [12] to analyse which metacognitive activities students use to regulate their solving process, and which metacognitive activities are triggered by the reflection questions.

7 Conclusions

In this work, we presented an online tool, the Disciplinary Learning Companion (DLC), for fostering students’ metacognitive abilities for problem solving through discipline- and topic-specific reflection on the solving process. We studied the relationship between students’ interaction with the DLC and their metacognitive abilities, academic achievement and conceptual understanding. We found no significant relationship between students’ interaction with the DLC and their metacognitive abilities as measured by a self-reported questionnaire. Hereby we contribute to the research evidence questioning the validity of such self-reported questionnaires for measuring metacognitive abilities. The results do show that students that used the tool more frequently obtain a higher final exam score and have a better conceptual understanding of the exam problem considered. Moreover, the results suggest that the topic-specificity of the reflection questions plays a role in the improvement in academic achievement. Future research will use qualitative observations to better understand the interplay between metacognitive activities, conceptual understanding, and problem solving strategies and how this is mediated by the reflection activities of the DLC.