Keywords

1 Introduction

Metacognitive competencies have already, for a long time, been of major interest in general education and educational psychology. In recent years, the issue of metacognitive competencies and their promotion has become even more and more important in teaching, especially in mathematics teaching. In the last decade, the topic metacognition and its role in modelling processes has gained significant importance. Within the international community on mathematical modelling and due to the work of Maaß (2006) and Stillman (2011) in the last decade, the topic metacognition and its role in modelling processes has gained significant importance. Maaß (2006) defines metacognitive competencies as a sub-competence of modelling competencies ; Stillman (2011) focuses on metacognitive barriers in the modelling process and the question of how to overcome them. Nevertheless, until now it could not be clarified how metacognitive competencies can be described theoretically, particularly with regard to the question which domain-specific metacognitive competencies are important for the students’ modelling process and how these metacognitive competencies can be measured. In this chapter, first steps to developing an instrument for measuring students’ metacognitive modelling competencies are presented.

2 Metacognition in Modelling Processes

Working on modelling problems autonomously and successfully is challenging for students all over the world. The difficulties can be explained by the complexity of the problems that requires competencies on different levels. Referring to Blum (2015, pp. 77–78), “modelling competency in a comprehensive sense means the ability to construct and to use or apply mathematical models by carrying out appropriate steps as well as to analyse or to compare given models”. In this sense, modelling competencies not only comprise the sub-competencies referring to single phases of the modelling process, but overall modelling competencies are needed in addition, such as cognitive skills and competencies that allow one to work on a modelling problem successfully and in a goal-oriented way (i.e. competency to structure a problem, to use heuristics and to work together in one group) (Kaiser 2007). According to Maaß (2006) and Blum (2011), metacognitive competencies are an essential facet of modelling competence as well. Thus, in the following a definition of metacognition in modelling processes and empirical results concerning the relevance of metacognition in modelling processes are given.

2.1 Definition of Metacognition

The concept of metacognition was introduced in the 1970s by John Flavell (1979) and Ann Brown (1978). Over the years, metacognition has become a fuzzy concept. Schneider and Artelt (2010, p. 149) define metacognition as

people’s knowledge of their own information-processing skills, as well as knowledge about the nature of cognitive tasks, and of strategies for coping with such tasks. Moreover, it also includes executive skills related to monitoring and self-regulation of one’s own cognitive activities.

In this definition, metacognition is separated into metacognitive knowledge and metacognitive skills (often called metacognitive strategies): The former refers to declarative meta-knowledge that is taken as explicit knowledge or as knowledge to be made explicit. It is subdivided into knowledge of the characteristics of tasks, knowledge of appropriate strategies and knowledge of persons’ own skills and competencies as well as those of other persons involved. Metacognitive skills consist of planning, monitoring and regulating the work as well as evaluating the whole process (e.g. Schneider and Artelt 2010; Veenman et al. 2006). As Veenman (2005) points out, the use of metacognitive knowledge depends on different motivational, cognitive and depositional aspects. Therefore, these aspects need to be taken into account when analysing metacognitive knowledge, although they may not easily be evaluated empirically. Another influencing factor for the usage of metacognition is the difficulty of the task: only tasks estimated on an intermediate difficulty level provoke the usage of metacognitive strategies (Hasselhorn 1992).

Referring to the distinction described above, metacognitive modelling competencies can also be divided into metacognitive knowledge and metacognitive strategies . However, until now it has not been researched how metacognitive competencies can be described theoretically.

2.2 Relevance of Metacognition in Modelling Processes

Looking at empirical findings concerning metacognition and mathematical learning in general and the role of metacognition in problem-solving processes, the role of metacognition is stated ambiguously. In their overview of theoretical and empirical work on metacognition in mathematics education from the previous four decades, Schneider and Artelt (2010) emphasised the importance of metacognition in mathematics education . They not only summarised the results of different studies that gave evidence of the positive correlation between metacognition and mathematical performance; but also they presented findings from intervention studies that succeeded in fostering students’ metacognition and mathematical performance. In contrast, Lesh and Zawojewski (2007), in giving an overview about research on metacognition in problem-solving processes, questioned whether performance improvement was due to metacognition or to the students learning mathematics concepts better or differently. In addition, they gave examples when metacognition (or teachers’ request for using metacognition) can be obstructive rather than helpful.

However, according to Blum (2011, p. 22), “there are many indications that meta-cognitive activities are not only helpful but even necessary for the development of modelling competency”. For example, the relevance of metacognition in modelling processes is emphasised by the respective studies of Stillman et al. (2007) (for an overview about the current state of the art, see Stillman 2011). Especially the complete lack of (or only a very low level of) meta-knowledge about the modelling process can result in considerable problems when dealing with modelling tasks. Problems occur as well in the transitions between the various stages of the modelling process as in dissolving cognitive blockages while performing modelling tasks (Maaß 2006; Stillman 2011). To overcome such difficulties, the modelling cycle can be used as a metacognitive tool (Blum 2011, 2015). In contrast, Schukaljow and Leiss (2011), for example, did not find any significant correlation between cognitive and metacognitive self-reported strategies (in general or task orientated) on the one hand and mathematical modelling competence on the other hand.

Empirical research (Cohors-Fresenborg et al. 2010) further shows that, in particular, procedural aspects of metacognition have a significant influence on learning success; it is therefore proposed to focus on the promotion of procedural metacognition instead of declarative meta-knowledge. Especially planning of the solution process is essential for performing complex tasks successfully as Schoenfeld (1992) and Verschaffel (1999) point out. Mevarech and Kramarski (1997) indicate that reciprocal asking and answering of metacognitive questions by students while working on a complex task can improve mathematical performance as well as metacognitive competencies at the same time. This finding is confirmed by the conclusion of Goos (1998): collaborative interactions deliver metacognitive benefits. Adaptive support by the teacher is indispensable for bringing students onto a meta-level. Hence, strategic interventions are most adequate (Blum 2011; Kaiser and Stender 2013). Not only metacognitive strategies referring to planning, monitoring and regulating the modelling process are of great importance for solving modelling problems; but also Blum (2015) points out that reflecting on one’s own activities is crucial for transferring knowledge and skills from one task to another.

3 Measuring Metacognitive Modelling Competencies

3.1 Methods for Measuring Metacognitive Competencies

In order to measure procedural metacognitive modelling competencies, there exist two possibilities: On the one hand, online methods like thinking aloud, observations, eye movement or log file registration enable process diagnostics concurrent with task performance. Thus, a deeper look into metacognitive behaviour of students without disturbing and influencing the subject too much is possible. But these methods cost a lot of time and money. Therefore, they can only be used for small samples (Veenman 2011). Especially the method of thinking aloud is often used for measuring metacognitive activities . Thinking-aloud protocols are considered to be fairly reliable, because thinking or doing, respectively, and verbalising are happening almost simultaneously. Furthermore, the pure verbalisation of metacognitive activities does not include any interpretations by the students. However, methods like thinking aloud and observation only lead to reliable results if students are able and motivated to verbalise all their thinking: neither activities and behaviours that are automatised and therefore do not occupy space in the working memory nor thoughts during phases of single work can be measured (Schellings et al. 2013). It is in the nature of online methods that data measured with the help of such instruments are bound to a given task.

On the other hand, offline methods like prospective or retrospective interviews or questionnaires can be used for measuring. In these cases, the results rely on the students’ self-reports. This method bears the risk that strategies may be used unconsciously or their use may be forgotten by the students. Furthermore, the item formulation may remind the students of the usefulness of certain strategies. Consequently, they will answer according to their metacognitive knowledge and not on the basis of their behaviour. In contrast to observations and thinking-aloud protocols, processes which were not verbalised for different reasons can be measured with the help of questionnaires or interviews (Schellings et al. 2013; Veenman 2011).

As questionnaires are less labour intensive, they are often used for measuring metacognitive activities . Over the years, the validity of online and offline methods has been compared in several studies, many of these comparing thinking aloud to questionnaires (Schellings et al. 2013). Usually, the correlation is not very high, and therefore self-reports are qualified as less valid; the students’ ability of reporting their applied strategies is doubted. However, Schellings et al. (2013) provide two different explanations concerning the low correlation between thinking aloud and questionnaires: The first assumption is that the compared measuring methods aim at different learning strategies. The second assumption refers to the fact that normally thinking aloud is task bound, whereas questionnaires often measure general learning strategies. Therefore, they developed a three-point-frequency questionnaire based directly on a taxonomy for coding thinking-aloud protocols. Twenty ninth-graders were asked to study a text, thinking aloud simultaneously. After studying the text, they were given the questionnaire. The overall correlation between the questionnaire and the thinking-aloud protocols was higher than in other studies (Schellings et al. 2013).

Thus, the development of questionnaires seems to be promising if you want to develop an instrument for evaluating the effectiveness of a learning environment for promoting students’ metacognitive modelling competencies . In order to measure applied strategies, the students should be asked to fill out the questionnaire just after working on a modelling task.

3.2 Results of Studies Aiming at Development of a Questionnaire for Measuring Metacognitive Modelling Competencies

In order to develop items for measuring metacognitive strategies for modelling, different studies have been conducted. The first studies were aimed at reconstructing metacognitive skills that are important for solving modelling tasks. This was done in two different ways.

Firstly, videotapes of the working processes of several groups of students were analysed by coding metacognitive knowledge and strategies that could be reconstructed by the students’ verbal expressions or their behaviour. In doing so, qualitative content analysis , according to Mayring (2014), was used. Thus the elaborated coding guideline gave an overview on the strategies that could be observed. Concerning metacognitive skills, the following strategies were observed:

  • Competencies for orienting and planning the solution process

    • P1: Subdivide the solution process into several steps.

    • P2: Allocate parts of work to different persons.

    • P3: Structure the solution process according to the time available.

    • P4: Choose useful solution strategies.

  • Competencies for monitoring and, if necessary, regulating the working process

    • M1: Identify different kinds of red-flag situations.

    • M2: Notice incomprehension.

    • M3: Keep track of the time available.

    • M4: Check the work habits.

    • M5: Reconsider solution strategies.

  • Competencies for evaluating the modelling process in order to improve it

    • E1: Evaluate the strategies used.

    • E2: Reflect on the working habit.

    • E3: Validate the solution (cf. Schroeder 2014).

By analysing the videotapes, it became obvious that some students used metacognitive skills but did not express them explicitly; so after some time, they expressed the results of the use of special metacognitive strategies in different ways. Unfortunately, we were not able to figure out when exactly these strategies have been used.

As mentioned above, retrospective observations can ignore the usage of metacognitive strategies. Therefore, based on the coding guideline as well as based on the conceptualisation of existing metacognitive questionnaires for other domains (like Lingel et al. 2014; Rakozky and Klieme 2005), 27 items divided into the subprocesses of planning, monitoring, regulating and evaluating have been developed and tested. According to the fact that metacognitive strategies are only used when they are helpful (i.e. the task is not too easy and the students are motivated to use them; see Sect. 15.2.1), students are asked as well to judge their motivation and the difficulty of the task on a four-point scale. The items were given to 66 students of grade nine from five different classes.

For testing the questionnaire, the students were introduced to a modelling cycle (see Kaiser and Stender 2013) and then they worked in groups on a modelling task. The working process was videotaped. After working on the task, the students were asked to fill in the questionnaire. While filling in the questionnaire, they were allowed to speak to each other and discuss the items. Furthermore, four pairs of students were asked to explain their answers to the items during an interview. Moreover four experts rated the students’ metacognitive behaviour with the help of the questionnaire as well as the videotapes.

Frequency distributions and item difficulties of the students’ self-reports as well as of the experts’ ratings were calculated. The results vary widely (for further information, see Janetzko 2014). Correlations between self-reports and expert ratings were low. With the help of the interviews, some reasons for low correlations were reconstructed:

  • Those students with low metacognitive skills aligned their answers with the students who have higher metacognitive skills. Assumingly, this will not occur if students would not be allowed to discuss their answers with each other.

  • Students claim that they monitor their working process but did not verbalise anything. This is a well-known problem (see Sect. 15.3.1) that can hardly be solved. But it becomes obvious that students’ self-reports are of great importance, because pure observations of metacognitive skills cannot adequately capture those skills.

  • Some formulations were simply not understood by the students. Especially the terminology of the modelling process was not familiar to them. So students have to become acquainted with the terminology beforehand.

  • The difference between some items was not recognised by the students. So these items have to be combined or the difference has to be made more explicit.

  • Sometimes the students did not know how to answer because some strategies they had only used on their own and did not share with the group. Others they had used only because a group member suggested doing so. So the items must clearly differentiate between the use of strategies in the whole group and strategies that were used for the monitoring and regulation of one’s own behaviour.

Consequently, the questionnaire has been reworked in the outlined way paying special attention to the item formulation. Items with a very high average size as well as those that were similar were reformulated and made more explicit. Especially it has been differentiated between single strategy use and strategies used in the whole group (e.g. see Sect. 15.3.3). Furthermore, the introduction of the modelling cycle as a metacognitive tool to the students has been reviewed.

3.3 Items for Measuring Metacognitive Strategies

For measuring metacognitive strategies, the reviewed questionnaire consists of 39 five-point Likert items, divided into four parts. Contrary to the division of metacognitive strategies into the processes of orientating/planning, monitoring/regulating and evaluating, the items have been aligned in the order of their appearance during an ideal modelling process . Thereby, students were guided to recapitulate their working process. Beneath the three phases of at the beginning, during and after working on the task, the students are asked to judge their motivation to work on the task and the task difficulty at the end of the questionnaire.

The phase before the working process is measured by six items. All these items are primarily related to the first step of the modelling cycle, which contains developing a real model by understanding and simplifying the problem. Most items relate to metacognitive strategies for orienting and planning. The items refer to reading and understanding the task, capturing needed information as well as possible interim goals and agreeing on a common approach. Depending on the results mentioned above, the items contain single processes (Table 15.1, item 1.1) and group processes (Table 15.1, item 1.2). Strategies that should be used on one’s own as well as shared in the group were mentioned twice, one time as a single and one time as a group process. Having the goal in mind of developing a task-bound questionnaire that is applicable to different modelling tasks, some items were generalised.

Table 15.1 Selected items of the questionnaire
Full size table

The second part of the questionnaire refers to the phase of working on the problem (from developing a real model to validating the real results) and therefore merely relates to metacognitive strategies for monitoring and regulating the working process. As regulation can only occur when monitoring has been applied, these processes were combined in one section. This section can be divided into items that aim at processes which happen without the occurrence of a problem (13 items) and those metacognitive strategies that are helpful (or restraining) if a problem occurs (12 items). The first 13 items not only refer to strategies for monitoring and regulating (Table 15.1, item 2.1) but also to those for planning the working process (Table 15.1, item 2.2). Altogether students have to give information about their own behaviour as well as about the cooperation in the group during the time of working on the modelling problem. Filter questions were subsequently used for items concerning the occurrence of problems. After these filter questions, possible questions on regulating strategies are posed (Table 15.1, item 2.3 and item 2.4). In this part of the questionnaire as well as in the first part, formulations were tested several times, and formulations were divided into single and group processes.

The third part of the questionnaire consisting of seven items refers to the phases after working on the modelling problem. Primarily the items of this group relate to metacognitive strategies of evaluating the whole process. Using these strategies is – similar to validating results of modelling problems – often forgotten or there is not enough time to do it. As pointed out in Sect. 15.2.2, it is very important to learn through reflection and to overcome some kinds of behaviour that are restricting the quality of a learning process. These items aim at assessing if students reflected on the solution (Table 15.1, item 3.1) and if they had drawn any conclusions for the next working process (Table 15.1, item 3.2). Students evaluate only self-acting, if they came to a solution and had enough time left. In order to measure only the evaluation done during working time, another filter question is posed in the questionnaire.

4 Conclusions

Although different studies have already pointed out the importance of metacognitive modelling competencies for solving modelling problems successfully, research about metacognitive modelling competencies is still at its beginning. With regard to the evaluation of learning environments to promote metacognitive modelling competencies, it is especially necessary to develop instruments for measuring those competencies. Concerning metacognitive skills, there are existing different methods of measuring that have different advantages and disadvantages.

In order to develop a task-bound questionnaire for measuring metacognitive strategies that is applicable to different modelling tasks, two studies have been carried out. The results presented above clearly indicate that a questionnaire seems to be a possible instrument for measuring metacognitive modelling competencies. However, other aspects have to be taken into account. This includes not only the item formulation but also the particular circumstances under which the students are asked to fill in the questionnaires. In order to reconsider the reliability and validity of the revised questionnaire presented in extracts above, the questionnaire has to be tested once more.