Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Introduction

This chapter demonstrates how the collaborative problem solving (CPS) framework, outlined in Hesse et al. (2015; Chap. 2), is applied to a selection of tasks and, in turn, how each of the tasks highlights the skills outlined in the framework. There are two distinct types of tasks presented here: content-free and content-dependent. Content-free tasks do not demand any prerequisite knowledge such as might be taught in traditional school-based subjects but rely on the application of reasoning. Content-dependent tasks draw on skills and knowledge derived from curriculum-based work. As discussed in Hesse et al. (2015), under the proposed CPS framework there are three strands of indicators that summarise social skills and reflect the collaborative aspect of problem solving: participation, perspective taking, and social regulation. Participation is the foundation for engaging with the task and other collaborators, and is reflected in the way people act or interact to complete tasks. Perspective taking skills emphasise the quality of interaction between students, reflecting the level of student’s awareness of their collaborators’ knowledge and resources as well as their responding skills. Social regulation refers to the strategies used by students when collaborating, such as negotiating, taking initiative, self-evaluating and taking responsibility. Cognitive skills are of equal importance within this framework and are similar to those employed in independent problem solving tasks. Indicators of such skills can be summarised under two headings: task regulation and knowledge building. Task regulation refers to the ability of students to set goals, manage resources, analyse and organise the problem space, explore a problem systematically, aggregate information and tolerate ambiguity. Knowledge building is concerned with a student’s ability to understand the problem and to test hypotheses. Knowledge building is underpinned by skills such as planning and executing, and reflecting and monitoring.

In teaching students how to become better problem solvers, a common constraint in traditional test design has been that the attainment of the solution is the sole criterion from which inferences can be made. This has occurred despite the fact that procedural aspects of problem solving have been considered important for some time (Polya 1945, 1957; Garofalo and Lester 1985; Schoenfeld 1985). Within the ATC21S projectFootnote 1 there is an increased focus on drawing inferences about how (and how well) students solve problems, as opposed to simply asking whether they are solving them. Problem solving has sequential phases or steps, such as understanding, planning, solving and checking, that are universally applicable across tasks and contexts. This information, together with information on student collaborative effort, might better support the decisions an educator must make when determining the instructional needs of individual students (Zoanetti 2010). Although goal-attainment is obviously important, it should not be the only criterion of interest. Educators stand to benefit from inferences about procedural quality when determining how best to improve student problem solving.

Problem and Task Characteristics

The differences between real-world problems and problems as they are often analysed in psychological research raise the question of whether the assessment of collaborative problem solving through well-defined problems is useful. A “well-defined” problem is one in which the guiding question and consequently the goal is known, where the elements or “artefacts” that are salient to the solution are known and present, and where the required processes to reach solution are understood. Such problems are amenable to measurement since they involve specific known steps, and have final correct solutions. Use of these types of problems also lend themselves to teaching since a sequence of steps is often clear. Well-defined tasks are typically found within the science and mathematics curriculum. On the other hand, “ill-defined” problems are characterised by ambiguity. They may relate to everyday problems and are not domain-specific; they may draw on many different types of knowledge. They will have many of the characteristics that are associated with what is known as “wicked” problems. These are problems in the real sense of the word – situations for which a solution is unknown, of which the elements or components are not identified, and concerning which useful processes have not been verified. Consequently, for ill-defined tasks there may be several solutions that are appropriate to different degrees, several solution paths or strategies, and it may be the case that not all information is presented or available. There may be no clear direction in which to proceed and no clear identification of how the correctness of a solution can be determined.

The difference between well-defined and ill-defined problems calls into question how valid might be the inferences about individuals’ problem solving capacities if drawn only from well-defined problems. The long term objective of teaching problem solving skills would be to equip students with the capacity to draw from a range of strategies when confronted with ill-defined problems – which latter actually constitute the real-world imperative.

Hesse et al. (2015) describe the nature of problems that might require collaborative activity. The salient feature is that resources will not be equally accessible to all the problem solvers, so there is a need for multiple solvers. Accessibility refers both to direct retrieval as well as to human capacity to understand and manipulate the required artefacts – whether these be objects, knowledge, or processes.

Together, the concerns about whether only well-defined problems can usefully indicate students’ problem solving capabilities, and the nature of problems that require collaborative activity, combined within the ATC21S approach to the deliberate design of tasks along a well-defined to ill-defined spectrum. The assessment tasks were constructed to reflect the characteristics of problems which require collaboration. These characteristics are ambiguity, asymmetry, and unique access to resources with consequent dependence between learners. With such tasks it is possible to test the construct definition model, the developmental learning progressions, the indicators of increasing competence, and the task development and delivery. At the most simple level, problem solving tasks were designed to make collaboration both desirable and essential. In the classroom, this can be achieved by the teacher giving different sets of information to different students in a group, rather than giving them all the same information. In order to solve the problem, the students then need to collaborate in order to access the required resource, in this case, information. Such an approach mirrors real life collaborative problem solving situations, where information may be derived from different sources and is not shared a priori. The dependence between learners that emanates from unique access to different resources provides a more authentic prompt for collaborative activity than mere instructions from a teacher for students to “work together”. Working together may be valued for its social aspect, yet might not be essential, and can be regarded by students as counter to their best interests – particularly when they are functioning in competitive classroom environments.

The tasks in the ATC21S project have many similar characteristics. Each task was constructed so that students would be able to click, drag and drop objects using the mouse cursor, with no requirement to use the keyboard. The tasks were designed for two students to work on and there is a ‘chat box’ for communication between collaborators, designed to facilitate student communication online throughout task completion. Each task presents an instruction stem followed by a problem with tasks ranging from 1 to 8 pages in length. The tasks were designed to be recognisable at face value as puzzles and to include graphics to attract and maintain student engagement. A few of the tasks present exactly the same images, perspectives, instructions and resources to the two students – these are referred to as symmetrical tasks. Many of the tasks present asymmetrical perspectives, providing different information and resources to each student, thereby increasing their need for collaboration. There is encouragement in the tasks for students to discuss the problem in order to manage the identification of resources, and sharing of these. The tasks vary in difficulty level; some require less collaboration but are cognitively more difficult, while others are cognitively easier but require efficient collaboration to solve. The difficulty of the tasks was varied taking into consideration arguments of Funke (1991) by adjusting several of the parameters, such as the number of problem states, the constraints on object manipulation built into each task and described in the problem stem, the complexity of reasoning or planning required to guide the search, and finally the configuration of objects and symmetries within the task.

The matter of symmetry poses challenges to assumptions made in education about equal access for learners. Although there may well be major differences in education provision across and within countries, the presumption is that in any classroom all students will have the same access to resources. In this context, resources refer to tools, texts, teachers, and the classroom environment with all of these supporting and enhancing the learning of the student. This provision is extended to equality of access in the assessment situation, with all students again typically being provided with the same resources. This equality of access has been contested in the last decade by virtue of emphasis in some learning environments, on group work. In this scenario, equality of resource is not assured, since different groups will present with different human resources, and the capacity of the individual to act will be determined not only by their access to resources, and their own capacities, but also by the capacities of others. This reality is reflected in the ATC21S assessment environment, where students are not provided with the same access to resources – either those constructed within the assessment environment, or those that ensue from the varying capacities that student partners bring into play. Both differential access to resources and the consequent dependence between students bring about asymmetry in the assessment task activity.

Asymmetry raises interesting challenges in the world of assessment, as well as in how students and their teachers cope with the learning and teaching activity. In this chapter we demonstrate how both symmetry and asymmetry is manifested in the assessment environment. Discussion of the consequences of this for scoring is presented in Adams et al. (2015; Chap. 6).

Content-Free Collaborative Problem Solving Tasks

Two tasks outlined in this section focus on students’ hypothetico-deductive reasoning skills in an online collaborative problem solving context. The translation of these steps into a process that can be generalised and called “collaborative problem solving” should enable teachers to assess and develop their students’ capacity for hypothetico-deductive thinking as it manifests itself in collaborative problem solving behaviour. Hypothetico-deductive thinking begins with a causal question. Students then generate hypotheses based on observations and data collection. In a virtual world it is possible to monitor this behaviour through analysis of chat and action events. These events can be seen to follow a pattern suggested by Griffin (2014), who argued that problem solving can be understood as a hierarchical series of steps moving from inductive to deductive thinking. Problem solvers first examine the problem space to identify its elements. Next they recognise patterns and relationships between the elements, and formulate these into rules. The rules are then generalised. When generalisations are tested for alternative outcomes, the problem solver is said to be testing hypotheses. While inductive reasoning focuses on establishing a possible explanation to test in the first place, deductive reasoning involves testing whether the explanation is valid or not. The deductive method attempts to “deduce” facts by eliminating all possible outcomes that do not fit the available information. Collaborative problem solving requires the formation of partnerships in which agreement is reached on the nature of hypotheses to be tested and the manner in which they will be tested.

The two “content-free” tasks described here are compatible with an individual problem solving approach in that each has a finite solution, and all the information required for problem solution is included in the problem space. The transition to identification of these tasks as collaborative problem solving tasks lies in the re-structuring of the problem space such that neither member of a pair of collaborating students has access to all necessary information. The first task, Laughing Clowns, is structured symmetrically – both students have access to all resources; while the second task, Olive Oil, is structured asymmetrically – each student has access to different resources. The term “problem space” here refers to the virtual environment which provides all the stimuli and resources that identify that there is a problem. The stimuli include text instructions and some explanation about the problem, as well as virtual artefacts, both static and dynamic, including the graphic objects on the screens, and the indicators of movement such as mouse cursor.

The tasks are hosted on a virtual platform that allows for real-time work activity by two students operating in a one-to-one computing environment. Students may work on the tasks on any computers that have internet access and up to date browsers. Technical requirements are outlined by Awwal et al. (2015; Chap. 5). Each task is described here in terms of the problem solving goals, and the activities or processes and artefacts available to the students. The description is followed by an analysis of the subskills from the conceptual framework that are drawn upon, and assessed through the task.

Laughing Clowns Task

This task requires students to find patterns, share resources, form rules and reach conclusions. The two students are presented with a clown machine and 12 balls to be shared between them. The goal for the students is to determine whether their clown machines work in the same way. In order to do this, the two students need to share information and discuss the rules as well as negotiate how many balls they should each use. The students must place the balls into the clown’s mouth while it is moving in order to determine the rule governing the direction the balls will go (Entry = Left, Middle, Right, and Exit = position 1, 2, 3). Each student must then indicate whether or not they believe the two machines work in the same way (see Fig. 4.1). Students do not have access to each other’s screen so are not able to determine the rule governing the other’s clown machine.

Fig. 4.1
figure 1

Laughing Clowns task

Full size image

Social Skill: Interaction

A fundamental requirement for successful completion of this task is interaction between partners. Students need to be aware from the start that their 12 allocated balls are shared and that the most effective way of finding the solution is to allocate six balls to each such that both students have adequate and equal opportunity to trial their machine and reach a conclusion. Students who do not interact may begin using the balls, and even use them all before realising the resources are shared. More proficient students are likely to be aware early in the task of the need to coordinate their and their partner’s activity and will promote interaction with their partner before they begin to use the balls and test their own machine.

Social Skill: Audience Awareness

Students possessing good perspective taking skills would be aware of their partner’s role in this task and the need to understand their partner’s perspective. Students who do not possess strong skills in this area are likely to proceed with the task with little consideration for their partner’s resource requirements or observations. Students who are proficient are likely to interact with their partner in between ball drops and adapt their behaviour to best suit their partner’s needs. An indicator of this skill is the number of moves students make before stopping and waiting for their partner to move or respond, fewer moves being, in this case, the preferred response.

Cognitive Skill: Resource Management

The ability to manage the available resources contributes to a student’s ability to regulate the task well. For example, students who have lower proficiency in this skill may only concern themselves with checking on how their own machine functions, thereby monopolising use of the resources, while more proficient students are likely to recognise the need for shared use of the balls and allocate them equally.

Cognitive Skill: Relationships (Representing and Formulating)

Students must identify the relationship between entry and exit point of balls, and determine if there is a consistency in how the machine functions. They then need to construct a way of representing this information that will communicate to the partner, as well as being able to understand other forms of representation that the partner uses. The student may choose to represent the relationships through listing discrete pieces of information, through narrative, or through formulation of rules. Each of these representations needs to be amenable to communication through the chat box which is part of every task. Proficient students will also challenge patterns and test the assumptions that underpin their observations – for example, consistency of patterns. The final step comprises the students comparing their representations such that a decision concerning similarity of clown machine functioning can be made (Table 4.1).

Table 4.1 Example of skills observed in Laughing Clown task
Full size table

Olive Oil Task

In this task students are presented with different resources. In order to achieve the objective of the task – which is to fill a jar with 4 l of olive oil – the students must work out what resources are available and are needed. Student A has a virtual three litre jar, olive oil dispenser, transfer pipe and bucket. Student B has a virtual 5 l jar, transfer pipe and bucket. Without knowing what is available to the other, the pair need to recognise that Student A must fill their jar at the dispenser and place it under the transfer pipe so that Student B can accept the oil from the pipe. Until this point, Student B cannot complete any meaningful actions and is dependent on the actions and interactions of Student A. Students need to explore and navigate the task space together until they can place 4 l in Student B’s jar. This task follows the reasoning processes required in the Tower of Hanoi problem popularised by mathematician Eduard Lucas in 1883 (Newell and Simon 1972; Petković 2009). The problem requires the solver to work out a sequence of movements to achieve the goal. It bears some resemblance to the forward planning requirements of a chess game – in thinking beyond one step to the next before implementing an action. This cognitive task is made more complex by the division of resources and the lack of information available to each student (see Fig. 4.2).

Fig. 4.2
figure 2

Olive oil task

Full size image

Social Skill: Interaction

While current technologies do not afford us the ability to analyse the actual text of the communication, the quality of interaction can be inferred through the placement of chat. In this task, interaction is assessed by the presence of chat during specific problem solving stages or ‘blocks’ indicating the level of interaction between students and the perceived importance of collaboration during specific processes. Students who have strong communication skills may initiate or prompt the interaction immediately after the task begins.

Cognitive Skill: Cause and Effect

The ability to use their understanding of cause and effect to develop a plan will enhance a student’s success in this task. A way to measure planning and executing skills is to assess the amount of time taken between actions. For example, after realising that their jar is empty at the start of the task, more able students operating as Student A will take a shorter amount of time than less able students to fill the jar at the dispenser. Another indicator of successful planning and executing for Student A is the time taken between their jar containing 1 l of oil and the transfer of that litre to Student B. This requires students to think of steps ahead of their current state and work out sub-tasks before acting. Some students may propose several rules of cause and effect before gaining success. An example is the presence of the bucket for Student A. This object is redundant but Student A may use the bucket to empty the 3 l jar before realising that this action does not provide a pathway to problem solution.

Cognitive Skill: Problem Analysis

Proficient students are able to analyse the problem before organising the necessary steps to solve it. One example of a student analysing the problem is the identification of their need for information and resources from their partner – which requires elements of task regulation – followed by their description of this problem in a mode of communication familiar to their partner. An indicator of this is the exchange of information, assessed by the presence of the key numbers (1, 3) within the chat, during the time that Student B’s jar contains 1 l of oil, followed by the acceptance by Student B of the 3 l of oil.

Cognitive Skill: Solution

Although students’ proficiencies are not being measured predominantly on their success or failure in completing the task, this factor is still measured. In this task, we assess whether students found the correct solution to the problem by checking whether their final action results in Student B’s jar containing 4 l. The steps taken to solve the problem can then be assessed to determine the processes used and the students’ efficiency in achieving the solution (Table 4.2).

Table 4.2 Example of skills observed in the olive oil task
Full size table

Content-Dependent Collaborative Problem Solving Tasks

The content-dependent tasks draw on particular skills and knowledge derived from school or curriculum based work. These tasks stimulate the development of assessable curriculum-linked problems that can be solved collaboratively and that connect with everyday teaching and learning in the mathematics and science curricula around the world. In the examples presented here only basic subject based knowledge is required.

The two content-dependent tasks outlined here were originally designed by World Class Arena Limited (WCAL) for use as online single student problem solving tasks. Under contract with WCAL the tasks were redesigned for use as collaborative tasks. This involved redesigning tasks so that they required iterations between collaborators, not merely a division of labour. The tasks were designed to be complex, unscaffolded and ill defined. The lack of scaffolding lies in omitting guidelines for the students that would help them to understand both how to proceed, and the fact that there might be multiple paths that could be followed. In the initial design of the single student tasks the problem solution was much simpler to achieve. Students could easily follow the path to solution by understanding the problem, selecting a strategy, and applying the strategy. The path to solution was a simple one and collaboration within this context would not provide much in the way of additional support, information, ideas or resources. To redesign the tasks to be more complex, the stages to problem solution needed to be less clear. This stimulated more sophisticated strategies that require both collaborators to be active participants. Together students are required to try several different strategies to solve the problem, sharing information with one another and reflecting before trying an alternative solution path. The collaboration between problem solvers is a parallel rather than a serial process. It is anticipated that students will be able to better understand the problem using this rigorous method of investigation and develop the ability to transfer this knowledge successfully to different contextual scenarios. Optimally, each collaborator is fully involved in each stage of the process, such that both will reach and agree on the problem solution, and gain an understanding of the process.

Within each of these tasks the complexity increases through subsequent pages with varying approaches to the problem, and allowing knowledge to build. Each page requires both students to participate in the task, and only together can either proceed to the subsequent page of the task. In this respect one student’s progress is linked to the other. It was the intention that this level of scaffolding would prompt students to communicate. Generally, with the content-dependent tasks the final page is designed for independent working and therefore both students must have gained enough knowledge collaboratively in order to apply their knowledge to the final answer independently. Communication on the final pages of the tasks is encouraged in order to optimize the chance that each partner has fully understood the problem, since their task completion jointly depends on it. Although complexity increases throughout the pages, if sufficient knowledge building and task regulation have taken place on previous pages, the final subtask should not present greater difficulty than experienced earlier in the task.

Balance Beam Task

The Balance Beam task is an example of a content-dependent task with elements somewhat reliant on an understanding of the science – in this case, the physics – behind the task. Students need to apply and test rules in order to balance the beam, leading to multiple correct solutions. Collaborating students share a balance beam but each can interact with only one side of the beam (see Fig. 4.3). Student A begins with four different masses, Student B has no masses, and is not directly aware of this resource. Initially masses must be passed from Student A to Student B, and each must place their masses in the correct notch on the beam in order to achieve balance. The students are able to pass the masses back and forth to each other. There are four notches on each side of the balance beam.

Fig. 4.3
figure 3

Pages 1 and 2 of the balance task

Full size image

On the second page of the task students are asked to provide the formula which best describes how they balanced the beam. In subsequent pages students are required to balance the beam in several different ways. The additional pages were constructed to ensure that the students understand the physics of the problem and reduce the probability of successful guessing. Examples are provided below of the elements within the theoretical framework that underpin the social and cognitive processes required to complete the task.

Social Skill: Responsiveness

The student needs to adapt and incorporate contributions from the other. One way in which this can be evaluated is to assess which masses have been transferred by Student A to Student B. If the correct masses are sent, we can infer that the student has grasped the task concept. A specific indicator measures whether Student A sends particular masses to Student B and whether Student B returns them immediately. The latter identifies that Student B successfully responds to their partner by acknowledging which resources are the most useful, and permits the inference that the student understands the task concept.

Cognitive Skill: Systematicity

Within this task it is possible to measure how systematically a student approaches the task. Where a specific sequence of actions is identified, it can be assessed to determine how students are implementing possible solutions and monitoring their progress. For example, functional systematicity skills can be assessed by measuring the number of trials of balance attempted by the collaborators. This can be done by counting the positions tried for each and all masses. Too few or too many would suggest a lack of systematicity. If a student tests every position once and exhausts all possible combinations, they are exploring the space and the resources thoroughly. An example of poor systematicity within this task is Student A continually attempting to pass a further mass to Student B when B already has the maximum number of masses permitted. Contingent on how the task is designed, this may mean that the student is not approaching the task systematically or monitoring their own actions efficiently. They may not have understood the task instructions or not identified the structure of the task (Vollmeyer et al. 2006), or they may not be learning from their mistakes.

Cognitive Skill: Sets Goals

Goal setting is a key skill in problem solving and can be measured in various ways across the assessment tasks. One example within the Balance Beam task is the presence of a numerical value within the chat which represents one of the mass amounts (100 g, 200 g, etc.). If the chat is from the student who does not have those corresponding mass amounts it can be inferred that the student is requesting those mass amounts from their partner and that their goal is to use them to balance the beam. In addition, if the mass amounts are the correct ones (that is, they would balance the beam) it can be inferred that the student has understood the physics underlying the task and intends to use the masses to attempt problem solution (Table 4.3).

Table 4.3 Example of skills observed in the balance beam task
Full size table

Game of 20 Task

The Game of 20 task involves students working together against the computer to reach a value of 20 by placing counters sequentially on a grid. The students need to identify crucial scores and limits in order to win the game. This task involves algebra equations relevant to the mathematics curriculum. There are six pages within the task; five of them present a sub-task where students begin the game at various stages, thereby allowing for varying degrees of difficulty. The first page allows students to play the game in full to help their understanding of the task concept (see Fig. 4.4). On the second page students begin with a game total of 18; on the third page they begin with a game total of 13, and on the fourth page with a game total of six. Page 5 presents students with a number line from which together they select the numbers they believe to be crucial to success in the game. Students are assessed on whether they implement the numbers selected within game play on the subsequent task page (page 6) when they replay the game in full.

Fig. 4.4
figure 4

Pages 1 and 5 of game of 20 task

Full size image

In order to play the game, students independently choose a number between zero and four which contributes to the combined team number from one to five (that is, the two student values are combined to create their team number). Each student needs to consider the input of their partner when selecting a number, as well as considering which number the computer will select. There are several rounds of number selection until the game total reaches 20. If either team (students or computer) enters an amount that exceeds the game total of 20, then that team loses. The aim of the game is for the student team to reach the exact game total of 20 before the computer does.

Social Skill: Responsiveness

Within the Game of 20 task, students’ ability to ignore, accept or adapt contributions from their partner can be assessed. Students who are strong in this skill may be observed selecting a specific number after their partner has sent them a chat message containing that number. It can be inferred from this indicator that their partner has contributed to their activity and that the student has accepted and incorporated this contribution into their game play. Another example of this skill may be observed as students work through page 5, where they need to agree on which numbers are crucial for game success by selecting them on the number line. Students who are less adept in this skill may not accept or consider contributions from their partner even if they are correct. This can be observed when a student deselects a number from the line that their partner has previously selected.

Social Skill: Responsibility Initiative

Students who are more collaborative tend to take more responsibility for their team and ensure that the activities required for task success are completed by themselves and their partner. One example within the Game of 20 task is a student attempting an activity and then reporting their actions to their partner. This may be observed if a student resets the team number, chats with their partner and then changes the numbers selected before progressing with the game. It can be inferred from this activity that the student was not satisfied with the initial number selection, opted to reset it, and then reported this to their partner, resulting in alternative number options.

Cognitive Skill: Cause and Effect

The extent to which students use their understanding of cause and effect to plan and execute can be assessed in this task. Students who are less proficient in this skill may undertake the activity with no clear regard for the consequences of their actions, but more proficient students will use their understanding of cause and effect to plan and execute a strategy or activity. An example of a proficient student in this task is one who selects specific numbers on the number line and then proceeds to use those numbers in game play on subsequent task pages. It can be inferred from this action that the student has determined from previous game play that these numbers are crucial to success and intends to use them in order to succeed in the game.

Cognitive Skill: Reflects and Monitors (Testing Hypothesis)

Students can be assessed on their ability to hypothesise effectively. Students who are not effective in formulating hypotheses tend to maintain one single approach throughout a task, are not flexible and therefore fail to monitor their progress efficiently. Students who have strong skills in developing hypotheses tend to reflect more on their previous actions, monitor their progress, reorganize a problem and try multiple approaches as they gain further information. An example within this task is a student who opts to retry the game after they have already attempted it. On each task page the student can opt to retry each sub task page. By doing so the student is attempting to reflect on the course of action that caused them to fail previously and is trying another approach in order to gain a different outcome and a successful solution (Table 4.4).

Table 4.4 Example of skills observed in the game of 20 task
Full size table

Conclusion

These task descriptions link problem solving and collaborative activities required of students as they engage with tasks. The tasks are engineered to provide opportunities for demonstrations of skills hypothesised to contribute to collaborative problem solving capacities. This approach to task construction reflects clearly the use of inference to attribute meaning to student test responses. Where we are interested in skills development and progression, as opposed to degree of finite content, skills, or knowledge held by individuals, such inferential approaches are essential.