Keywords

1 Introduction

Collaborative problem solving (CPS) is a problem-solving strategy where a number of participants (2 or more) have shared experiences related to a given problem state and goal, and involves cognitive and social skills to achieve a solution [1,2,3]. However, CPS continues to be a complex skill to model and assess. Current work that assesses collaboration often involve simulations, games and team-based activities [4, 5] which engage students as well as provide opportunities for evidence that can be used to measure skills that are hard to address with more familiar item types. In this paper, we present preliminary qualitative findings from ongoing work to design and develop models for three CPS components: cooperation, persistence and problem-solving, using a collaborative game activity as context. Limitations and future work are provided.

2 Methods

We used an evidence-centered design (ECD) approach [6] in defining the constructs that comprised the CPS student model (cooperation, problem-solving, persistence). We designed the tasks, student roles, and instructions to elicit evidence of each construct, and facilitate the measurement of these constructs. We defined indicators using in-game and out-of-game observed behaviors. The study involved three hours of collaborative gameplay from 11 teams of 3 middle students randomly assigned based on availability. Students were instructed to work together to solve problems when playing the educational game Physics Playground (formerly known as Newton’s Playground in [7]). Each team consisted of 7th or 8th grade students, with a mix of male and female students for some teams. The goal in each challenge was to move the green ball to hit the red balloon by drawing simple machine agents (ramp, lever, pendulum, and springboard). Each member of a team was assigned a specific role (e.g., player, questioner, and recorder) in each challenge, rotating roles for each new challenge. Teams were instructed to verbally discuss ways to solve the challenges and what happened as a result of their actions. Gameplay sessions for each team were video recorded using two camera views (front focused on faces, rear capturing gestures) and a computer screen capture. Together with the knowledge of the game features and the CPS constructs themselves, evidence rules were designed to define behavioral indicators for each construct. Using the resulting rubrics for cooperation, problem-solving and persistence, the behaviors for each student were coded [8, 9] by trained raters (inter-rater reliability of above 0.70). A mark was noted for each occurrence of a behavior and different behavioral indicators were linked to each construct. We applied agglomerative clustering based on Ward’s algorithm [10] to extract student clusters (across all 33 students) for cooperation, problem-solving and persistence. The clustering process used the sum indicator occurrence for each student across eight identified games (played by all groups). Below is a list of the final indicators used to code behavior for each construct from the video data, as well as the resulting dendrogram of student clusters for each construct.

3 Results

The resulting behavioral indicators of cooperation, problem-solving and persistence (Table 1) were iteratively engineered and mapped by researchers based on existing frameworks of each construct [3, 11] and behaviors commonly observed during group gameplay by the students. The behavioral indicators across all three constructs can be categorized by students discussing solutions, action steps taken by students to solve the challenge, student behaviors after solving a challenge, and student interactions with one another. Using this rubric, expert raters observed the videos and coded instances when these indicators were observed for each student in each challenge they attempted to solve.

Table 1. ECD-based behavioral indicators of cooperation, problem-solving and persistence
Full size table

After conducting the clustering process, two student clusters emerged and were qualitatively evaluated for cooperation, problem-solving and persistence (looking at the average instances per behavioral indicator and inspection of its values per cluster). From the resulting dendrogram for cooperation (dendrogram figures are not included in this paper due to page limits), one student cluster (13 students) exhibited more instances of communicating by building and improving ideas from others (Indicator 4 in Table 1), talking about solutions (indicator 2) and discussing before attempting a challenge (indicators 7, 8) compare to the other student cluster (20 students). For problem-solving, one student cluster (10 students) emerged to have more discussions in coming up with a solution for the challenges using physics terms (indicators 2 to 8) and providing reasons for a solution or action (indicator 11) than the other student cluster (23 students). Lastly, for persistence, one small student cluster emerged (2 students) that exhibited far higher instances of attempting a challenge (with or without discussion, indicators 16, 17) and far lower instances of engaging in off-topic conversation (indicators 25, 26).

4 Discussion and Future Work

We present in this paper the creation of game-based behavioral indicators for the CPS components cooperation, problem-solving and persistence, in the context of collaborative gameplay. An evidence-centered design approach was used. This study is part of ongoing work to develop valid measurement models for each CPS construct and CPS itself. Findings presented in this paper include mapping of these behavioral indicators to each construct and preliminary analysis of video-coded data using these indicators. This qualitative analysis generated two student clusters for each component and showed how these designed indicators were able to distinguish student groups based on construct definitions. For example in cooperation, indicators related to communication were more evident in one group than the other. In persistence, indicators related to attempts were more evident in one group than the other. And in problem-solving, discussing physics-related concepts in solving the challenge was more evident in one group. Knowing such prevalent indicators per construct may be useful in potentially designing AI-driven pedagogical agents that can evaluate CPS competencies and provide scaffolds in team-based learning activities. Although a limitation in this study included a relatively low number of students, there were numerous identified indicators for each construct. The next phase in this work includes (1) creating Item Response Theory (IRT) models to measure each component and compare observations to self-reported construct measures, and (2) including game-log information in the creation of the CPS models (i.e., multimodal data analysis).