Introduction

Collaborative problem-solving (CPS), as one of the most prominent competencies in the twenty-first century, has received widespread attention in recent years (Dindar et al., 2020; Gao et al., 2022; Herborn et al., 2020). Many recent calls for education reform have called for fostering CPS skills among students and transforming schools into CPS support organizations or communities. For instance, the US Center for Research on Evaluation and Student Testing (CRESST) identified interpersonal and teamwork skills and problem-solving as crucial workforce skills (Care et al., 2016). The National Assessment Center for Education Quality (NAEQ) in China proposed a framework for students’ twenty-first century core competencies specifying that collaboration is one of the important skills and advocating that schools should devote themselves to creating a supportive learning environment (Ma & Corter, 2019). In addition, there is also an increasing amount of educational and psychological research dealing with a variety of CPS-related topics, such as the construction of CPS framework (e.g., Cukurova et al., 2018), the examination of the effectiveness of CPS as a pedagogical approach (e.g., Rosen et al., 2020), the formative assessment of CPS skills (e.g., Stoeffler et al., 2020), and the exploration of potential factors that may facilitate or impede CPS outcomes (e.g., Tang et al., 2021).

Despite the demonstrated abundance of research on CPS in the literature, several issues or challenges exist in evaluating and interpreting these prior results. First, typical methods used to explore CPS include questionnaires, self-ratings or peer ratings, situational judgement tests, and coding and counting approaches (e.g., Gu & Cai, 2019; Sun et al., 2020; Ferguson-Patrick, 2020). These methods can yield limited information, as they lack consideration for the actual CPS processes (von Davier, 2017). The rapid development of computer technologies in recent years creates new possibilities for exploring CPS processes; for instance, the Program for International Student Assessment (PISA) launched a computer-based CPS assessment in 2015, in which students’ interactions with teammates (computer agents) are recorded with time stamps. The process data generated in the CPS assessment contains rich information of individual CPS behaviors. Key features extracted from individuals’ collaborative processes can be used to predict individuals’ potential competency and analyze the reasons behind their performance differences (Andrews-Todd & Forsyth, 2020; von Davier, 2017). In this regard, more empirical studies concerning CPS processes are needed (Du et al., 2022).

Second, most prior empirical studies on CPS have been conducted using variable-centered approaches (e.g., linear regression, structural equation models), exploring the potential factors of CPS for the overall sample (Ma, 2021). These variable-centered studies might ignore important differences among subgroups of the overall sample. On one hand, this may cause inconsistent results across prior studies (Ma, 2022). On the other hand, practical implications or interventions aimed at improving individual CPS skills may be limited, as they tend to treat individuals in the same way. It would be desirable to consider person-centered approaches (e.g., cluster analysis, latent profile analysis) to identify subgroup differences in individual CPS behaviors and accordingly develop individualized and adaptive strategies.

Finally, many prior studies related to CPS have been conducted in Western cultures, such as in the USA (e.g., Avry et al., 2020), Sweden (e.g., Schindler & Bakker, 2020) and Germany (e.g., Stadler et al., 2019). Increasing global interest in promoting CPS skills makes it important to investigate CPS for non-Western students. It also seems important to explore whether the results from prior CPS studies conducted in Western countries generalize beyond Western cultures. China would seem to be a promising testing venue, as a typical Eastern country in which a collective culture is predominant, and where the historical emphasis on rote knowledge acquisition in school education makes it urgent to provide evidence-based advice on supporting and fostering student CPS skills (Ma & Corter, 2019).

To address the above questions, the present study was designed to utilize a person-centered approach (i.e., latent profile analysis) to investigate unique collaborative problem-solver profiles of 15-year-old Chinese students based on fine-grained process indicators (i.e., time-on-task, actions-on-task, three specific CPS skills) from the PISA 2015 CPS assessment, and further to examine how these profiles may relate to student demographic and motivational characteristics, as well as CPS performance. This study may contribute to deeper understanding of individual CPS behaviors and shed light on practical interventions to improve student CPS skills.

Prior research on collaborative problem-solving

Definition of collaborative problem-solving

In PISA 2015, collaborative problem-solving (CPS) is defined as “the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution” (OECD, 2017, p. 49).

In this definition, CPS competency is considered as a combination of three core skills (Stadler et al., 2019): (1) establishing and maintaining shared understanding (EMSU), which refers to the competency to “identify the knowledge and perspectives that other group members hold and establish a shared vision of the problem states and activities” (OECD, 2017, p. 50); (2) taking appropriate action to solve the problem (TASP), which refers to the competency to “identify the type of collaborative problem solving-related activities that are needed to solve the problem and carry out these activities to achieve the solution” (OECD, 2017, p. 50); and (3) establishing and maintaining team organization (EMTO), which refers to the competency to “understand one’s own role and the roles of other agents, follow the rules of engagement for one’s role, monitor group organization, and facilitate the changes required to optimize performance or to handle a breakdown in communication or other obstacles to solving the problem” (OECD, 2017, p. 50).

To objectively assess the abovementioned CPS skills, PISA develops a range of individual computer-based CPS tasks (e.g., Xandar), which requires individuals to collaborate with one to three virtual computer agents (human–agent assessment approach, H-A) to solve problems in simulated real-life scenarios.

Prior research on CPS process

With the rapid development of computer technologies, researchers have tried to explore CPS processes in addition to its outcomes (e.g., Cukurova et al., 2018; Sun et al., 2022; Haataja et al., 2022). Compared with CPS performance outcomes that may solely provide information about what has been achieved during CPS, CPS process data can provide additional insights into how the responses or outcomes are produced.

Researchers have proposed and utilized certain CPS process indicators to evaluate individual CPS skills (e.g., Cukurova et al., 2018; Stoeffler et al., 2020). For instance, Cukurova et al. (2018) put forward four nonverbal indexes of physical interactivity (i.e., synchrony, individual accountability, equality, and intra-individual variability) to measure student CPS competence in practice-based activities, and empirically validated the indexes using a multimodal learning analytics system. Stoeffler et al. (2020) designed a computer-based assessment to evaluate individuals’ CPS skills in which individuals were required to interact with a virtual agent to solve a series of challenges in a first-person maze environment. Based on telemetry-based (e.g., log file, clickstream) and item response data, such CPS skills as reaching the goal, persistence, problem feature awareness, perspective taking, and strategy were assessed.

Researchers have also shown that process indicators can be used to identify unique CPS interaction patterns (e.g., Dowell et al., 2020; Rosen et al., 2020). For instance, Herborn et al. (2017) identified three types of collaborators during CPS: passive low-performing (non-) collaborators, active high-performing collaborators, and compensating collaborators. The process indicators they utilized include student performance in knowledge acquisition, knowledge application, questioning, requesting, and asserting. Similarly, Dowell et al. (2020) proposed four socio-cognitive roles during CPS interactions, i.e., drivers, influential actors, lurkers, and socially detached. These profiles were shown to have distinct features related to participation, social impact, overall responsibility, newness, internal cohesion, and communication density.

Researchers have also suggested that CPS processes have statistically significant relationship with CPS outcomes (e.g., Haataja et al., 2022; Hao et al., 2016; Reilly & Schneider, 2019; Zheng et al., 2020). For instance, Sun et al. (2022) examined the effects of particular CPS behaviors among triads on CPS outcomes. They found that talking about appropriate ideas contributed to desirable outcomes, while discussing inappropriate ideas might divert the team in a nonproductive direction. Li et al. (2022) investigated the effects of action transitions on CPS outcomes among sixth graders in Finland. The actions in the sequential processes of computer-based CPS tasks included using a mouse to drag objects and typing texts in chat windows. Results indicated that pairs having at least one member with high social and high cognitive CPS skills completed more actions and showed more action transitions, indicating that they attempted more ways to solve the problem and achieved more productive CPS than other pairs.

As noted above, individuals may experience different CPS processes and demonstrate different CPS performance, which may in fact depend on a variety of personal and contextual factors such as task difficulty, individual motivation, and competency. One of the seminal theories that explains why individuals show different behaviors and performance while they work on a specific task is Weiner’s (1972) attribution theory. According to the theory, two major factors that may determine the success or failure of one’s behaviors are ability and effort. Ability is one of the relatively invariant properties of a person, while effort is largely determined by the momentary intentions of the person (Weiner, 1972). In achievement-related contexts, success might be attributed to high ability and/or effort, while failure might be due to low ability and/or lack of effort. In this study, we take Weiner’s (1972) attribution theory as the theoretical framework to help classify and interpret the profiles of collaborative problem-solvers.

Prior research on potential factors associated with CPS

Demographic and motivational characteristics play important roles in CPS processes and outcomes. Specifically, the relationship between gender and CPS competency was found to be inconsistent across prior studies (e.g., Ahonen & Harding, 2018; Li & Liu, 2017; Tang et al., 2021). For instance, Ahonen and Harding (2018) reported no significant difference in cognitive and social CPS skills between female and male students in Finland. However, Li and Liu (2017) found that female students were significantly better than male students in CPS skills in Taiwan. Similarly, Tang et al. (2021) found that female students tended to excel more in CPS skills than male students based on Chinese data from PISA 2015. Socioeconomic status was found to be positively correlated with CPS competency, with consistent results across studies (e.g., Tang et al., 2021; Wang, 2018).

Moreover, researchers (e.g., Tang et al., 2021; Wang, 2018; Xu & Li, 2019) have also investigated how motivational factors may affect CPS processes and outcomes. For instance, Xu and Li (2019) found that, while collaborating with others to solve problems, students who put more value on interpersonal relationships and teamwork tended to perform better in the CPS assessment. Tang et al. (2021) also found that students who valued interpersonal relationships more performed better in CPS assessments, however, they also found that students who valued teamwork more tended to perform worse. They argued that students who thought highly of interpersonal relationships would be highly motivated and would utilize more communication skills to ensure the collaboration would be carried out more smoothly.

Method

Data source

Data used in this study was retrieved from the PISA 2015 website (https://www.oecd.org/pisa/data/2015database/). It is worth noting that student CPS process data was based on one CPS unit called Xandar, which is the only released CPS task in PISA 2015. A total of 9841 students from China [Beijing, Shanghai, Jiangsu, and Guangdong (BSJG)] participated in the CPS assessment, yet only 1677 students (799 female students, 47.6%; 878 male students, 52.4%) were assigned the Xandar task due to test design.

The Xandar task

In the Xandar unit, a three-person team comprising a student and two computer agents named Alice and Zach takes part in a contest where they must collaborate to explore and answer questions related to the geography, people, and economic situation of the fictional country of Xandar (OECD, 2017). The unit consists of four parts, and the student needs to make decisions and coordinate the task by making multiple-choice selections to respond to the team members (OECD, 2017). Specifically, in Part 1, the student is familiarized with the chat interface and the task space so that they know how the contest will proceed. The student first needs to click “Join the chat” in the task space. Then the team member Zach indicates that he wants to go ahead and start answering questions without a strategy, and the student should choose to state their preference for developing a strategy.

In Part 2, the student is informed that each team member will be responsible for questions in one of the three subject areas (i.e., “Geography,” “People,” and “Economy”), and they will apportion the subject areas among themselves. However, both Alice and Zach show preference for taking the subject of “People,” and they give reasons as to why they both want to do so. Hence, the student should advance the problem-solving process and use the information provided by Alice and Zach to assign the subject “People.” Then after Alice and Zach have each claimed a subject area, the student needs to claim the last subject area for themselves.

In Part 3, the student knows that their assigned subject is “Geography,” and they must enter the contest and answer questions about Xandar’s geography. The student first must click on the button “Geography” in the task space to start the questions, and then click on the icons on the map of Xandar to obtain answers. After clicking the “Click Here to Continue” button but before the student has a chance to click on of the icons on the map, a checkmark is placed on the scorecard to indicate that one of the questions on Xandar’s geography has been answered. The previously agreed upon rules of engagement stated that it should be the student answering the geography questions. Alice makes a remark to this effect in the chat interface, and the student should come up with an appropriate response, which in fact tests whether the student has observed that the previously agreed upon rules of engagement have not been followed. After answering the item, the student is interrupted and informed that they have made progress in some, but not in all, subjects and that Alice has sent another message. This is the end of Part 3.

Part 4 picks up from Part 3, and the student is required to evaluate the team progress and fix any problems that have resulted. First, Alice asks the team about its progress, so the student should provide an accurate response to Alice’s question. Then Zach responds that he is having trouble with the questions in his assigned subject area of economy. Then the student should choose to encourage Zach and propose how the student and Alice might help him.

Finally, the student is informed that their team won the contest by answering all the questions correctly. The task ends here. For more details, a complete version of the Xandar task (with the chat interface, task space, and exact items) is presented in the Appendix.

Measures

Gender

Gender was dichotomously recorded, with 1 = female and 2 = male.

Economic, Social, and Cultural Status (ESCS)

The PISA index of ESCS was derived from three variables: “parents’ highest occupational status,” “parents’ highest level of education,” and “home possessions” (OECD, 2017). The average of the index is 0 and the standard deviation is 1 across Organisation for Economic Co-operation and Development (OECD) countries. Higher ESCS scores indicate better socioeconomic status.

Attitudes toward collaboration

Attitudes toward collaboration were assessed from two dimensions: valuing relationships, with four items (i.e., “I am a good listener,” “I enjoy considering different perspectives,” “I take into account what others are interested in,” and “I enjoy seeing my classmates be successful”); and valuing teamwork, with four items (i.e., “I prefer working as part of a team to working alone,” “I find that teamwork raises my own efficiency,” “I find that teams make better decisions than individuals,” and “I enjoy cooperating with peers”). A four-point Likert scale was utilized in the questionnaire, with 1 = “strongly disagree,” 2 = “disagree,” 3 = “agree,” and 4 = “strongly agree.” Higher scores indicate more positive attitudes toward collaboration.

Achievement motivation

Students were asked to answer five questions related to achievement motivation: “I want top grades in most or all of my courses,” “I want to be able to select from among the best opportunities available when I graduate,” “I want to be the best, whatever I do,” “I see myself as an ambitious person,” and “I want to be one of the best students in my class.” A four-point Likert scale was used in the questionnaire, with 1 = “strongly disagree,” 2 = “disagree,” 3 = “agree,” and 4 = “strongly agree.” Higher scores indicate stronger achievement motivation.

Collaborative problem-solving skills

The three types of CPS process skills were assessed based on a total of 12 dichotomously scored items in PISA 2015: (1) establishing and maintaining shared understanding, which in the Xandar unit was assessed using five items, with 0 = no credit and 1 = full credit, and then the total score of the five items was used to represent a student’s proficiency level for this specific skill; (2) taking appropriate action to solve the problem, which in the Xandar unit was assessed using one item, with 0 = no credit and 1 = full credit, and the score was utilized to represent a student’s proficiency level for this specific skill; and (3) establishing and maintaining team organization, which in the Xandar unit was assessed using six items, with 0 = no credit and 1 = full credit, and then the total score of the six items was used to represent a student’s proficiency level for this specific skill. Please see the Appendix for the exact content of the above CPS items.

It is worth noting that all the items are independent of one another. No matter which response a student selects for a particular item, the computer agents respond in a way so that the unit converges. All students are therefore faced with an identical version of the next item.

Total collaboration time (time-on-task)

In the PISA 2015 CPS assessment, the time a student spent on the CPS task was recorded numerically. Consistent with prior studies (e.g., Boeck & Scalise, 2019), we used a log transformation of the total time spent for the following statistical analyses to correct for nonnormality.

Total number of actions (actions-on-task)

In the PISA 2015 CPS assessment, the number of actions a student made while working on the CPS task (e.g., keystrokes, clicks and double-clicks, drag and drop) was recorded numerically. Consistent with prior studies (e.g., Boeck & Scalise, 2019), we used a log transformation of the total number of actions for the following statistical analyses to correct for nonnormality.

Collaborative problem-solving performance

In PISA 2015, student collaborative problem-solving performance was estimated as plausible values (PVs) by drawing on all the questions in the CPS assessment, and a total of 10 PVs was generated. Similar to many prior studies (e.g., von Davier et al., 2009), we used the average of the 10 PVs to represent student overall collaborative problem-solving performance.

Data analysis

Data in this study were analyzed in Mplus 8.3 (Muthen & Muthen, 1998–2019). First, a latent profile analysis (LPA) was utilized to identify the appropriate number of collaborative problem-solver profiles within the BSJG-China sample. LPA is an advanced person-centered method that can be used to obtain different clusters or profiles of observations that have similar levels on measures of interest (Ma, 2022). The person-centered approach (e.g., LPA, cluster analysis) differs from the more traditional variable-centered approach (e.g., linear regression, path model) in several ways. Notably, the variable-centered approach assumes that all individuals from a sample are drawn from a single population and that a single set of averaged parameters are estimated (Meyer & Morin, 2016). In other words, the variable-centered approach allows for the detection of complex relationships among variables, yet it overlooks population diversity. In contrast, the person-centered approach considers the possibility that the sample might in fact reflect multiple subpopulations characterized by different sets of parameters (Meyer & Morin, 2016). The objective of person-centered approach, therefore, is to identify potential subpopulations presenting differentiated profiles with regard to a set of variables. Typical person-centered approaches include cluster analysis and LPA. Compared with cluster analysis, LPA utilizes model-based technique, and thus it can generate more robust and reliable results (Bamaca-Colbert & Gayles, 2010). LPA is now regarded as one of the most flexible person-centered methods, and researchers have utilized LPA to address a wide range of research questions (Cowden et al., 2021).

In this study, five collaborative process indicators (i.e., time-on-task, actions-on-task, and three CPS process skills) were entered into the LPA models. Model fit indices include maximum log-likelihood (LL), Akaike information criterion (AIC), Bayesian information criterion (BIC), sample-adjusted Bayesian information criterion (SABIC) and entropy. LL, AIC, BIC, and SABIC are information criteria that penalize model complexity, and thus models with lower values are regarded to be better. However, these fit indices may not always arrive at the lowest values. In this case, the best model can be determined by using the elbow criterion. Entropy examines the model classification accuracy, and higher values indicate more clear classification of profiles.

After the optimal LPA model was determined, students were categorized into specific collaborative problem-solver profiles. We further validated profile differences by exploring how demographic variables (e.g., gender, ESCS) and motivational characteristics (e.g., achievement motivation, attitudes toward collaboration) may predict profile membership through conducting multinomial logistic regression, and by examining how these profiles may differ in their overall mean CPS performance.

Results

Descriptive statistics

Table 1 presents the descriptive statistics of the variables used in this study.

Table 1 Descriptive statistics for the variables used in this study (N = 1677)
Full size table

Latent profile analysis

Latent profile analyses with two to six profile solutions were performed based on the five CPS process indicators: time-on-task, actions-on-task, EMTO, EMSU, and TASP. The results are presented in Table 2, Fig. 1, and Table 3.

Table 2 Model fit indices for LPA models across profile solutions
Full size table
Fig. 1
figure 1

Information criteria change patterns of different profile solutions

Full size image
Table 3 Average latent class probabilities for the four-profile model
Full size table

The four-profile model was determined to be the best solution because first the AIC, BIC, and SABIC values dropped dramatically at the three-profile and four-profile model (Fig. 1). However, the four-profile solution had a higher entropy value (Table 2), indicating better classification accuracy. Second, the probability that each student belonged to a given class would be correctly categorized for the four-profile solution that ranged from 0.863 to 0.999 (> 0.80, see Table 3), indicating that incorrect classification was unlikely and that the model had good reliability. Third, the sizes of the four profiles were 351 (20.93%), 64 (3.82%), 35 (2.09%), and 1227 (73.17%), respectively. No profile size was lower than the threshold of 1% of the total sample size or 25 cases (Spurk et al., 2020). Finally, in terms of interpretability, the four-profile solution characterized an additional profile that was qualitatively different from the remaining three profiles and can be supported theoretically (further discussed in the Results and Discussion section).

Descriptive statistics for latent profiles

Based on the four-profile model, the Chinese sample was divided into four latent groups. Descriptive statistics for the four latent profiles are shown in Table 4.

Table 4 Descriptive statistics for the latent profiles
Full size table

For a better illustration, we transformed the above raw data into standardized mean values. Specifically, we first generated standardized z-scores for each variable in the full sample, and then calculated and plotted the standardized mean values for each profile in Fig. 2. Values at the 0 mean indicate average performance. Values below 0 indicate lower performance, and values above 0 indicate higher performance than the average.

Fig. 2
figure 2

Standardized mean values for the latent profiles

Full size image

As seen, the four latent profiles had distinct features in collaborative processes. Specifically, (1) Profile 1 (N = 351, 20.93%) could be referred to as a “Disengaged Collaborative Problem-Solver” because students in that profile spent least time on the task, performed relatively fewer number of actions, and demonstrated lower CPS skill levels. Hence, it may be the case that students in that profile were not interested in or even not willing to pay attention to the collaborative task. (2) Profile 2 (N = 64, 3.82%) could be referred to as a “Struggling Collaborative Problem-Solver” because students who fit that profile spent a relatively longer amount of time and performed more actions on the task. However, their demonstrated CPS skill level was not satisfactory. It is likely that students in Profile 2 struggled a lot during the CPS process but failed in the end. (3) Profile 3 (N = 35, 2.09%) could be referred to as an “Adaptive Collaborative Problem-Solver” because students in that profile spent a relatively longer amount of time and performed the greatest number of actions on the CPS task, and they also demonstrated relatively high levels of CPS skills. It seemed that students in that profile were able to complete the task after many trials and explorations. (4) Profile 4 (N = 1227, 73.17%) could be referred to as an “Excellent Collaborative Problem-Solver” because students in that profile demonstrated the best CPS skill performance while spending an average amount of time and performing relatively fewer actions.

It is worth noting that the frequency of students in each of the four profiles was not well-balanced, which might occur due to the fact that the Xandar task is relatively easy, and the path to solve the problem is designed to be relatively more structured so that students do not necessarily need to conduct many actions to attempt different paths (OECD, 2017). Therefore, a majority of students (N = 1227, 73.17%) were able to successfully complete the task and were categorized into the Excellent profile, while the number of students who fit the Struggling (N = 64, 3.82%) and the Adaptive (N = 35, 2.09%) profiles were relatively few. If the tasks are more complex, it is expected that students should be more evenly distributed. For example, in the study by Herborn et al. (2017), it was found that students had been roughly evenly distributed into three different profiles: the low-performing (non)collaborator profile (N = 163, 33.89%), the active high-performing collaborator profile (N = 171, 35.55%), and the compensating collaborator profile (N = 147, 30.56%).

Latent profile analysis with predictors

The four profiles were validated by exploring how student demographic and motivational characteristics may relate to profile membership. Five covariates were added to the extracted four-profile model and regressed on the latent profiles. The results are presented in Table 5.

Table 5 Multinomial logistic regression coefficients and odds ratios across the profiles
Full size table

It can be seen from Table 5 that the profile membership can be differentially affected by demographic and motivational variables. Specifically, girls were more likely to be classified into the Excellent profile than into the Struggling profile (OR = 0.42) or into the Disengaged profile (OR = 0.23). Students with higher ESCS were more likely to be classified into the Excellent profile than into the Struggling (OR = 1.75) profile or into the Disengaged profile (OR = 2.62). Students who put more value on interpersonal relationships were more likely to be classified into the Adaptive profile than into the Disengaged profile (OR = 1.50) or into the Struggling profile (OR = 2.15). Similarly, they were more likely to be classified into the Excellent profile than into the Struggling profile (OR = 1.70). As for students who put more value on teamwork, they were less likely to be classified into the Adaptive profile than into the Struggling profile (OR = 0.37), the Disengaged profile (OR = 0.46) or the Excellent profile (OR = 2.13). No significant results were found for achievement motivation (p > 0.05).

Latent profile analysis with outcomes

We further validated the four profiles by comparing the average CPS performance of students in the four latent profiles (Fig. 3).

Fig. 3
figure 3

Overall CPS performance of students in the four latent profiles. Notes: The OECD average CPS performance was set at 500 score points, and the standard deviation across the OECD countries at 100 score points

Full size image

For a better illustration, we transformed the above raw data into standardized mean values. Specifically, we first generated standardized z-score for CPS performance in the full sample, and then calculated and plotted the standardized mean values for each profile in Fig. 4. Values at the 0 mean indicate average performance. Values below 0 indicate lower performance, and values above 0 indicate higher performance than the average.

Fig. 4
figure 4

Standardized overall CPS performance of students in the four latent profiles

Full size image

As seen from Figs. 3 and 4, on average, students who fit the Excellent profile showed the highest CPS performance, followed by students who fit the Adaptive profile and then the Disengaged profile, and those who fit the Struggling profile scored the lowest. Significant mean differences in CPS performance were found between all of the profiles (p < 0.05) except between the Adaptive profile and the Excellent profile (p > 0.05).

Discussion

The purpose of this study was to identify unique student profiles of collaborative problem-solvers by jointly considering five CPS process indicators, i.e., time-on-task, actions-on-task, and three CPS process skills, from the computer-based Xandar task in PISA 2015. Moreover, this study also validated the four profiles by exploring how profile membership may relate with student demographic and motivational characteristics, as well as the overall CPS performance.

Four latent profiles of collaborative problem-solvers

In this study, we have identified four distinct profiles of collaborative problem-solvers, i.e., Disengaged, Struggling, Adaptive, and Excellent, based on CPS process indicators, which may deepen our understanding of individual behaviors while they work on a computer-based CPS task.

Specifically, students who fit the Disengaged profile are characterized as spending the least time, performing few actions, and demonstrating poor CPS skills. In other words, these students seemed to be unwilling to engage in the CPS task and lack necessary CPS skills to work with others (Eichmann et al., 2020; Greiff et al., 2018; Scherer & Gustafsson, 2015; Teig et al., 2020), which is similar to such undesirable effects for collaborative tasks as social loafing (Petty et al., 1977) and free riding (Delton et al., 2012). For one thing, this might occur because the PISA 2015 CPS assessment is low stakes in nature, so students in this profile might not attach great importance to the task. For another, the human–agent assessment approach in PISA 2015 may make these students feel less pressured and motivated to get involved in the collaborative task, as they do not have to take responsibility for other “human” teammates (Jacobs & Goh, 2007; Johnson & Johnson, 2003).

Students who fit the Struggling profile are characterized as spending substantial time and effort on the collaborative task, but they demonstrate poor CPS skills. In other words, students who fit the Struggling profile may have difficulties in delving into the Xandar task and lack necessary CPS skills to respond better to the teammates (computer agents). In fact, the difficulty might come from the fact that the CPS assessment in PISA 2015 is delivered in a computer-based format (OECD, 2017). These students may not be comfortable with using computers or not familiar with the new human–agent collaborative task. In addition, this may also relate to their limited collaboration practices or experiences in school education.

Students who fit the Adaptive profile are characterized as spending a large amount of time and effort on the CPS task and demonstrating high levels of CPS skills. In other words, students who fit the Adaptive profile managed to solve the CPS task after trials and explorations even though they may not be able to do so at first.

Finally, students who fit the Excellent profile can be considered to be the most efficient collaborative problem-solvers because they demonstrate excellent CPS skills while using an average amount of time and performing relatively fewer actions (Greiff et al., 2018).

In fact, the identification of the four latent profiles fits well with Weiner’s (1972) attribution theory in psychology. As mentioned previously, the attribution theory states that two of the major factors that may determine the success or failure of one’s behaviors are ability and effort. On the basis of ability and effort, students can be categorized into four quadrants as shown in Fig. 5, which in fact corresponds well with the four latent profiles identified in this study.

Fig. 5
figure 5

Four latent profiles of collaborative problem-solvers

Full size image

As seen in Fig. 5, the Disengaged profile is characterized as being low in both ability and effort; the Struggling profile is characterized as being low in ability but high in effort; the Adaptive profile is characterized as being high in both ability and effort; and the Excellent profile is characterized as being high in ability and low in effort. The attribution theory may validate and support the four profiles identified in this study to some extent, and in turn the result may provide empirical evidence for the attribution theory, as well. Moreover, this classification may also help deepen the understanding of why students have different CPS performance.

It is worth noting that the three-profile solution (Fig. 1) could also be arguably viable for the Chinese sample in PISA 2015. Compared with the four-profile solution, the three-profile solution keeps the Disengaged and the Excellent profiles, but it does not discriminate between students who fit the Struggling and the Adaptive profiles. In other words, the three-profile solution combines the students who fit the Struggling and the Adaptive profiles, and this new profile could be characterized as the Middle profile, in which students tend to devote time and effort on the task but show a medium level of CPS skills. We adopted the four-profile solution in this study, as it contains richer and more detailed categorization information than the three-profile solution, and we believe that the difference between students who fit the Struggling profile and the Adaptive profile is meaningful, and it can also be supported by the attribution theory in psychology.

The association between student demographic and motivational characteristics and profile membership

To validate the four profiles, we further explored how student demographic and motivational characteristics may relate with the profile membership. In terms of demographic covariates, it was found that girls were more likely to be in more successful CPS profiles (e.g., the Excellent profile) than boys. In fact, the result is consistent with some prior studies (e.g., Emerson et al., 2015; Li & Liu, 2017; Tang et al., 2021), in which girls were found to have better social skills and collaborative problem-solving performance than boys. In fact, the gender differences may be explained by the gender socialization process described in social learning theory (Bandura, 1977). Specifically, gender socialization is the process of learning the social expectations and attitudes associated with one’s sex. Once children are known to exist within a gender group, reinforcement from parents, teachers, and peers is differentially applied when children’s behaviors conform to gender-based expectations (Hajovsky et al., 2022). In Chinese society, girls are expected to be more cooperative, submissive, kind, gentle, responsive, empathic, and prosocial than boys from an early age (Abdi, 2010). It is more accepted for boys to be outgoing and less cooperative. Therefore, given stereotypic gender roles prescribing more social traits and behaviors for girls than for boys, it is more likely for girls to develop better social skills and display more social behaviors than boys.

Furthermore, limiting cross-gender friend selection may also lead to the abovementioned gender differences (Abdi, 2010). Extensive research has shown that there are differences in gender-normative communication styles between girls and boys. Examinations of cross- and within-gender conversations indicate a significant difference in interactional style (Maltz & Borker, 1982). Girls’ conversation has been found to include fewer interruptions, fewer statements of disagreement, more positive nonverbal and verbal content (e.g., nodding, short statements of agreement), and more question asking than that of boys. Furthermore, girls’ question asking often serves to signal attentiveness to the conversational partner rather than boys’ simple request for information (Hajovsky et al., 2022). These interactional differences are likely to be bolstered given the higher frequency of child-selected, same-gender play groups (Hajovsky et al., 2022).

In this study, we also found that students with better socioeconomic status were more likely to be classified into more successful profiles (e.g., the Excellent profile), which is in line with many prior studies (e.g., Wang, 2018; Tang et al., 2021). For example, Wu et al. (2020) examined how family socioeconomic status (SES) was related to children’s social skill development through family processes among 508 Chinese preschool children. Results indicated a significant indirect effect of family SES on the initial levels of children’s social skills and growth mediated by maternal depressive symptoms, marital relationships, and parenting practices. For one thing, families with better socioeconomic status are more likely to provide rich experiences inside and outside the home (e.g., books, travel) that are important for the development of children’s social skills. For another, higher family income and parents’ educational levels predict better parenting practices that are conducive to children’s social development—that is, more warmth and less negativity (Emmen et al., 2013). In contrast, parents with financial difficulties are less likely to provide warm parenting and sensitive and consistent responses to children’s needs (Conger & Donnellan, 2007).

Regarding motivational characteristics, first, student achievement motivation was found to be not significantly associated with profile membership, which might occur due to the low stakes nature of the CPS assessment in PISA 2015. In addition, some researchers (Ames, 1992; Nichols & Miller, 1994; Summers et al., 2005; Hänze & Berger, 2007) have suggested that a collaborative problem-solving environment may create a more mastery-oriented context, leading students to adapt mastery/learning goal orientations instead of performance/ability goals. The achievement motivation in PISA 2015 is similar to a performance goal that measures student desire to demonstrate competence by outperforming others (Kim et al., 2012). Hence, the achievement motivation may not effectively differentiate students in the CPS assessment.

As for attitudes toward collaboration, it was found that students who valued interpersonal relationships were more likely to be classified into the Adaptive and Excellent profiles than those who did not, which is consistent with previous studies (e.g., Andrews-Todd & Forsyth, 2020; Howard et al., 2017). This result makes sense because students who valued interpersonal relationships tended to possess good collaboration qualities such as considering different perspectives, listening to others’ opinions, and taking into account what others are interested in. These qualities are indeed conducive for effective CPS processes.

Moreover, we found that students who considered teamwork as a way to improve work efficiency and bring benefits were more likely to be classified into the Excellent, Struggling, and Disengaged profiles rather than the Adaptive profile. In other words, low-achieving (i.e., the Struggling or Disengaged profiles) and high-achieving students (i.e., the Excellent profile) were more likely to appreciate teamwork compared with those average-achieving students (i.e., the Adaptive profile). This might occur because low-achieving students could get assistance, encouragement, and stimulation from working with higher-achieving students (Johnson & Johnson, 1989; Slavin, 1987; Webb, 1982), and thus could perform above their current level of development (Vygotsky, 1978). For high-achieving students (i.e., the Excellent profile), it is likely that they could improve their cognitive skills through explaining and elaborating concepts to lower-achieving students (Johnson & Johnson, 1989). However, for students in the middle (i.e., the Adaptive profile), they may not participate well in collaborative groups, especially when working with both high- and low-achieving students, because they tend to be excluded from the productive “teacher–student” relationship.

The association between profile membership and CPS performance

We also further validated the four profiles by comparing the mean CPS performance among the four profiles. The results indicated that all the profiles were significantly different from one another in terms of CPS performance except for the Adaptive profile and Excellent profiles. Students in more successful profiles tended to achieve better CPS performance, which is consistent with many prior studies (e.g., Eichmann et al., 2020).

One surprising result is that students who fit the Disengaged profile tended to have significantly better overall CPS performance than those who fit the Struggling profile. In other words, students conducting more collaboration actions were even less successful in terms of their CPS performance. This result is consistent with some prior studies (e.g., De Boeck & Scalise, 2019), yet contradictory to some others (e.g., Unal & Cakir, 2021). This may occur because the PISA interactions are computerized human–agent interactions (i.e., a student interacts with computer-simulated agents rather than with other students), which results in limited opportunities for communication or negotiation among team members and hinders the demonstration of CPS skills and the overall CPS performance when compared with the human–human approach (i.e., a student interacts with other students; De Boeck & Scalise, 2019; Dindar et al., 2020). In addition, there is no socio-cognitive conflict or structured assistance for students when they collaborate with computer agents, and thus more actions may not improve student CPS performance (Caceres et al., 2018; Doise & Mugny, 1984).

Implications, limitations, and future research

The CPS tasks developed in PISA 2015 aimed at assessing student CPS skills. To obtain objective results, the assessment adopts a standardized human–agent assessment approach in which students collaborate with computer agents to solve simulated real-life problems (OECD, 2017). The scenarios in the problems are typical ones that are expected to be encountered by students when they collaborate with “human” students. For example, students need to resolve disagreements between team members, monitor the progress of the team, and so on. One major difference, however, between the human–agent assessment approach in PISA 2015 and traditional human–human collaboration is that students in the human–agent collaboration are instructed to choose their responses to team members from a limited number of options instead of making their own reactions freely (Li et al., 2022). In other words, the PISA CPS assessment in fact provides limited opportunities for students to interact or communicate with the team members. Therefore, the PISA 2015 assessment outcomes could represent individual CPS skill levels and behaviors well but may be limited in terms of reflecting student collaborative learning results (that is not the focus of the PISA assessment).

Despite the above limitation, the results of the present study can also contribute to the computer-supported collaborative learning (CSCL) community in several ways. First, one major contribution of the present study is the identification of the four profiles of collaborative problem-solvers using CPS process indicators, which enhances our understanding of individuals’ diversified CPS behaviors. In fact, students may undergo different CPS processes and exhibit different CPS behaviors even if they both succeed and fail in the CPS task. Therefore, performance in the task, which often influences human experts in their evaluation of a learner’s progress in the CSCL literature, may not always be a reliable prediction of learning. Process even matters.

Second, the results of this study confirmed some of the findings in the CSCL literature: for example, better socioeconomic status is significantly associated with higher CPS skill levels; girls tend to demonstrate better CPS skills than boys; and high- and low-level students are more likely to appreciate teamwork. In addition, this study also provides insights into some inconsistent results across previous CSCL studies. For example, so far there is no consensus on whether more actions leads to better CPS performance or not (De Boeck & Scalise, 2019; Unal & Cakir, 2021). Based on the results of this study, we argue that it may in fact depend on the nature of the actions. Specifically, the actions recorded in the Xandar task refer to all kinds of non-chat actions (e.g., keystrokes, clicks and double-clicks, drag and drop), including both necessary actions (e.g., click on the “Join the chat” button to continue the task) and non-necessary actions (e.g., random clicks that do not lead to actual progress or solution of the problem). Hence, more actions may not necessarily be associated with better CPS skill levels or performance, which is consistent with the results of prior studies (e.g., Chung et al., 1999; Rummel et al., 2012; De Boeck & Scalise, 2019).

Third, this study has many practical implications regarding assessing and teaching CPS skills in the school settings: (1) More authentic and interesting collaborative tasks should be developed and used for the assessment of student CPS skills. (2) It would be desirable for teachers to provide timely structured assistance to students while they collaboratively solve problems, as leaving students to explore by themselves may not improve their CPS performance. In particular, individualized strategies or interventions should be exercised. To be more specific, it seemed important to enhance the engagement of the Disengaged profile students in the CPS tasks by demonstrating to them the benefits of collaborative problem-solving; for students who fit the Struggling profile, it is critical for educators to diagnose their problems in a timely manner and provide targeted guidance and appropriate exercises; for students who fit the Adaptive profile, more exploration and collaboration time should be allowed; and for students who fit the Excellent profile, more challenging CPS task exercises should be provided. (3) Educators should pay more attention to facilitating the social and group work skills of male students and those from economically disadvantaged families, as well as providing more care to average-achieving students to foster positive attitudes toward collaboration. For example, teachers could provide these underperforming students with more opportunities to share ideas and opinions in class, reward them if they make progress during CPS, etc.

Finally, from a methodological perspective, latent profile analysis has been shown to be a feasible person-centered method that can be used to reveal individuals’ diverse CPS behaviors (or other measures of interest), which may make a complementary contribution to the CSCL literature in which more traditional variable-centered methods are adopted.

This study has some limitations. First, the present study only focused on Chinese students who were assigned to interact with the one released task, Xandar, in PISA 2015. Therefore, only 17% of the total Chinese sample were included in this study. In this regard, the results of this study may not generalize well even to the whole Chinese population (or other populations) and to other CPS tasks. Second, we have identified four profiles of collaborative problem-solvers, which have been supported and validated to some extent from both the theoretical and practical perspective in this study. However, the size of the four profile groups is not even, particularly in the case of the small size of the Struggling and the Adaptive profile groups. Hence, caution needs to be exercised when interpretating the results. Future studies could further explore whether the profile results hold for students in other countries or on other CPS tasks in PISA 2015. Third, the results of this study were based on student interactions with computer agents; it is worth investigating whether the results differ under other conditions where individuals interact with human teammates. Finally, PISA 2015 has released a limited range of information regarding the CPS task—for instance, actions were numbered but not described in the dataset. Hence, although we have tried our best to explore student CPS behaviors using PISA 2015 process data (e.g., time-on-task, actions-on-task, CPS process skills), further investigation of student collaborative processes using self-collected data, or other relevant PISA data when available, needs to be made.

Conclusion

In this study, we have demonstrated that students may exhibit different behaviors and skill levels during collaborative problem-solving. Four types of collaborative problem solvers are identified in this study (i.e., Disengaged, Struggling, Adaptive, and Excellent), and these profiles are found to be significantly correlated with student demographic (e.g., gender, socioeconomic status) and motivational variables (e.g., achievement motivation, attitudes toward collaboration), as well as CPS performance. Nonetheless, the results of this study are based on the human–agent assessment approach of PISA 2015, and thus whether the results can be generalized to traditional human–human interactions is clearly a topic that merits further investigation. This study provides a starting point for such investigations.