Keywords

1 Introduction

This paper presents an epistemic network analysis (ENA) of students’ collaborative problem-solving (CPS) processes in the context of legal training to model and visualise students’ discourse at the individual and group levels. While there are existing studies that used ENA to study CPS [10, 11], some of the studies are neither based on authentic settings nor tightly connected to a theoretical perspective relevant to CPS. In this study, we apply ENA to a context of legal students working on an ill-structured problem, leveraging self-regulated learning (SRL) [12], a well-known theoretical perspective, to identify relevant collaborative actions. In this paper, we investigate how CPS strategy usage can be identified between group members and within the group over time. These insights, if provided to students and teachers promptly, may support students’ reflection on their own collaboration process, group dynamics, and provide teachers with an informed and timely understanding of students’ CPS discourse to support the collaborating groups.

2 Background

Collaboration, problem solving, communication, and effective teamwork are key skills for employability in the twenty-first century [1]. Employers across different industrial sectors require that most school and university graduates understand domain-specific concepts well. Yet, studies indicate that graduates are often ill-equipped to deal with novel, complex, real-world challenges [2]. CPS is a complex process that requires practice, awareness of group dynamics, and learning from feedback on previous activities [3]. During CPS activities, teachers are expected to monitor group processes and provide feedback to group members about the collaborative process to facilitate meaningful collaboration. In practice, however, identifying students’ collaborative actions and related group dynamics (e.g., who is doing what, how teams plan and resolve conflicts) may be impractical and unsustainable in contemporary educational scenarios [4]. In particular, in conventional face-to-face and blended learning situations, the heavy workload and limited teacher time make it difficult to monitor students’ behaviours. This affects the quality of support and feedback teachers can provide students [5]. Moreover, another challenge is assessing the contributions of individual students while accounting for how they relate to the contributions of other members over the course of a collaborative task [6].

The literature suggests that the most common methods of providing feedback to students during CPS are summative assessments, debriefing sessions, and relatively simple self or peer ratings of CPS performance [7]. However, such approaches rarely allow teachers to record all the key moments they wish to discuss during debriefings and consider the redesign of courses. This is particularly problematic in co-located settings, where evidence of events during a group activity is invisible and difficult to log [8]. In addition, while these approaches improve upon our understanding of CPS processes, the techniques used (e.g., observer ratings or self-reported surveys) tend to treat students’ individual actions as isolated and independent. Yet, individual students’ contributions during a CPS task are part of the entire group’s discourse and might affect the dynamics of the entire group since CPS is characterized as an interactive and synergistic phenomenon [6, 9].

A promising way to approach this challenge could be leveraging analytical approaches that make it possible to capture the interactive and dynamic collaborative behavioral patterns of students working in teams to generate comprehensible, actionable insights to support team reflection [5].

3 Theoretical Background

Collaborative problem solving (CPS) is a key competency in today’s fast-paced and interconnected world. CPS refers to the coordinated attempt of two or more people to share their skills and knowledge to construct and maintain a unified solution to a problem [13]. In this sense, the collaborating students are required to solve complex, ill-structured, and sometimes well-formed problems without fixed answers to achieve the goal of collective knowledge co-construction [13]. However, the CPS process is a multimodal, dynamic, and synergistic phenomenon where interactive, cognitive, regulative, behavioral, and socio-emotional aspects of collaboration happen and might affect the outcome of collaborative activity [6, 9]. While all the different collaboration aspects are important, in this paper, we focus on the regulative dimension, highlighted in previous studies [14] as critical in a successful CPS process. The regulative dimension of CPS seeks to explore how students plan, negotiate, set goals, reflect, and monitor their collaborative tasks.

According to [15], SRL is a social cognitive process achieved in cycles of (i) planning (i.e., task analysis, goal setting, and planning), (ii) performance (i.e. execution of the learning task and progress monitoring), and (iii) self-reflection (i.e., self-evaluation of outcomes and the effectiveness of their learning strategies) [16, 17]. Self-regulated learners make decisions not only about what, when, and where to study but also set and adjust goals, choose fitting learning strategies, monitor their progress, and evaluate the learning outcomes and the effectiveness of their learning strategies [12]. In this study, we present the results of our investigation of students’ collaborative patterns during CPS in an online, synchronous collaboration environment. Using SRL as the lens to interpret CPS, we code and analyze students’ online video meetings through ENA to identify collaborative patterns and how they relate to and inform each other. The following research questions guide this study:

  1. 1.

    How do students engage in CPS strategies in a group setting OR/as a group over the course of two online synchronous collaboration sessions?

  2. 2.

    How do students contribute individually in meaningful ways, with regard to CPS strategies, over the course of two online synchronous collaboration sessions?

4 Methodology

4.1 Participants and Context

This study is part of a larger project at a research-intensive university in Norway. It aims to study how CPS can be guided by providing automated feedback to student teams about their teamwork during and after CPS in online and co-located environments. The data was selected from a group of students undertaking a master-level legal technology course called Legal Technology: Artificial Intelligence and Law. This is an elective course (approximately 77 students) that is run both online (using Microsoft Teams) and face-to-face (e.g., boot camps and physical lectures). The course is intended to explore current trends and future possibilities of using technology, software and computer analytics to provide legal services and justice. Four teachers and four mentors facilitated the course and the group projects. As part of the course assessment, students were asked to work on an ill-structured and open-ended problem related to the course content (e.g., Legal Technology: Artificial Intelligence and Law). A subset of all students enrolled in the course were recruited to participate in the research objectives. These students comprised four groups. The students agreed upon the work mode (e.g., how to meet, the resources to use, the type of task to choose, etc.) and had 11 weeks to work on the project. For this initial investigation, we chose to investigate data from one group, which was composed of four female exchange students. As part of the course project, the group worked on the topic called GDPR Fine Calculator. The project's goal was to create a legal technology solution as an application that can predict the amount of a GDPR Article 83 administrative fine. The group had two online meetings, which lasted approximately 3.5 h. This data was utilized after obtaining informed consent from all the group members and gaining approval for the TeamLearn project from the National Ethics/Scientific Committee in Norway.

4.2 Data Processing and Coding

Video data were transcribed verbatim, and later, researchers converted it into a qualitative data table, where each row contained one student’s turn of talk. Metadata was added by including the meeting number and the timestamps representing the start and end of an utterance. To code the transcribed video data for use in ENA, we used a hybrid approach where codes were inductively developed by looking at the data to identify aspects relevant to CPS and deductively developed based on the three stages of SRL theory: (i) Planning/forethought, (ii) performance, and (iii) self-reflection [15]. Based on this hybrid approach, we identified ten codes in the data: task analysis, role allocation, goal setting, monitoring, questioning, subject matter knowledge, contribution, affirmation/confirmation, socio-emotional, and reflection (see Table 1 for explanations and examples). Social moderation was used to code the data with four raters coding it manually using a spreadsheet and meeting several times to compare the results of their respective codes and settle any discrepancies. The entire dataset was composed of 480 utterances/turns of talk. It is important to note that during the coding, one sentence/turn of talk could include multiple codes (e.g., representing planning but also subject matter knowledge). In this case, multiple codes were assigned.

Table 1. The qualitative coding table illustrating codes, explanations, and data examples
Full size table

4.3 Data Analysis

4.3.1 Analysis of Code Frequencies

The analysis started with exploring the code segments within student discussions across the two group meetings. We began our analysis with a quantitative count of code occurrences to ensure how often students participated in CPS strategies individually and across the meetings. Frequency distribution was calculated for each code occurrence across meetings and individual students (see Table 2).

4.3.2 An Epistemic Network Analysis Approach

Once we determined differences in the frequency of the codes, we sought to understand how the CPS behaviors were employed in connection with one another across the two sessions. For this purpose, we use ENA to visualize the co-occurrence of codes within the collaborative sessions. To form the ENA models, we used the ENA web software (https://www.epistemicnetwork.org/). The ENA model we constructed used a conversation segment based on the meeting students were in and employed a moving window of 4 lines to limit connections to discourse occurring too far outside the recent temporal context [18, 19]. This decision was based on the assumption that when students are working in teams, their actions or responses could be influenced by what their peers say within the moment or micro-context, which in turn forms the chronological sequence of the recent dialogue segments [18, 19]. The model is based on an adjacency matrix generated by aggregating across all lines for each unit of analysis in the model. In our case, we made two models to address the group dynamic vs. individual student question. The first model used units of analysis defined as meetings composed of students to compare how students behaved within the two meetings; meetings 1 and 2 had 3 and 4 student units, respectively. The second model reversed this orientation, comparing students who each had two units, one of which represented their behavior in each meeting.

Model 1 included the following codes: Reflection, Goal Setting, Task Allocation, and Task Analysis. This allowed for the exploration of the planning behaviors across meetings. To further facilitate the comparison of meetings, the ENA model used a means rotation comparing first and second meetings along the x-axis. Model 2 was more concerned with individual student actions within the planning process, including the following codes: Subject Matter Knowledge (SMK), Monitoring, Contribution, Feedback, Questioning, Affirmation, Confusion, and Socio-emotional. We drew this distinction because the latter codes, like Questioning, are more based on individual contributions to the discourse. We wanted to be able to infer how students were contributing to the larger goals of planning through these behaviors. Because we were primarily concerned with differences between individual students, we used the evenly spaced, unit circle ENA plot, which relies on the strength of code connections and disregards spatial representations for codes. Both ENA models defined conversations as all data lines associated with a single value of Meeting and used a moving window of 4 lines.

5 Results

We began our analysis using counts of code occurrence to consider how often students participated holistically in CPS strategies across the meeting times and as individuals. When we consider the meetings as a whole, it seems that the group dynamic engages more in particular behaviors, Task analysis, Subject matter knowledge, and Confirming/Affirming in the first meeting. In contrast, the second meeting sees an influx of Monitoring and Socio-emotional engagement. This indicates a potential transition point between meetings that we explore further using ENA. Students also take on different roles when we consider counts of the number of times they engage in CPS strategies. Student 3, for example, is more likely to use Subject-Matter Knowledge than any other students, and Students 1 and 3 are more likely to engage in Task Allocation than Students 2 and 4. In part, this may be because Student 4 was only active during a portion of Meeting 2, thus missing opportunities to divide labor within these meetings. Their absence also accounts for the reduced amount of coded lines for them (See Table 2).

Table 2. Counts of code occurrence between Meeting 1 and 2 and across student participation in both meetings.
Full size table

Given the discrepancies in CPS strategy usage across meetings and between students, we sought to understand better how these differences manifest in coordination with one another. To do this, we use ENA to quantify and visualise the connections between CPS strategies as students engage as individuals and as a group over the course of the two online synchronous collaboration sessions.

First, we look at Model 1, which compares participation in Meetings 1 and 2 using a means rotation to maximize group differences (Fig. 1). This model yielded Pearson correlation values of 0.97 (X-axis), demonstrating that the model itself is a visually accurate representation of the underlying data. The model explained 0.52 of the variance on the X-axis. Along the X axis, a two-sample t-test assuming unequal variance showed that Meeting 1 (mean = −1.56, SD = 1.08, N = 4) was statistically significantly different at the alpha = 0.05 level from Meeting 2 (mean = 1.56, SD = 0.55, N = 4; t(4.47) = −5.14, p = 0.01, Cohen's d = 3.63).

Fig. 1.
figure 1

The comparison plot between meeting 1 (red) and meeting 2 (blue) illustrates the overall CPS discourse for meetings 1 (red) and 2 (blue) for the entire group. In meeting 1, task analysis and Goal Setting are central to several other group CPS behaviors. In meeting two, there is a shift to task allocation, monitoring, and reflection as the primary SRL dimensions.

Full size image

In meeting one, the SRL planning dimension of Task Analysis is central to several other group CPS behaviors, Goal Setting, Monitoring, and Monitoring. Meeting 2 shows stronger connections between Task Allocation, Monitoring, and Reflection. Task Allocation was expected to be prominent in the early weeks as students are sorting out a strategy for project responsibilities, but the occurrence counts demonstrate its presence across the two sessions. However, when visualized with the comparison plot (Fig. 1), it is clear that the role of Task Allocation shifts from Meeting 1 to Meeting 2. In Meeting 2, students are both reprising roles that they have as well as Monitoring and Reflecting on the progress within their roles. This result is less surprising given that as the group progresses with the task, they are expected to concentrate on checking on the task's progress and identify any issues to address (Monitoring) other than planning. For example, Student 1 says, “So basically, (Student 4), what we all said, I don't know about you, but we were all working on our cyber security papers, so we didn't really do much.” Here, they are discussing where they have gotten in their own work (Monitoring) in relation to the Task Allocation of everyone’s current job. This connection does not exist in Meeting 1. However, while the findings highlighted some connection to Task Allocation, the groups’ discourse revealed that the group did not have assigned leaders and task allocation was not done explicitly. S1 highlighted this by making the following comment: “Also…. I feel like we didn't really assign any. Like we didn't say, ‘Okay, you're responsible for this’ and ‘you're responsible for that’. We're just working on everything a bit together”.

A similar issue was identified in meeting 2, where student 4 was wondering how the group would proceed with the task:

“Yeah, so I uploaded the two things I did, which was the template. I was just sort of thinking through everything that we needed to include. I'm not sure if I'm missing anything out; you can just add that in. And then I just did a mockup of the design, which was somebody did a paper mockup of the design, so I just put that into a computerized format. But other than that, I was just wondering when you wanted to start with the writing and if I should just start with that, whether we're doing it all together and how that works”.

In this response from student 2, we can see the relationship of Questioning to the Task Allocation code; as they surface their own wondering about the process, they allow their fellow students to chime in on what the next steps should be. For this reason, we focus on the more individual codes for Model 2.

Model 2 focuses on individual students’ behaviors across the performance codes. The model yielded a Pearson correlation of .91 and .53 across the X and Y-axes. This indicates that the horizontal axis is well fit and an accurate visual representation of the data, while the vertical axis may not be. The two-plotted dimensions can be described as moving from the pragmatics of CPS, like Subject Matter Knowledge (SMK) and Clarification, to the more uncertain behaviors, like Questioning and Confusion. We use an evenly distributed unit circle plot to better visualize the students in relation to all of the codes (Fig. 2). From these plots, students 1–3 are similar in the types of connections they are making within the group work for CPS strategies. However, Student 4 is quite different in their network, likely due to them missing key planning elements in Meeting 1.

Fig. 2.
figure 2

Overall Comparison of Student 1 (Red), Student 2 (Blue), Student 3 (Purple), and Student 4 (Green).

Full size image

We can further observe differences between meetings in this plot. For each student, the units in the upper left are from Meeting 1, and the units further to the bottom right are from Meeting 2. Student 4 only has one node representing the mean because they did not participate in Meeting 1. Thus, their mean is equivalent to Meeting 2. There is no statistically significant difference between the students on either axis.

However, although no statistical differences were observed, the analysis showed differences in how students contributed to the group discussions. For example, as illustrated in Fig. 2, in meeting 1, student 1 (red) was strongly associated with Affirmation, Contribution, and Questioning behaviors. In contrast, student 2 (blue) was connected to Affirmation, Contribution, Questioning, and Socio-Emotional behaviors. Student 3 (purple) connected similarly to student 2 (e.g., Affirmation, Contribution, Questioning, and Socio-Emotional behaviors). It was observed that SMK was equally distributed among the students. In contrast, given that student 4 missed the first meeting and joined the second meeting late, their connections were different from the other students. For example, although student 4 had connections between Affirmation, Contribution, and Questioning, these were weakly connected. Moreover, student 4 has no connection to the codes of SMK and Clarification. This can be attributed to several things: Student 4 was not familiar with the group dynamics or things discussed before the meeting, which is crucial when working with a group. However, although Student 4 joined late, they tend to connect contributions to both socio-emotional and questioning more often than any of the other students. One possible explanation is that when Student 4 joined, they wanted to make up for their absence to hit the ground running and contribute quickly.

In addition, among all four students and between both meetings, Affirmation is an important behavior that tends to connect to all other codes. The robust interconnection between affirmation and most CPS-related codes in both meetings (see Fig. 2) is likely attributed to the presence of uncertainty towards the task, which in both meetings was often demonstrated through expressions such as “I think”, “I don’t know” or “I'm not sure.“. For example, students 1, 2, and 3 were engaged in the following discussion during meeting 1.

“I think if we because I was wondering about that as well, but if we stick to the company, it could be used for both. I don't know”.

“I also, I don't know if we get more information about the paper from class because I think we're talking about the project and the planning, but they don't give us much about the paper». «Does someone know the word limit? Actually. I don't remember”.

“I don't know”.

[20] highlighted that such expressions not only convey the speaker's uncertainty but also implicitly request validation from the listener. As a result, students engage in a mutual exchange of validation, reciprocally seeking and providing confirmation during their interactions and interventions with peers.

6 Discussion and Implications for Future Research

Identifying students’ CPS strategies and related group dynamics (e.g., who is doing what, how teams plan and assign roles) may be impractical in synchronous collaboration settings [4]. The present study supports the potential of leveraging data from students’ CPS activities (e.g., online meetings) to understand how students engage in CPS strategies as individuals and as a group. Model 1 demonstrates that there are differences in how students engage in CPS strategies at the different stages of group work. Furthermore, model 2 demonstrates students 1–3 are similar in the types of connections they are making within the group work for CPS strategies. However, although students’ CPS actions were relatively similar across the two observed sessions, Student 3 appeared to dominate the discussions in both meetings, demonstrating strong connections between Subject-matter knowledge, Task analysis, and Goal setting. Moreover, Student 4 stood out as an outlier with limited connections to CPS actions such as Subject-matter knowledge, Clarification, and Socioemotional. This is likely due to student 4 missing the key planning elements in Meeting 1 and arriving late for the second meeting. In addition, group members likely possessed varying levels of skillset and knowledge towards the task, hence demonstrating different standards for metacognitive monitoring [21]. This implies that collaborative teams could benefit from structured protocols embedded within the learning design or collaborative tasks, explicitly highlighting expectations regarding participation and contribution to the group task.

While the findings highlighted some connection to Task Allocation, the groups’ discourse revealed that the group did not have assigned leaders and Task Allocation was not explicit. The lack of clear Task Allocation during the collaborative activity suggests that the students may not have been aware of the expected roles in the group activity. In such a case, if teachers identify such a situation early enough, teachers can promote role distribution by explicitly asking students to assign roles and providing opportunities to practice and reflect on their roles. Teachers can also provide feedback to students on their collaborative skills and strategies and help them develop metacognitive strategies to improve their learning.

Moreover, the findings also showed that students rarely engaged in Reflection. Yet, it is a crucial component of CPS and self-regulation, as it helps students consolidate their learning and identify areas for improvement [14]. In practice, if teachers are presented with information about such discourse (e.g., limited reflection), teachers can support reflection by asking students to share their thoughts and providing meta-cognitive scaffolding such as prompts and guidelines for self and group monitoring to support their problem-solving endeavors [15].

Moving towards considerations of research question 2, on modeling CPS within groups and across individuals, we have demonstrated through this analysis that ENA allows for differentiating patterns across meetings and between students, even when students are not present for each meeting. That being said, questions arise for the best parameters for such a model. For example, in such a small group, window size may have a larger impact on how students are perceived in making connections in the model. They may receive inflated connections between two codes even if they only engage in one manner just because of the frequency of the other code. We addressed this in this paper by focusing on the comparison between meetings and students, so sheer presence was not the focus.

7 Limitations and Directions for Future Work

There were three main limitations in the current study. First, the analysis was based on one group of four students and video data from meetings. While this could provide insight into students’ CPS processes, a detailed dataset with more groups and data sources that capture students’ learning processes could improve the interpretation of students’ CPS behaviors. For example, from the analysis of the discourse, students referred to information that seemed to belong to other sources other than the video data transcribed. For example, student 3 asked other group members in meeting one to ‘do some things from the to-do list.’ This means that relying on one data source to understand students’ CPS processes may not be adequate. In this regard, future research could consider multiple groups and leverage multimodal data sources (e.g., digital traces, assignment drafts, and revision history) to understand the complexity of CPS in asynchronous settings better. Second, this study used Zimmerman's [15] SRL framework as the lens to identify and model CPS strategies. While this framework is relevant and widely used, we found it more oriented towards the cognitive aspects of regulation and focusing more on individuals other than group-level regulation. Since CPS involves multiple people and is inherently a dialogical process, future work can consider frameworks that consider the cognitive, meta-cognitive, and social aspects of regulation. Lastly, although ENA revealed patterns and the nature of discourse individual students were engaged in during the group meetings, it was difficult to establish the interactional, emotional, and temporal patterns of students’ collaborative process. Future research can overcome this challenge by using approaches that account for the recent temporal context, such as ordered network analysis [10] and combining ENA with other analytical approaches, such as social network analysis and sequence mining, to detect social interactions and CPS sequences and how they evolve over time at an individual and group level.

8 Conclusion

In this paper, we sought to identify the kind of collaborative actions manifested during CPS activities at an individual and group level in an online synchronous collaboration environment. Using ENA and SRL as the theoretical perspective to identify evidence of CPS actions, the results showed differences in discourse between students at different stages of the CPS process. Moreover, models showed specific SRL actions less prevalent in students’ discourse (e.g., reflection). While the current study is at an exploratory level whose findings cannot be used to draw strong conclusions about CPS actions, the insights presented in this paper point to the potential of ENA in modeling individual and group behaviors during CPS. This information could be employed by educators as a basis for providing timely feedback and adapting learning design to support CPS processes.