Learning to monitor and regulate collective thinking processes

Abstract

In this paper, we propose a conceptual framework to guide the design of a computer-supported collaborative learning intervention to help students learn how to improve collaborative knowledge building discourse at the level of the small group. The framework focuses on scripting individual and collective regulatory processes following collaboration. Individuals are required to evaluate their team’s chat transcripts against rubrics to score discussion quality. These theoretically supported rubrics provide individuals with concrete examples of desired communication processes. After this individual assessment, the team is prompted to discuss their individual scores, identify strengths and weaknesses of their collaborative discourse processes, and select strategies to improve the quality of their collaborative discussion in a future discussion session. To evaluate our framework, we created a prototype of an online system and asked students to use it over ten weeks as part of five discussion sessions. Participants included 37 students, divided into 13 teams, from an undergraduate online course in information sciences. We used quantitative and qualitative analysis techniques to examine students’ collaborative processes over time, with teams as the main unit of analysis. All teams followed the same general activities, but there were two different conditions for scripting individual reflections that preceded the collective sense-making activity: one (Future-thinking) focused on pushing individuals to pay attention to advice on how to improve existing processes in future sessions and another (Evidence-Based) pushed individuals to pay closer attention to the chat transcripts to provide evidence for their group process scores. Our results suggest (1) use of the framework can help students’ monitor and regulate collaborative processes and improve collaborative discourse over time and (2) the Evidence-Based condition can help students engage in higher quality reflective analysis.

Learning to monitor and regulate collective thinking processes

There is an increasing recognition that collaboration can provide opportunities for more sophisticated forms of learning than individual activity, because it pushes people to think about ideas they would likely not have considered alone (Chi and Wylie 2014). There are also many demands for collaborative competencies as our society becomes increasingly dependent on teams to solve complex problems or lead innovation (Frey and Osborne 2017; West 2007). Though collaborative activity is becoming increasingly necessary for everyday life, studies indicate that collaboration is a skill that is often underdeveloped (Carroll, Jiang, & Borge 2014; Kozlowski and Ilgen 2006). Specifically, collective sense-making processes that occur as groups synthesize information into collective knowledge and negotiate what is known are prone to a variety of problems that lead to poor group performance and learning outcomes (Barron 2003; Borge and Carroll 2010; Borge and Carroll 2014; Kerr and Tindale 2004; Kozlowski and Ilgen 2006; West 2007). Though many researchers in Computer Supported Collaborative Learning (CSCL) have devised ways to help students enhance collaborative discourse processes, many do so by actively guiding and constraining how students collaborate within a collaborative system. We argue, along with others, that the key to improving collaborative activity is to actively guide and constrain how students make sense of and regulate their own collaborative activity (Borge and White 2016; Järvelä and Hadwin 2013). This type of metacognitive regulation at the level of the group has been referred to as socio-metacognitive expertise (Borge and White 2016).

Although there have been attempts to help students develop socio-metacognitive expertise, attempts thus far have only been partially successful (Borge and White 2016; Hogan 1999a; Järvelä et al. 2013). Many are able to help students increase awareness of processes or strategies, but none have been able to help students regulate collaborative processes so as to significantly improve collaborative activity (Hogan 1999a; Järvelä et al. 2013).

In this paper, we extend the work on socio-metacognitive development by proposing a framework for supporting socio-metacognitive development with technology, examining the impacts of different types of reflective scripting, and devising a way to examine changes in collaborative discourse quality over time.

We begin the paper by defining collaboration and providing an example of a real-world team engaged in poor sense-making activity. Drawing on relevant literature, we explain why such collaborative activity is prone to poor sense-making and why regulation of group activity poses so many challenges for students. We use this information to inform a conceptual framework for computer-supported group regulation. We then describe a study where we tested the foundational principles of the framework and report findings that show how use of the framework helped students learn how to modify their own collaborative interactions and significantly improve them over time. As such, this study provides a new generalizable method for supporting socio-metacognition with computer support that can help students improve collaborative processes for themselves, thereby contributing to what is known about computer-supported group regulation.

Defining collaboration

Our work is highly influenced by theories of group cognition (Stahl 2006) and knowledge building (Scardamalia and Bereiter 1996). Stahl (2006) argues that collaboration is a form of nested cognition where knowledge building occurs at multiple levels of scale, at the individual, the group, and the community level. During collaboration, the construction of shared meaning occurs as individuals work to make sense of new information or artifacts and then share their ways of knowing with the group. Once individuals externalize their thinking through language it moves from individual to group cognition. The group must then work to synthesize information from individuals into a form of shared knowledge and negotiate what is known at the level of the group to control what information is transferred to the larger community (Stahl 2006). The purpose of this form of sense-making is not to learn facts or develop skills, but to build new knowledge (Scardamalia and Bereiter 1996).

Given the complex nature of collaborative activity, it is not surprising that the term “collaboration” has been used in varying ways to describe a wide range of collective activity. Unlike theories of knowledge building, which focus on understanding and evaluating collaborative knowledge construction at the level of the community (Scardamalia and Bereiter 2006), we are concerned with understanding and evaluating collaborative knowledge construction at the level of the small group. Building on previous definitions of collaboration (Roschelle and Teasley 1995; Stahl 2006), we define collaboration at the level of the small group as a synchronous activity that occurs as individuals engage in collective thought processes to collectively synthesize information and negotiate what is known in order to create shared meaning, make joint decisions, and create new knowledge. New knowledge can include the development of new artifacts, solutions to problems, or ways of thinking, but does not include the transfer of existing information from one member to another.

From this framework, we conceptualize collaboration at the level of the group as a specific type of communication that occurs when groups work to create new knowledge, defined by the existence of two separate, yet complementary, types of macro-communication patterns: collective information synthesis and collective knowledge negotiation. High quality collaborative sense-making occurs when teams produce collaborative behaviors associated with high quality collective information synthesis and collective knowledge negotiation processes.

Contextualizing the problem

It is difficult for students to engage in the types of sophisticated collective sense-making that good collaboration requires. Students are usually so focused on completing the collaborative task that they do not pay attention to their own processes. A good example of this problem can be seen in an excerpt from Borge and Carroll (2010), where a software development team was working on a collaborative design project. This team came from a course on human-centered design and was supposed to be predicting the types of problems that users might experience when trying to use a photograph-sharing website they developed. Once they identified problems, they were supposed to propose the best type of support to help the user overcome them. The team was under pressure to finish the activity. One student, John, suggested that a user might experience problems uploading pictures. The other students quickly agreed and asked him to log in to see how many pictures a person could upload at once, but John could not log in because he forgot the password (turns are numbered for ease of referencing):

• 1. Yu: Yeah go to the homepage... yeah, there you go...

• 2. John: What’s the password, anybody...

• 3. Bob: Um the other Mike has that, but he’s not here... I don’t know the password of---

• 4. Juan: Why couldn’t he make it something easy like one, two, three, four...

• 5. Bob: He made it something easy but it was just like---

• 6. John: Group project?

• 7. Yu: Try it.

• 8. Juan: Did he actually send you the---

• 9. John: ---No, I just remembered it was something stupid.

• 10. Yu: He said it out loud to us.

• 11. Juan: Oh... I can’t remember!

• 12. Yu: Security question?

• 13. Bob: Um?... Calendar help training solution, we could just provide like a document that showed you how to do stuff.

The team unsuccessfully tries to determine the password (Turns 1–12). Unable to log in to try uploading pictures so as to develop a scenario around providing support for those features, Bob provides an alternative scenario the team could develop: to provide their users with help using the calendar feature (Turn 13). The team agrees with Bob after this episode and develops a scenario for calendar support.

Even though the task required the team to identify a common user problem, the team was so busy trying to complete the assignment that they failed to notice the one they themselves were experiencing: forgetting the password. It is tempting to think that they were unaware because they did not care about the quality of their work; they just wanted to finish the assignment. While this may be true at times, it has been our experience that even students that care a great deal about their work fall victim to common group-process problems.

It is much easier to identify group problems when you are reading a transcript of activity than it is to see these issues as they occur in real time (Cooke et al. 2000). This is because collaboration is a complex multifaceted form of collective thought, prone to a great deal of human error (Kerr and Tindale 2004), and many of these errors stem from poor communication patterns that teams cannot improve without instructional support or specialized training (Kozlowski and Ilgen 2006). Unpacking these patterns of communication can provide us with an understanding of common problems that interfere with high quality collaborative activity so as to develop means to address these problems with technologically enhanced support.

How communication patterns impact collaborative processes

Studies show that teams are prone to poor patterns of communication during collaborative sense-making discourse regardless of the age of team members, their individual cognitive abilities, level of expertise, or the amount of time the team has spent working together (Barron 2003; Hogan 1999a; Borge and White 2016; Kozlowski and Ilgen 2006). While experts in group decision-making have developed long lists of potential group process problems (see Kerr and Tindale 2004) and theories of how teams come to develop new knowledge (Fiore et al. 2010; Salas et al. 2013), we focus on problems specific to building knowledge as defined by Stahl (2006). These are problems that interfere with how teams collectively synthesize information from individuals to create shared meaning and how the team negotiates what is known to ensure that new knowledge generated by the team is of good quality.

Collective research suggests there are key micro-communication patterns known to impact the quality of collective information synthesis and knowledge negotiation (see Table 1). These micro-communication patterns include verbal equity, developing joint understanding, idea building, exploring alternative perspectives, proposing high quality claims, and engaging in constructive discourse.

Table 1 Summary of macro and micro patterns associated with high quality collaborative communication behaviors

Verbal equity can impact the extent to which a team can integrate the perspectives of different team members. To create new knowledge, a group must first share individual information and work to synthesize it into the group’s collective knowledge base (Stahl 2006; Roschelle and Teasley 1995). Many problems can interfere with this process and cause information to be held back (information loss) or misinformation to build into erroneous knowledge. For example, teams have a tendency to ignore knowledge held by a small minority of the team and accept knowledge held by the majority of the team (Stasser and Titus 2003). Individual tendencies to dominate group discourse can also prevent others from sharing relevant information that can be used by the group to inform decisions and problem-solving processes (Barron 2003; Borge and Carroll 2014; Borge et al. 2012; Hogan 1999b; West 2007). As such verbal inequities can lead to information loss, as speakers with relevant knowledge or unique perspectives are ignored or dismissed by the team, which is likely why teams with inequitable patterns of communication have less potential to solve a variety of different problems (Woolley et al. 2010).

It is also necessary for teams to develop joint understanding of shared ideas in order to establish common ground, “mutual knowledge, mutual beliefs, and mutual assumptions” (Clark and Brennan 1991). When information is shared, each member must interpret that information from their own perspective. In doing so, individuals often misunderstand what others mean when they share ideas. These small pieces of misunderstood information can have a snowball effect every time the team builds on this seemingly insignificant piece of false information; larger decisions and actions resulting from these misunderstandings can cause big problems for a team (Borge et al. 2012; Stahl 2006). As such, verbal contributions need to be checked by individuals to correct misunderstandings and develop joint understanding, i.e., mutually understood information (Roschelle and Teasley 1995; Schegloff et al. 1977; Stahl 2006). Thus, teams that do not take time to check their understanding by rewording, rephrasing, or clarifying shared information may be more prone to erroneous knowledge building.

In order for teams to move from sharing information and checking their understanding of information to creating new knowledge, previously held ideas have to be extended through discourse (Bereiter 2002; Scardamalia and Bereiter 1996). Human tendencies to simply acknowledge or ignore team-member contributions, rather than explore and extend them, can lead a team to simply transfer existing individual ideas to other members rather than create new knowledge that no one person had prior to collaboration. Failure to build on and extend shared ideas can also lead to information loss or misunderstandings, resulting in poor decision-making and performance outcomes (Barron 2003; Borge and Carroll 2014).

Sub-optimal, communication processes also negatively affect the quality of collective knowledge negotiation. A common problem during collaborative discussions is the tendency for individuals to agree with the first viable idea, rather than explore differing perspectives (Atman et al., 2007; Ball et al. 1994). Under time pressure, these problematic tendencies increase. When teams feel pressure to finish a task, they are more prone to accept ideas without considering alternatives and reject minority opinions regardless of quality (De Grada et al. 1999). It is important to seek out competing ideas in order to combat groupthink, where groups avoid conflict at all costs by striving for consensus and ignoring alternatives (Callaway and Esser 1984; Janis 1972).

When alternative perspectives are proposed, a group must work to compare alternatives, carefully analyze related information, and critique shared ideas. This process has been shown to pose both cognitive and socio-emotional problems for teams. Cognitively, students’ argumentation quality is generally poor because they fail to critically analyze information and provide logical, fact-based claims (Duschl and Osborne 2002; Noroozi et al. 2013; Weinberger et al. 2007). Socio-emotionally, evaluation of ideas is an emotionally-charged, social process that is often mismanaged (Edmondson 1999). Problems associated with how information is evaluated can negatively impact the psychological safety of a team, the feeling that it is safe to take cognitive risks and share different perspectives without fear of harsh judgment (Edmondson 1999). Maintaining trust and productively managing errors are important aspects for innovative knowledge-creating groups and workforce teams (Cannon-Bowers and Salas 2014; McGrath, 1999b). Destructive discourse, the type of discourse that devalues speakers by rejecting, or belittling, or ignoring their ideas, can make people feel unsafe to share their experiences and negatively impacts a group’s ability to learn from each other, overcome difficulties, and improve over time (Edmondson 1999).

Collectively, these six micro-patterns can impact how groups develop knowledge and the quality of the knowledge that is developed. They impact the extent to which a team uses its members as cognitive resources, establishes common ground, and develops new knowledge that did not exist before collaboration. They can also impact the quality of knowledge that is developed, whether the group considers different or contrasting ideas, builds on logical or credible information, and whether the team critiques ideas in a manner that will maintain an environment where members feel that their ideas and experiences are welcomed and valued by the team. The collective literature on group processes for knowledge building imply that improving collaborative communication behaviors requires that students carry out these six micro-communication patterns well, something we know many teams to be incapable of doing. Though the educational community largely agrees that students need to improve their collaborative activity, there is much debate on how best help them do so.

Helping students to improve collaborative activity

Fischer et al. (2013) argue that the main cause of problems associated with poor collaborative discourse patterns stems from students’ conceptual models of collaborative activity: their understanding of what the process should look like, the rules of the activity, the roles they should play, and the different discourse moves they could make. Students’ conceptual models of collaborative activity are often not fully formed or at odds with optimal collaborative procedures. One of the most influential theories in CSCL has been dynamic memory theory, which argues that memory is largely episodically organized, ever-changing, and dependent upon the internalization of scripts (Schank 1999). The idea of providing externalized scripts of desired collaborative activity emerged as a way to enhance existing memory systems. When students enact scripts, others see these examples and internalize these models of collaborative activity (Dillenbourg and Hong 2008; Schank 1999).

Scripting of collaborative activity can work quite well in many contexts, but there are a number of open questions that remain. For instance, some questions include: how should the level of needed support be determined, how should activities be navigated, and when should support fade. Too much, too little, or the wrong kinds of scripting can cause problems for groups (Dillenbourg 2002; Stegmann et al. 2011), which is why Fischer et al. (2013) argue that we are still in need of developing theories to strategically guide use of scripts to support collaboration.

The communication patterns we highlighted in the previous section are well-known in CSCL, but are rarely simultaneously supported with scripting. Most studies focus on scripting a few of the micro-processes. For example, scripting of argumentation may focus on supporting the weighing of alternative perspectives and quality of claims, but may not focus on aspects of idea-building or psychological safety. What is more, common scripting techniques, like constraining or modifying activity during collaboration, is likely not the best approach, because it could lead to over scripting and the creation of an inauthentic, sterile, collaborative context (Dillenbourg 2002).

Recently, increasing attention has been given to regulation of group cognition, which is similar to individual regulation (Zimmerman 2002), but at the level of the group (Borge and White 2016; Järvelä and Hadwin 2013). Researchers argue that learning how to regulate cognition at the level of the group can enhance the quality of collective thinking and help the group to improve and adapt over time (Kozlowski et al. 2009). Such an approach would require providing groups with opportunities to examine their processes and compare them to models of competence in order to identify problems and develop plans to improve their own activity.

Järvelä and Hadwin (2013) have conducted pioneering work on the design, development, and testing of technological tools to support group regulation. Though this work has succeeded in helping student to become more aware of their collaborative activity, teams in these studies are largely unable to direct regulation at desired outcomes to improve activity, often failing to understand why regulation is needed. However, technological support in these studies focused on enhancing awareness and planning, which is only a small aspect of the regulation process. There are many problems throughout the process of regulation that can prevent students from regulating individual and collaborative activity.

Problems associated with regulation at different levels of scale

Similar to collaboration, the process of regulation is dependent on a series of interrelated processes that are prone to error. Findings from decades of research on self-regulation provide us with insights on the process of individual regulation and all the potential problems that can prevent it from occurring. Winne and Nesbit (2009) propose that the likelihood of regulation is determined by a series of conditions that must be sequentially met: recognition of a problematic state, accurate appraisal of the problem, ability and desire to apply a strategy, and access to sufficient cognitive capacity to exert cognitive effort on the process. As shown in Fig. 1, we add attention to the sequence of events, as it was not explicitly addressed by Winne and Nesbit’s model.

Fig. 1

A model of the process of self- regulation “in the wild”, as informed by theoretical and empirical work

Attention plays an important role in regulation by serving as the initial gateway to what can be regulated. If a student is not paying attention to a process or behavior during an activity, then they cannot be consciously aware of the impact it has during the activity and cannot regulate it (Koch and Tsuchiya 2007; Lamme 2003). If a student is paying attention to a process or behavior and it comes to their conscious awareness, then there is an opportunity for regulation, provided the student can also identify a problem.

In the event that students succeeds at identifying a problem during an activity, the student must shift attention from cognitive and physical activity to metacognitive activity as they engage in gap analysis as part of reflection: analysis of the difference between existing and desired activity (Weick et al. 2005). Effective analysis during reflection includes sense-making activity targeted at discerning questions: what happened, why did it happen, why did we act/think/feel as we did, how does this impact goals for desired activity, how does it impact others, and what are the implications for future activity (Nesbit 2012). This specific type of reflective processing is something individuals rarely to do well without guidance (Gabelica et al. 2014).

As Winne and Nesbit (2009) explain, if a student manages to accurately assess a problem state in comparison to a desired state, they still need access to a repertoire of strategies to select one that best matches the present context in order to plan future activity. They add that a student also needs time and space to think about the processes they are regulating in order to be able to carry out selected strategies and continue the cycle of regulation. According to Winne and Nesbit (2009), meeting the requirements at each step in the model makes subsequent steps more likely. This explains why regulation of processes and behaviors often fail, because it is difficult to make it past the first few steps.

All of the problems associated with regulatory activity are compounded when moving from individual to group cognition. Group cognition occurs during collaboration when individual thinking processes are externalized through language and groups work to synthesize and negotiate what is shared to create new knowledge (Stahl 2006, 2010). As a result of the nested nature of collaboration, many of the problems that interfere with individual regulation of cognition also emerge at the level of the group. Thus, we propose that the process of group regulation looks similar to the model in Fig. 1, except that it is occurring simultaneously for individuals and the collective (see Fig. 2).

Fig. 2

Model of regulatory processes during collaboration, with individual and group regulation occurring simultaneously

Just as attention poses problems for individual regulation, it too affects and further complicates regulation of group cognition. Collaborative activities pose large demands on attention, as individuals must pay attention to their own thoughts and behaviors, to those of others, to interactions, to developing joint attention in coordination with differing goals, and to the products to be completed. Time constraints can also push groups to focus attention on completing products at the expense of paying attention to group processes (Kerr and Tindale 2004). This is why many teams do not pay attention to process problems that occur during collaboration and are completely unaware of their communication patterns (Borge and Carroll 2010, 2014). Recognizing this problem, human-factors experts have argued for the need to enhance attention on team processes by giving teams access to video or communication archives after collaboration, as a means to more accurately reflect on activity (Cooke et al. 2000).

Even with access to archived process information, recognizing process discrepancies can be a challenge because ill-structured problems common to collaborative activity have no correct path to solution. Moreover, recognition of discrepancies is not sufficient to correct collaborative problems. If one or multiple members recognize that a group process problem exists, they have to find a way to shift the group’s attention from working on the product to addressing the process problem in order to facilitate joint comprehension, diagnose the problem, collectively agree on a strategy, and coordinate efforts to try to correct the problem.

The collective research explains why so many collaborative teams fail to correct process problems and improve collaborative activity. Teams focus more of their attention on completing the product, i.e., the collaborative assignment, than they do on the quality of their collaborative processes. As such, teams are unable to (a) pay attention to patterns of communication, (b) be aware that a communication pattern is problematic, (c) diagnose the problem, (d) or apply strategic knowledge to correct the problem during collaborative activity. Given the nested nature of collaboration, it is also possible that different members may recognize different problems, diagnose problems differently, have different strategic knowledge to address the problem, or have differing levels of cognitive space and desire to address process problems.

A framework for computer-supported-group regulation

The literature on group process problems and regulation highlights the complex nature of group regulation of collaborative activity. Designing theoretically informed technological support to help students learn about and regulate collaborative processes requires we find ways to enhance individual knowledge and awareness of collaboration, while reducing the complexities associated with regulation of group processes. In order for teams to develop their ability to regulate collaborative activity, they need opportunities to practice collaborative activity while engaged in real course content, so as to carry out collaboration and assess it multiple times. Such collaborative activities would push teams to engage in both content and process learning. We designed a technologically-supported collaborative activity that provides these types of learning opportunities through sense-making of difficult course concepts. The basic premise of the activity is for students to individually prepare for collaborative discussion, engage in it, and then take time to individually and collectively make sense of their collaborative processes for the purpose of improving it. Figure 3 shows the six phases of this activity, which we describe in more detail below.

Fig. 3

Orchestration of sense-making activity between the individual and the group

Ongoing course sense-making activities provide good opportunities for repeated practice. In Phase 1, content reading/research, individuals develop knowledge about domain content. In Phase 2, individual content reflection, they complete an individual sense-making activity to help them think about course content. For example, students could read a chapter and then answer questions that push them to make sense of difficult concepts by synthesizing readings or searching for additional information on the web. In Phase 3, group discussion, students meet in a collaborative online environment to discuss their perspectives on these questions with the aim to develop a shared understanding of the difficult concepts and negotiate what is known in order to develop new knowledge about these concepts.

Regulation without technological support requires individuals to pay attention to collaborative processes as they happen, be aware of ongoing processes, and then identify problems (Recall Fig. 1); a process prone to error. So, we modify this process in Phase 4, individual process reflection. We begin by directing individual attention to team processes in their chat archives, pushing them to evaluate these processes, making them more aware of these processes, and enhancing their ability to identify process problems (see Fig. 4).

Fig. 4

(Top) An activity model depicting how the use of technology (below) could enhance individual knowledge about existing group processes by reordering the gap analysis process

We support individual process-attention by using the affordances of an online text-based environment to provide archived records of discussions and asking individuals to examine and assess them, thereby enhancing individual ability to relive activity and pay attention to important processes after collaboration (Cooke et al. 2000). We enhance the individual’s ability to identify the gap between existing and desired collaborative processes with concrete, research-based reflective assessments that provide a model of desired activity. Each of the six micro-communication patterns is described as a series of less to more desirable patterns of communication (see Fig. 5 for example of one assessment item). Individuals assess collective contributions of the group, not individual contributions. Students are asked to match the collective communication patterns they see in the transcript to those described in the assessment. They reflect on the quality of all six micro-communication patterns from Table 1, thereby enhancing each student’s knowledge about desired collaborative activity and constraining individual reflection to focus on comparing existing states to desired states.

Fig. 5

Screen shot of one micro-assessment item, exploration of different perspectives

To account for problems associated with known inaccuracies of self-assessment (Kruger and Dunning 1999; Dunning et al. 2003) and develop joint understanding, students calibrate their individual reflective assessments through collective sense-making activity. After individuals score their team’s communication patterns, instructions in the computer system prompt them to move on to Phase 5, where they collectively compare, identify their biggest strengths and weaknesses, and select strategies from a guide that they can use to improve future discussion sessions scores. As individuals share their knowledge with the team and work to develop shared understanding of their group processes, regulation moves from the level of the individual to the level of the group. During group regulation, the textual archive enhances joint attention by serving as a shared object of reference for teams to compare and calibrate assessments. Students can use the archives to point out specific patterns for others to examine or provide evidence for problems they have identified in order to develop joint awareness of key processes and joint recognition of a problem state. They can then collectively diagnose problems and complete the final phase, Phase 6, group planning, where they develop a joint plan for how the team can correct or prevent the problem in their next discussion. In between cycles, the group can receive additional feedback on their processes from an instructor to help them further calibrate. The group repeats these activities for every new discussion with the aim of improving collaborative discussion quality.

Important questions arise from the research literature and the proposed design. First and foremost, to what extent does use of our group regulation framework facilitate students’ ability to improve collaborative processes? Second, given the nested nature of collaboration, to what extent does the orchestration and scripting of individual reflective activity impact collective socio-metacognitive sense-making and improvement of discourse over time?

The literature review suggests that enhancing group regulation processes after collaboration should help teams to carry out regulation more effectively, but whether students could improve the quality of collaborative processes remains uncertain, since no studies in CSCL to date have been able to show such improvements over time. The literature also suggests that reflection processes should include a comparison between what happened to what should happen, identifying problems, thinking about why actions occurred, and the implications for future activity. However, whether it would be more helpful for individuals to focus on their attention on specific reflective questions prior to collective discussion remains uncertain. For example, would it help a team to improve more if individuals were prompted to (1) spend added time identifying strategies for future improvement from a guide that presented students with goals for collaboration, common problems, and strategies to prevent/correct them (see top of Fig. 6), or (2) providing evidence from the text archive to support assessments (see bottom of Fig. 6)?

Fig. 6

A depiction of two different conditions for scripting gap analysis: Future Thinking (top), which provides added individual scaffolding for strategy selection and Evidence-Based (bottom), which provides added individual scaffolding for evidence selection

According to Winne and Nesbit (2009), both approaches could potentially enhance the likelihood of group regulation by enhancing individual problem appraisal, but in different ways. Each approach prepares individuals for the collective sense-making activity by having them pay attention to and be aware of slightly different things. Prompting students to select a future strategy pushes individuals to pay attention to strategy guides that include common problems and strategies to prevent or correct those problems. Spending extra time thinking about and selecting improvement strategies enhances depth of individual strategic knowledge for future joint planning. Prompting students to look for evidence from their discussion transcript (the textual archive), pushes individuals to pay more attention to specific communication processes they could use to justify their assessments of group processes. Spending extra time examining the textual archive, enhances depth of individual knowledge surrounding what is occurring during the collaborative discussion. For this reason, the authors wanted to inform system design by examining both the overall utility of their design framework and subtle differences in how they scripted reflection. Thus, our research questions were as follows:

• (RQ1) Does technological support informed by our framework succeed in helping teams regulate activity to improve collaboration?

  • RQ1.1 Do teams engage in productive socio-metacognitive activity?

  • RQ2.1 To what extent do teams improve discussion quality around science concepts overtime?

• (RQ2) To what extent do different individual reflective scripts impact joint sense-making about collaborative processes and group regulation outcomes?

  • RQ2.1 Does the use of the Evidence-Based and Future Thinking individual, reflective scripts impact frequency of productive socio-metacognitive talk during group reflection and planning.

  • RQ2.2 To what extent does the use of the Evidence-Based and Future Thinking individual, reflective scripts impact improvement of group content-based discourse over time?

To answer these questions, we examined 13 online groups over five discussion sessions, over a 10-week period of course activity. We assessed collaborative discourse quality, measured change over time, and examined patterns of communication that occurred during socio-metacognitive sense-making activity.

Methods

Participants and course context

Thirty-seven online students in a class on information sciences and technology formed the participants of the study, each belonging to one of thirteen groups. The majority of students were part-time students with full-time jobs. Eleven students (30.5%) were female and 25 students (69.4%) were male. The female to male ratio was fairly representative of the enrollment of information sciences and technology courses at the college. The groups were formed based on consideration of availability for online group meetings, gender, expertise in information sciences and technology. Groups were assigned to different reflective conditions, Condition 1 (Future Thinking) and Condition 2 (Evidence-Based), such that the groups in each condition were comparable. Seventy-one percent of participants in the Future Thinking (FT) condition were in the 25–44 age range; 75% in the Evidence-Based (EB) were in the 25–44 age range. With regard to group composition, there were five teams of three and one team of two in the FT condition; six teams of three and one team of two in the EB condition. In the FT condition, 66.7% of the teams were majority male compared to 71.4% in the EB condition. Each condition had two all-male teams: a team of two and a team of three. Neither group had all female teams. Of those that reported work hours, 91.6% reported working full time in the FT condition; 90% reported working fulltime in the EB condition.

The study took place in a 16-week university level introductory online course on information sciences and technology. The main aim of the course was to introduce students to concepts and research areas central to information sciences, i.e., security and risk analysis, human computer interaction, emerging technologies, effects of technology on society, and informatics. The course was organized in a learning management system (LMS) with weekly lessons, student resources, course communication, and course materials all housed in the LMS. The course instructor was expected to organize and maintain the course, revise instructional materials as needed, grade student work, answer student questions, and help students to think more deeply about course content. As part of the course, students had to learn to work as part of effective teams and had to complete a team project. Collaboration and collaborative skills were often points of conversation brought up during whole class discussions related to important design thinking and business skills.

Procedures

Students were required to follow our framework as part of their required course activities. They read a chapter from the information science text or supplementary materials each week. Students were assigned to teams in weeks three through five. In weeks six, eight, ten, twelve, and fifteen, students were required to meet in a synchronous online environment to complete the sense-making activities in the framework. These activities counted towards 25% of students’ course grades. In session one, teams received full credit for the discussion regardless of the discussion quality. After the first session, students were given initial assessments and told that the subsequent discussions would be graded based on discussion quality.

The pre-discussion reading activity included five questions that each person had to answer on their own: (1) what were the main learning goals of the chapter, (2) what were the most difficult concepts or parts of the reading, (3) what did you find most interesting, (4) what four questions could you ask yourself, the instructor, or others regarding this chapter, and (5) were you able to fully meet the learning goals for this chapter. Individuals had to respond to these same questions and submit their responses before each discussion session.

Each online collaborative activity consisted of three parts. In part one, the team had 60 min to discuss questions and issues raised by the reading activity through text-based chat. After 60 min, the team was instructed to stop chatting and move on to part two of the activity: the highly scripted individual reflective assessment where individuals evaluated the quality of six micro-communication patterns (15 min). The instructor informed students that an expert rater would also assess these patterns and that they would determine the accuracy of their scores based on the difference between their scores and the expert score. The instructions stated, “it is more important to be accurate than it is to say your team did well. It will not help your team at all to give yourselves unrealistically high scores. It is better to be critical, as this will help your team improve.” After individuals finished the reflection, the team moved on to part three, group reflection and joint planning. Teams were instructed to “discuss how each of you assessed your team, identify your strengths and weaknesses, and some strategies from the information synthesis or knowledge negotiation guide that you can use to improve your next chat session.” Teams were required to export their chat files after completing the discussion sessions. The exported files had to be submitted to a drop box folder in the LMS.

Condition manipulation

As stated above, individuals assessed six micro-communication patterns as part of the scripted reflection activity. However, we modified what individuals were asked to pay attention to after scoring, in order to help us determine best practices for scripting. So, for each micro-pattern, individuals had to score the quality of their existing communication, but then follow one of two additional scripts. Individuals in the Future Thinking (FT) condition, were required to score each micro-communication pattern and then “provide a strategy from the guides to improve on this item for your next session”. Individuals in the Evidence-Based (EB) condition also had to score each micro-communication pattern, but then “Provide evidence from the discussion session to support your score”. Thus, individuals in FT condition groups were pushed to pay attention to the problems and strategies guide so as to build their knowledge of socio-metacognitive strategies, while individuals in EB condition groups were pushed to pay more attention to their existing communication processes from the discussion transcripts. The difference between conditions was based on the individual reflective scripting that occurred prior to the group discussion as a means to support individual attention and enhance individual awareness.

Collaborative discussion quality assessment

We conceptualized collaborative discussion quality as the extent to which teams were able to demonstrate the ability to meet desired goals for each of the six micro-communication patterns identified in the literature review: verbal equity, joint idea-building, developing joint understanding, exploration of alternative perspectives, quality of claims, and constructive discourse. Discussion quality is measured at the level of the group by looking at the entire chat session for specific interactional communication patterns. In order to accurately measure changes in communication patterns, we operationalized the six micro-communication patterns into items with a range of more to less desirable communication patterns similar to the rubrics individuals used to assess their teams (see Table 2). Drawing on examples of reflective-assessments (White & Frederiksen, 1998), we  specified levels of sophistication from one-to-five, with different descriptions of communication patterns for each level. Each score required specific evidence from the transcript to justify the communication pattern (see Table 3 for an example of one rubric item, quality of claims). A research assistant with two years of communication analysis training served as the expert rater and evaluated each team’s discussion transcripts at the five session points. The expert used the entire discussion session to determine the score for each pattern of communication by examining the transcript for evidence of the desired activity. Twenty percent of the total data was double coded. There was a high level of agreement with regard to the correlation of scores, r = .86; p < .001, as well as substantial categorical agreement: Kappa = .64; p < .001.

Table 2 Pragmatic model of collaborative discourse competence

Table 3 Example of one of the six micro-assessment items from the discourse quality assessment

The rated items were averaged to produce a single Collaborative Discussion Quality score, a continuous value between 0 and 5 used to track improvement over time in collaborative discussion processes. Team process measures at the first session point were used to identify groups’ initial strengths and weaknesses prior to the intervention. Given that the expert rater was aware of when in the course students were engaged in discussion, as they provided students with ongoing feedback, we had a second expert, a graduate student with 1.5 years of communication analysis training, to double check scores and evidence from transcripts at sessions, 1, 2, and 5 to ensure that sufficient evidence was present to justify scores.

Selection of microanalysis team

We wanted to develop a deeper understanding of what quantitative changes, identified by the rubric, actually looked like at the level of discourse. For this reason, we conducted a microanalysis of changing patterns for one team. We selected the team based on most improvement of discourse quality, according to the assessment, between sessions 1 and 5. There were two teams that tied for most improvement, with nine points of improvement, out of the 30 points possible, from Session 1 to Session 5. Both teams were in the evidence-oriented condition; one team had two people and one had three. Since the majority of our teams were three-person teams, the three-person team, Team 4, was selected.

Analysis of group socio-metacognitive talk

Socio-metacognitive talk is assessed at the level of the individual post, as they externalize their thinking through discussion with the group. Thus, frequency of team socio-metacognitive talk is conceptualized as the total socio-metacognitive group talk provided by all team members during group reflection. Data from part three of the group discussion, where individuals share their understanding of their existing group processes with the group was examined in order to determine whether socio-metacognitive talk occurred and what types were frequent. We coded these joint sense-making activities for each team across Sessions 1 through 4; Session 5 did not include a reflection. Each chat turn/post was coded based on the type of talk act. Talk acts were classified as “productive process-centered” or “other” talk. Productive process-centered talk focused on using the reflective assessments to make sense of existing or future collaborative activity, whereas “other” talk did not. Productive process-centered moves included reporting process scores, process monitoring, process reflecting, process planning, and process revising (see Table 4 for coding construct). Two coders, trained in micro-coding of speech acts, coded 23% of the total data, 455 chat turns. Inter-rater reliability was Kappa = .806; (p < .001), indicating almost perfect agreement (Landis and Koch 1977). Most of the disagreements centered on distinguishing between reflecting and reporting, and reflecting and planning. Disagreements were based on the tendency for participants to reflect on part one of the discussion session as a means of providing evidence for scores or selected strategies. We discussed these differences, resolved disagreements, and coded all the data. In total, 1959 chat turns were coded as part of this analysis.

Table 4 Categorizing socio-metacognitive sense-making activity: definitions and examples

A second round of coding was conducted for reflective acts where groups attempted to identify what led to their specific communication patterns. All reflective acts were coded for what groups paid attention to as they discussed their perspectives of what led to the group’s patterns of interaction: Chat-based communication patterns that occurred during the discussion or external factors. A reflective act was coded as a Chat-based communication patterns if the turn included evidence that the group attributed successes or failure to concrete communication behaviors, characteristics, effort, or specific strategies occurring during the chat. Reflective acts were coded as referring to external factors if the post included evidence that the group attributed process successes or failure to external factors and did not pay attention to specific process behaviors from the chat session. External factors included type of reading content, rules of the activity, the chat environment, computer problems, absent members, etc. The first author coded all of the reflective acts for attention orientation, but to check the reliability, two other raters independently coded 30% of all reflective acts using the same rubric and achieved 89.3% agreement.

Findings

(RQ1) does technological support informed by our framework succeed in helping students regulate activity to improve collaboration?

Baseline scores

As expected, teams initially displayed common collaborative sense-making problems. Teams in this population were better at collective information synthesis than they were at collective knowledge negotiation. At session 1, verbal equity was the lowest scoring area (M = 2.46, SD = 1.38) and idea building was the highest scoring area (M = 4.15, SD = 0.58). With the exception of verbal equity, participants’ initial baseline scores for collective knowledge negotiation were lower than those for collective information synthesis; M = 9.69, SD = 1.5 for collective knowledge negotiation and M = 10.31, SD = 2.18 for collective information synthesis, where the maximum score for each area was 15. Collective knowledge negotiation included exploration of different perspectives, quality of claims, and norms of evaluation. The majority of our teams (9/13) either did not provide any alternative perspectives for presented claims, or presented an alternative that was immediately agreed upon or ignored without discussion. Six teams displayed patterns of logical, opinion-based rationale, where all arguments were supported by anecdotal evidence with no reference to course readings or other online resources. Only two out of the thirteen teams took time to validate the ideas of others before criticizing them or pointing out flaws in logic. After session 1, teams began the process of guided reflection in the system following each session.

Frequency of productive collective socio-metacognitive sense-making talk

Our reflective assessments were intended to model desired collaborative practices and enhance collective socio-metacognitive sense-making activity, talk aimed at understanding and improving collaborative processes. Thus, it is necessary to examine the extent and type of socio-metacognitive talk that occurred during collective reflection and planning to ensure that students were carrying out important, collective process-based sense-making activity.

Three hundred and eleven chat turns (40% of all chat turns that occurred during part three of the activity) were coded as productive process-based talk, where students discussed and made sense of their collaborative discussion activity. Of all productive process-based talk, 61.4% went beyond simple reporting of scores and included socio-metacognitive sense-making activity: planning, monitoring, reflecting or revising acts. Monitoring acts accounted for 18.4% of all socio-metacognitive sense-making activity and included posts similar to the following: “[For exploring alternative perspectives] I went with a score of 3 because when we finally did critique [each other] we really did not elaborate on the disagreement.” Reflecting acts accounted for 12.2% of socio-metacognitive sense-making activity and included such statements as, “I'm pretty sure if we all had a stronger grasp of what the article was trying to say we would have been able to challenge [each other] a bit more”. Planning acts were the most frequent form of socio-metacognitive activity and accounted for 28.1% of all socio-metacognitive talk. They included larger episodes where groups worked to figure out how they could regulate activity to improve discussion quality:

Bill	How are we going to get better though?
Marice	We need to be more vocal about what we're discussing Instead of just, I ask a question and you answer a question.
Julie	I agree, Marice.  The goals and strategies guide suggests relating each idea to something you know.  i.e. - “That reminds me of … .”
Marice	Exactly, and then tie it to the facts.

Revising acts were the least frequent, accounting for only 1% of socio-metacognitive activity. These acts target discussion related to the way the team altered their existing plans based on new information. Overall, we found that all teams engaged in productive socio-metacognitive talk. They shared their perspectives on their collaborative processes, worked to identify problems with their communication processes, and made plans to try to improve upon their weaknesses so as to improve the quality of their discussions as defined by the assessment model.

Improvement of collaborative discourse quality of course content over time

An analysis of average baseline scores at session 1 compared to session 5 shows how patterns of communication improved from the initial to the final session (see Table 5). Ten weeks after their initial session, at session 5, the majority of teams were no longer displaying low collective knowledge negotiation communication patterns. Though the lowest and highest scoring areas remained verbal equity and idea building respectively, all average scores increased. There was an increase in quality of claims, with no dysfunctional patterns present and 12 teams showing evidence of at least one instance of claims supported with logical, Evidence-Based rationale that referred to course content from the assigned readings or another information source.

Table 5 Descriptive statistics for quality of collaborative discussions at sessions 1 and 5

We ran a repeated measures ANOVA, and found a significant difference in Collaborative Discussion Quality score, F (4, 48) = 10.94, p < .001. As p < .001. A Post hoc test using the Bonferroni correction revealed that scores showed significant increases between session 1 and session 3 (mean difference = .60, p < .001), between session 1 and session 5 (mean difference = .96, p < .001), between session 2 and session 5 (mean difference = .73, p < .001). Thus, collaborative discussion quality improved significantly across three sessions, but not between consecutive sessions as teams worked to improve communication processes. This suggests that our general approach towards supporting group process awareness and regulation facilitated the improvement of collaborative discussion quality over time. As scores for each communication process increased for collective information synthesis and knowledge negotiation, so did each team’s total discussion quality score (see Fig. 7).

Fig. 7

Discussion quality performance in Time Point 1 vs. Time Point 5

A case study of changes in communication patterns

We conducted a case study of a team that improved substantially over time to get a deeper understanding of what the changes in collaborative activity looked like at the level of discourse. In Session 1, Team 4 had some relatively unequal contributions, but most of their low scores were the result of problems with collective knowledge negotiation. Similar to the majority of our teams, their two lowest scoring areas were alternative perspectives and quality of claims.

Figure 8 shows a visual representation of talk moves associated with assessment criteria in sessions 1 and 5. There were three topics of discussion in Session 1, each made up of a series of communication acts (see top of Fig. 8). Their communication pattern for idea building was less than optimal, originally scoring an average score (3/5) on the assessment. One topical episode consisted of one person sharing their perspective about what was most difficult to understand from the reading, followed by another person quickly changing the subject to what they experienced as difficult (see top of Fig. 8). Simple agreement and off-topic acts are not depicted in the representation. Other topical episodes included a slightly more sophisticated pattern, where a topic was proposed, a member responded with input, and these responses were extended with elaboration or requests over two to three talk turns. Thus, the number of turns related to each topic were relatively small, averaging nine speaker-turns. With regard to collective knowledge negotiation, there were no instances where a claim was followed by an alternative idea or weighing of evidence. Furthermore, students made claims without justifying them with evidence from the course textbook or other online resources. Here is an excerpt from one of these episodes. Though there are five posts, they count as two talk turns because we followed conventional definition of turns of speech, where one turn ends when another member speaks (turns are numbered for ease of referencing):

Turn	Speaker	Utterance
1	Tom	My questions were more of the technical nature regarding the future of databases... Specifically, what will happen to large centralized databases in the future if we’re moving towards more client-centric databases with natural language? Will multimedia records replace large government databases?
	Tom	All of our records are in databases now, but as the technology grows and develops, how much will the government spend to upgrade?
	Tom	I can tell you from being on one side of it, it’s a huge cost, and I’m curious to see how soon the gov’t would progress with the technology.
2	Juan	That will be interesting. It will be difficult to predict as technology changes so quickly. Less than 20 years ago we lived in a world of static web pages with no need to access databases in real time. Now, most pages that you visit are custom made on-the-fly based on information retrieved from a database.
	Juan	Wei-yu, do you have any other questions regarding databases?

Fig. 8

A comparison of patterns of communication in Sessions 1 (top) and 5 (bottom)

In this example, Tom proposes that technology use places a huge cost on the government (Turn 1). Though there are assumptions inherent in this claim, no one addresses them. For example, the team does not weigh the cost of technology upgrades against maintaining paper-based records, or losing records. No one questions the basic premise of his claim, that society will move to language-based databases, or questions the extent to which multimedia records make sense. One student, Juan, does add to the comment by elaborating on the interesting nature of the idea before closing the topic (Turn 2).

In Session 5, the group’s discussion also included three topical episodes, but the length of episodes increased to an average of 24.3 speaker turns. The team was not perfectly equitable in turn-taking in session 5 (33%, 40%, and 27%), but there was less variability between speakers than in Session 1 (42%, 42%, and 16%).

In Session 5, idea building from one topical episode also leads to a related idea or a completely new idea, as students push each other to go back and reevaluate the question. For example, in the bottom of Fig. 8, Tom proposes Topic 1: information sharing on the web and whether people will continue sharing information. Wei-yu proposes Idea 1, that it depends on whether people mind exposing their lives in detail or if they prefer privacy. Juan extends Wei-yu’s idea by stating that there will always be those who wish to remain private, but suggests technology may not allow them to do so. This was expanded upon with examples, elaboration, and humor from multiple members, until Wei-yu brings the discussion back to the central topic and refines the question to be whether continual sharing is problematic and proposes a second idea.

Besides including more complex idea-building moves, the team’s discussion in Session 5 also includes more sophisticated forms of evidence and weighing of ideas. The following is an example from Topic 2, bottom of Fig. 8: how far is too far with technology (a discussion that took place prior to the public development of the technology in question). Idea 4 consists of Juan proposing that he is concerned about the security implications of technology dependence, for example, the idea of a self-driving car being hacked. Juan claims that self-driving cars are likely, to become mainstream, but Tom questions this possibility. In response to Tom’s questioning, Wei-yu and Juan provide additional information about self-driving cars:

Turn	Speaker	Utterance
1	Wei-yu	you can start them with your phone and even lock them I think.
2	Juan	http://www.extremetech.com/extreme/181508-googles-self-driving-car-passes-700000-accident-free-miles-can-now-avoid-cyclists-stop-for-trains
	Juan	700,000 miles without accidents.
3	Tom	But what about people that can’t afford that technology?
	Tom	We’re still at their whim to not cause accidents.
4	Juan	Just like everything else, it will eventually come down in price and become mainstream.
	Juan	I would not be surprised if congress starts requiring all cars to be self-driving at some point in the future, citing safety.
5	Tom	Wow, that’s crazy.
6	Wei-yu	That’s what I was thinking.
	Wei-yu	Once it becomes required people will either get it or find another way to get around public transportation.
7	Juan	I can see many positives of it.
	Juan	More safety, better use of road capacity, vehicle sharing, etc...
	Juan	But there are also many negatives, like security, privacy, loss of jobs in the transportation sector, etc...
	Juan	Imagine a hacker corrupting the map database, causing vehicles on a specific road to turn into a cliff.
8	Tom	Well, if that’s the direction we’re headed, hopefully the software will be developed to avoid that.

Wei-yu supports Juan’s idea by elaborating on features of a self-driving car (Turn 1) and Juan adds evidence to support his claim (Turn 2). The team elaborates on the implications of Juan’s idea (Turns 3–4, 6). Juan also weighs the potential costs and benefits of such a technology, including the potential security risks (Turn 7). Tom responds to the idea of this new risk by referring to a concept from the course, the co-evolution of human activity and technology (Turn 8).

In examining communication patterns in Sessions 1 and 5, we can see distinctive changes from less to more sophisticated discourse. Even though the team spent the same amount of time on the discussion in both sessions, in Session 5, students ideas in more depth and longer, more diverse, and more cohesive communication acts.

(RQ2) To what extent do different individual reflective scripts impact joint sense-making about collaborative processes and group regulation outcomes?

Effects of reflective condition on socio-metacognitive talk

Given that the different conditions pushed individuals to focus their attention on examining the transcript or examining strategies to improve future discussions, one could expect that the Future Thinking condition might talk more about planning while the Evidence team might spend more time on talking about instances in the transcript that explain existing problems or talking about why problems occurred. However, when we examined how groups made sense of their collaborative processes, we saw no significant differences in the frequency of different types of socio-metacognitive talk or frequency of overall socio-metacognitive talk. There were differences between groups with regards to how they engaged in reflection talk, talk centered around determining why problems occurred.

During reflection talk, six out of seven groups in the Evidence-Based condition paid attention to specific interactions from the content-based discussion when diagnosing communication problems:

“I gave a 2 for the next goal [contributing alternative ideas]. We did a lot of agreeing, and we used a lot of "I think" or "I feel" statements, most of which were opinions. We referenced the textbook itself a few times, but I think we could have benefited from more fact-based evidence. This is especially true when we were talking about privacy toward the end of the discussion”.

In contrast, when trying to diagnose problems, four out of six of the groups in the Future Thinking condition primarily focused on discussing generic, external factors not connected to the specific behaviors housed in the chat transcript: “it’s hard for multiple people reading from the same book to have diverse ideas”, or “I think the only thing keeping us from a perfect score is again, it is hard to have rich argumentation when there are only 2 people.” As such, teams in the Evidence-Based condition had more targeted reflective analysis, identifying and evaluating specific micro-communication patterns from the discussion session.

Effects of reflective condition on improvement of discourse quality

We used a 2 (Condition) × 5 (Time) mixed factorial design to examine the effect of individual reflective scripting condition on communication patterns over the five sessions. In this model, Time represents when in time the sessions took place. When accounting for different baseline scores and conditions (i.e., team nested within condition, and time nested within condition), we found that teams in the Evidence-Based condition had significantly higher scores on average, M = 11.87, SD = 2.11, than teams in the Future Thinking condition (M = 11.07, SD = 2.19); F(1, 110) = 5.46, p < .05. Time also had partial correlation on Discussion Quality in this model: r = .45, p < .005. However, there was no significant interaction with session time and type of individual reflective condition, meaning that there were no differences in how much the conditions improved over time.

Discussion

Given how important collaborative activity is becoming for our society and the fact that many are unable to collaborate well, we wanted to develop a method to help students regulate collaborative discourse processes so as to improve them. To our knowledge, no study has shown that technological support for group regulation can help teams to improve their course-based, collaborative discourse over time. Supporting group regulation of collaborative activity in a theoretically informed way is a challenge because it requires a substantial amount of knowledge about what types of activities are desired for healthy collaborative functioning and what types of individual and group support students would need to regulate collaborative activity so as to inform design.

Building on theory, we proposed a framework for computer-supported group regulation that includes a method for analyzing group discussions, concrete ways to support groups in analyzing their own discussions with an eye to improvement, and different possibilities for scripting individual reflection. Our findings clearly show that our approach can help students to systematically improve the quality of collaborative discourse. Our findings also show that our approach helps groups to view their collaborative processes as objects of thought and work to improve them through joint socio-metacognitive sense-making and regulation. These findings are especially promising given that research on group sense-making and collaboration repeatedly show that students rarely display regulatory behavior and teams do not naturally improve on these types of communication processes over time (Kozlowski and Ilgen 2006).

We also tested whether individual reflective practices impacted the quality of collective sense-making about collaborative processes and the team’s ability to improve over time. We tested two alternative individual reflective scripts, Future-Thinking and Evidence-Based. Asking students to provide evidence to support their assessments of process quality (Evidence-Based condition) was associated with higher quality discourse at each session and more targeted reflective analysis than asking students to provide strategies they could use to improve future discussions (Future Thinking condition). This is important because it suggests that computer-support should include prompts that push students to use the archives to support their evaluation process.

One explanation for the difference between the two reflective assessment conditions comes from an understanding of problems with individual and group attention. Attention is an important factor that can help or hinder regulation. What an individual focuses attention on can affect individual error detection and interpretation of a problem (Hofmann et al. 2012; Rueda et al. 2005), but can also impact a team. In order for a group to be aware of a process, at least one individual must pay attention to it and bring it to the collective attention of the group when neccessary. It is possible that requiring individuals to provide evidence for their reflective assessments focuses their attention on concrete communication acts, enhancing the quality of their reflective analysis by recognizing how specific patterns of communication impact the quality of group processes. Thus, this type of reflective prompting may enhance the depth of knowledge of existing collaborative processes. Depth of individual knowledge may help the group select strategies that are specifically suited to the team’s patterns of communication. However, more research is needed to determine the relationship between attention, problem detection, problem interpretation, and regulation within a nested system and implications for the design of computer-supported group regulation.

Given the that our approach can help students to think about and improve their collaborative processes we believe this framework would be useful as a means to guide the design of technologies aiming to help students learn how to manage and improve collaborative processes. Though the instructional methods we used, i.e., reflection, self-assessment, are not new, the way that we used them are novel. We developed a theoretically supported way for students and researchers to evaluate the quality of collaborative sense-making discourse and a framework for using these assessments and technological support to help students improve their discussions over time. Our study also sheds light on the need to broaden our use of scripting beyond support of collaborative processes in action, because supporting how students individually and collectively make-sense of their collaborative processes after they occur may be a powerful way help students learn how to manage these processes for themselves.

Limitations and future research

The purpose for this study was to inform the design of a computer-supported group regulation system that could enhance the quality of reflective analysis and help teams develop socio-metacognitive expertise. Our findings are an important step in this direction. Nonetheless, it is important to note that research on the development of socio-metacognition is still in its infancy and more collective work needs to be done in this area. Given how little is known about the regulation of group cognition, we chose to prioritize a deep analysis of a small population over a semester, rather than a broader analysis of a larger population over a short period of time. However, both types of studies will be necessary to better understand how socio-metacognitive expertise develops and how it affects different aspects of group cognition. This is especially true with regard to our findings on the effects of reflective conditions on socio-metacognitive sense-making activity. Given the small sample size, it is uncertain to what extent the findings can generalize to a larger population. Moreover, we did not analyze the written individual responses that occurred during this part of the activity. Thus, more research is needed on the effects of reflective scripting on socio-metacognitive development and collaborative process improvement. Of specific interest is the extent to which targeting attention to specific conversation processes impacts socio-metacognitive sense-making activity in general.

Another limitation of the current work is that our scope of the problem may be too narrow. For example, this study largely ignored the impact of emotion on learning and reflection, but there is growing evidence that cognition and emotion are largely intertwined (Meyer and Turner 2006; Tully and Bolshakov 2010). Emotion may play a key role in activating learning centers in the brain, influencing attention and sophistication of thinking (Hu et al. 2007; Petty and Briñol 2015). These studies suggest that emotion may serve as the initial gateway, before attention, and can fundamentally shape what we pay attention to, as well as our awareness and interpretation of events and therefore what is remembered and learned (Immordino-Yang 2015; Norman 2004; Ortony et al. 2004). Therefore, more research needs to be done on the role that emotion plays throughout the process of regulation of collaborative discourse.

There is also little agreement in the field of CSCL as to what high quality collaboration looks like concretely or how to measure it (Gress et al. 2010; Jeong et al. 2014; Ong and Borge 2016). Yet, if students need to be able to regulate their collaborative activity, research suggests they need models of competence they can use to productively analyze their own activity (Nesbit 2012; Weick et al. 2005; Winne and Nesbit 2009). Even imperfect models and summative feedback assessments have the potential to help students understand and regulate their collaborative activity (Borge and White 2016).

Given the need to carefully examine what productive collaborative processes look like in practice and the extent to which different patterns may predict the quality of collective thinking processes and decision-making outcomes, there is a need to develop an understanding of this complex phenomenon through small field-based qualitative case studies and larger laboratory-based empirical studies. In this way, important markers of collective thinking processes can be identified, possible reasons for their importance can be uncovered, and relationships between key variables can be examined in controlled and real-world settings. Moving back and forth between classroom-based and laboratory-based studies will allow researchers to leverage both realism and precision of measurement when making sense of this phenomenon.