Introduction

What we know about Computer-supported collaborative learning (CSCL) suggests desiderata for a design methodology. CSCL has been widely adopted in the fields of higher education and K-12 education. CSCL is grounded in the sociocultural theory proposed by Vygotsky (1978) who believed that learning first takes place intersubjectively before it takes place intrasubjectively. Collaboration is conceptualized as a process of constructing a shared meaning (Stahl et al. 2006). Collaborative learning is also a dynamic interaction process in which group members co-construct knowledge to achieve a common goal (Dillenbourg et al. 1996). Collaborative learning is regarded as a source of cognitive development and a basis of human learning (Stahl and Hakkarainen 2019). An ideal design methodology for the field would embody these values and support refinement processes that enhance the underlying processes we know are valuable for learning.

Previous studies have indicated that effective collaborative learning does not occur automatically (Wang and Mu 2017) and collaborative learning needs to be carefully designed (Dillenbourg et al. 2009). Furthermore, Rienties et al. (2017) have demonstrated that learning design is a critical component in developing instructional activities that have positive impact on student learning. Moreover, a collaborative learning design is a crucially important foundation for establishing learning objectives and pedagogical plans (Carvalho et al. 2019). This highlights the need for design methodologies to support effective intervention design, in other words, design of interventions that effectively manage group processes for the purpose of improving group outcomes. Consistent with this foundational literature and associated insight, in this paper we propose a design approach that affirms the central importance of key processes in consideration of collaborative learning objectives, tasks, interactive approaches, learning resources, interventions, and assessment methods. These must be carefully planned and considered in light of valued processes from the beginning in order to improve learning performance.

In the field of CSCL, published studies typically propose particular interventions such as scripts or teachers’ guidance to facilitate collaborative learning (Gerard et al. 2019; Heimbuch et al. 2018; Ingulfsen et al. 2018). Though much work on evaluation of interventions in the field of CSCL is quantitative and theory driven, the most popular approach to iterative intervention design within the field of learning sciences is to adopt the design-based research (DBR) approach, and specifically to use it to examine and possibly improve on the effectiveness of particular interventions in the process. For example, Zheng et al. (2015) adopted a four iteration DBR approach to enhance wiki-supported collaborative learning activities by developing nine instructional strategies. Leinonen et al. (2016) used a DBR method to support K-12 teachers when reflecting on group work, and found that use of mobile device apps can achieve a higher level of reflection. In addition, the DBR approach was also used to create meaningful online discussions for undergraduate students to enhance their deep learning (Johnson et al. 2017).

There are, however, many challenges with DBR. First, it is very difficult to validate the effectiveness of proposed interventions since the effectiveness is difficult to replicate (Zheng and Yang 2014) and there is a dearth of clear prescriptive accounts of DBR processes (Easterday et al. 2014). Second, DBR requires a sustained commitment of researchers to support an intervention through multiple iterations, leading to a potentially large degree of subjectivity and bias in the results due to differences in persistence across endeavors (Anderson and Shattuck 2012). Third, though iteration is desirable, Zheng (2015a) has found that most studies that identify as having adopted DBR have in fact only carried out one deployment and evaluation cycle without providing the details about how they followed up (if at all) to revise the interventions. Finally, it is very difficult to generalize and replicate DBR since the specific characteristics of the learning context and learners are an integral consideration as the approach is applied, and these constantly change (Barab and Squire 2004).

The need for an innovative approach

To overcome the above stated challenges, a new approach is needed for engaging in iterative design of interventions. In this paper, we propose such an innovative approach, which we refer to as design-centered research (DCR), which focuses on how to design interventions and evaluate the instructional alignment between the design and its enactment on each iteration, and then to extract insights for improvement (Yang 2013). Instructional alignment specifically refers to the consistency between the expectations that motivated the instructional design and the findings from its enactment (Yang 2013).

As an illustration of this new method, the study described in this paper has adopted the DCR approach to evaluate the alignment between one specific large scale collaborative learning design effort and its enactment. The concept of alignment specifically applied to CSCL is defined as the consistency between design plans and enactment during a collaborative learning activity, with a focus supporting effective group processes. Thus, alignment in CSCL focuses on whether or not the design elements have been enacted by students in their manner of working together, specifically in terms of whether and how their enactment deviates from what was intended in the original design. Thus, the focus is the interplay between design elements and these processes. It is that interplay that this approach seeks to make visible and then use to optimize the efficacy of the design to support collaboration. The evaluation of the alignment can also examine the extent to which the design plan achieved its desired end product and to what extent learners achieved the expected learning objectives. The mission of research on alignment in the CSCL context is to provide guidelines for efficient data-driven improvement of collaborative learning experiences through reflection on deficiencies discovered through a quantification offered by the proposed approach.

Research purposes and questions

In the context of CSCL, as in other areas of education, the fidelity of the implementation of a learning design in the classroom might be compromised for many reasons. For example, the pace at which teachers have carried out a design has frequently not matched the original plan, and as a result, learners have frequently not met the desired learning objectives because of portions of the activities they did not have a chance to benefit from. This underscores the importance to the goal of improving the quality of collaborative learning of evaluating whether or not the design elements have been implemented and whether learners have achieved the learning objectives. However, to the best of our knowledge, there is still a dearth of empirical studies evaluating the alignment between the design and its enactment in detail in the CSCL context. This study aims to propose an innovative method of evaluating the alignment between the design and its enactment specifically tuned to the needs of the CSCL context in order to facilitate the implementation of more published studies of this type going forward. In particular, we address to what extent teachers’ design decisions were implemented during collaborative learning. We propose the following three research questions based on the goals of the study:

  1. (1)

    How can one evaluate the alignment between a collaborative learning design and its enactment in computer-supported collaborative learning?

  2. (2)

    To what extent is the optimization process facilitated through DCR of a collaborative learning design able to enhance the alignment between the collaborative learning design and its enactment in a future iteration?

  3. (3)

    To what extent does improving the alignment between a collaborative learning design and its enactment increase group performance?

Literature review

Design-based research in CSCL

As a research approach, DBR emerged at the beginning of the twenty-first century and has expanded its influence in recent years (Ludvigsen 2016). It has been used across prominent lines of research within the field, and has produced noteworthy designs that have been valuable to the field of CSCL. In particular, DBR has been widely used to examine the effectiveness of interventions through several iterations. For example, Lin and Reigeluth (2016) adopted DBR to design, implement, and refine instructional methods used in wiki-supported collaborative learning. They found that the wiki instructional methods promoted collaborative learning in an undergraduate design class. Chen et al. (2015) employed DBR to judge promising ideas in a knowledge-building discourse with two cycles of promising-idea identification and discussion among Grade 3 students. DBR was also used to develop and improve a scripted learning environment to enhance collaboration in project-based learning (Alharbi et al. 2018). It was found that teachers preferred the enhanced environment for the purpose of improving collaboration. In addition, researchers have also employed iterative DBR to identify problems in CSCL environments and use findings from analysis of group processes to recommend innovative functionality to address problems (Stahl 2017). Tissenbaum and Slotta (2019) adopted DBR to support classroom orchestration through real-time agent-based support and teacher tablets in a collaborative inquiry design activity. Altebarmakian and Alterman (2019) investigated group cohesion in a one-semester course using a DBR approach and found that the level of engagement determined the scope and quality of group cohesion. Rodríguez-Triana et al. (2015) used DBR with three iterations over three years to evaluate the alignment between scripting and monitoring in two CSCL scenarios. They found that the alignment of the pedagogy and monitoring enables teachers to improve the design and management of collaborative learning.

In the field of CSCL, many studies have proposed scaffolding or scripting to support collaborative learning using different approaches. For example, Martinez-Maldonado (2019) adopted real-time collaborative learning analytics through a handheld dashboard to support the orchestration of a CSCL classroom. They analyzed teachers’ perspectives on the use of a real-time learning analytics tool based on a qualitative analytical approach. Näykki et al. (2017) developed a macro regulation script to promote cognitive and emotional monitoring during collaborative learning. They found that active script discussions stimulated more monitoring activities based on a qualitative and quantitative analytical approach. Wake et al. (2018) adopted a DBR approach to engage students in collaborative design, development, and evaluation of location-based games.

Design centered research

Design centered research (DCR) was proposed by Yang (2013) as an alternative to DBR with the argument that it is advantageous with respect to efficacy in evaluation of the alignment between an instructional design and its enactment. While DBR has widespread adoption in the field, DCR has been rarely applied. In particular, only Yang and Liu (2018) adopted a DCR approach to conduct a case study to design and refine scaffolding to support collaborative learning. Therefore, there is a dearth of empirical studies on evaluating the alignment to support collaborative learning using the DCR approach. The major differences between DBR and DCR are as follows:

First, the epistemic stances of DBR and DCR are different. The epistemic stance of DBR has pragmatic philosophical underpinnings (Barab and Squire 2004). In contrast with DBR, DCR is grounded in the theory of falsifiability proposed by Popper (1963) who believed that scientific theories should be at least falsifiable and replicable. A related second point is that the findings of DBR are not meant to be replicated and generalized because the philosophical foundation for DBR is strongly qualitative and thus grounded within the local context (Design-Based Research Collective 2003). However, DCR is a quantitative methodology, and thus it is meant to produce generalizable knowledge, and thus the technological insights about design are meant to be replicated in other contexts.

Third, the primary aim of DBR and DCR is different. DBR focuses on the effectiveness of a particular educational intervention embedded within a specific context, while DCR focuses on how to design a particular educational intervention with the aim to apply it across contexts through working towards alignment between the design and its enactment (Zheng and Yang 2014). From a slightly different angle, a strongly related fourth point is that DBR aims to evaluate a particular educational intervention through several iterations in a particular educational context (Design-Based Research Collective 2003). On the other hand, DCR aims to develop technological knowledge about design through designing educational interventions, evaluating the alignment, and analyzing the deficiencies of the design. Therefore, the purpose of DBR is to obtain an effective intervention while DCR aims to generate a stable and replicable design process and develop generalizable knowledge about how to design better interventions.

Fifth and finally, DBR involves researchers, practitioners such as teachers, administrators, and other stakeholders. However, the involvement of researchers, administrators, and other stakeholders often complicates the research context (Barab and Squire 2004). While this approach has many merits, in contrast to DBR, DCR has more of a pure focus in that it only involves practitioners such as teachers themselves and thus avoid other stakeholders’ interventions. The mission of DCR is to promote teachers’ professional development rather than satisfy other stakeholders.

The state-of-the-art in conceptualization of alignment

In the field of education, alignment refers to the degree to which expectations and assessments are in agreement with one another (Webb 1997). Alignment studies promote reflection on the different components of an educational system as an integral part of the optimization process for achievement of this intended agreement (Martone and Sireci 2009). Instructional alignment is defined as structuring the key elements of instructional design so that the instruction and assessment are aligned with the instructional objectives (Bober et al. 1998). Aligned instruction has been found to be about four times more effective for learning than misaligned instruction (Cohen 1987). Students are more likely to demonstrate what they have learned if teaching objectives and practice are aligned (Roach et al. 2008). Instructional alignment is also a prerequisite for meaningful learning (Carter 2008).

If the instructional design is reasonable, then its enactment should be aligned with the design. However, the enactment often deviates from the instructional design. The main reason is that instructional systems are complex and rife with uncertainty (You 1993). Furthermore, interactions among peers are emergent and spontaneous (Kapur et al. 2011), which contributes both to uncertainty and misalignment. The analysis of misalignment is helpful for identifying design deficiencies, which can be addressed so that the quality of the instructional design improves and knowledge about the design is generated in the process. This is the process through which instructional alignment facilitates achieving the goals of DCR.

However, instructional alignment has received limited attention in previous literatures. Martin (2011) highlighted the importance of instructional alignment and found that the matrix model was very effective at aligning instructional design elements. Macphail et al. (2013) examined how preservice teachers experienced instructional alignment in physical education and found that preservice teachers understood the process of instructional alignment and designed instructionally aligned lessons through group discussion and problem solving. Furthermore, Burroughs et al. (2019) found that instructional alignment and time on mathematics had a mediating effect on learning outcomes. Zheng (2015b) conducted a case study on the alignment between an instructional design plan and its enactment in a science course within an elementary school. She found that it achieved medium consistency between the instructional design and its enactment.

Furthermore, alignment research in the field of collaborative learning has received less attention, as mentioned above. To the best of our knowledge, only two collaborative learning studies have conducted alignment research. One examined to what extent students enacted their assigned roles in collaborative learning (De Wever et al. 2008). Another aligned the learning design and learning analytics through scripting and monitoring in CSCL scenarios (Rodríguez-Triana et al. 2015). To close this gap, this study aims to evaluate the alignment between a collaborative learning design and its enactment using the DCR approach.

An introduction to the method of evaluating the alignment

The procedure

The procedure of evaluating alignment in the CSCL context includes five steps, as shown in Fig. 1:

  1. 1.

    First, design a collaborative learning activity and write a design plan, including the collaborative learning objectives, target knowledge map, collaborative learning task, interactive approach, learning resources, and assessment methods. The interactive approach includes interactive strategies, role assignment, and interactive information types (Zheng 2017). These design elements are grounded in activity theory proposed by Engeström (1999).

  2. 2.

    Second, conduct the collaborative learning activity in a first round.

  3. 3.

    Third, analyze the discussion transcripts of each group using an IIS (Interaction Information Set)-map-based analysis method (Zheng et al. 2012). The following section will illustrate how to use this method.

  4. 4.

    Fourth, evaluate the alignment between a collaborative learning design and its enactment through three proposed alignment indicators and the interaction path graph. These are unpacked in turn below. The interaction path graph represents how the designed collaborative learning activity and the actual collaborative learning activity unfold step-by-step.

  5. 5.

    Fifth, reflect on the misalignment and use insights from this reflection to decide how to fine-tune the collaborative learning design. If necessary, a second round of collaborative learning can be conducted to examine the effectiveness of the chosen optimization strategies.

Fig. 1
figure 1

This figure illustrates the five step procedure for evaluating alignment in a CSCL context

Full size image

Data analysis method

Here we illustrate how the IIS-map-based analysis method is used to analyze discussion transcripts for each group. This method includes three steps:

  1. 1.

    First, draw the target knowledge map using our interaction analytical tool (Zheng et al. 2012). The knowledge map represents the target knowledge and the associated relationships. Take the first collaborative learning task about learning motivation as an example. Figure 2 shows the target knowledge map for the first round of collaborative learning.

  2. 2.

    Second, code each group’s discussion transcripts into information flows through the interaction analytical tool (Zheng et al. 2012). Table 1 shows fragments of the discussion transcripts from one group. Two coders represented each information flow using the following structure: <Time > <IPLi > <Cognitive level > <Information type> < Representation format> < Knowledge submap> (see Fig. 3).

Fig. 2
figure 2

The target knowledge map for the first round

Full size image
Table 1 The fragments of discussion transcripts
Full size table
Fig. 3
figure 3

The portion of coding results for the first round collaborative learning

Full size image

In particular, Time indicates the staring time of each information flow. IPLi represents who is processing the information. Cognitive levels include remembering, understanding, and applying. Representation formats include texts, graphs, videos, animations, and sounds. The knowledge submap refers to the different types of knowledge discussed and the relationships between them as mapped by information flows. In this study, Cohen’s Kappa as applied to the information flow segmentation was computed at 0.89, indicating good interrater reliability.

  1. 3.

    Third, generate the knowledge map of each group using the analytical tool, as shown in Fig. 4. The number next to each node in Fig. 4 denotes the activation quantity of each node. The activation quantity represents the level of knowledge building, which is measured by the activation entropy. The precise formula can be found in an earlier publication (Zheng et al. 2012).

Fig. 4
figure 4

The knowledge map generated by a group for the first round

Full size image

Alignment indicators

Collaborative learning can be a very effective instructional approach, and particularly so when learners share knowledge and engage in collaborative knowledge building (Eryilmaz et al. 2013; Shin et al. 2018). Barron (2003) proposed that collaborative knowledge building should be considered as a goal when designing collaborative learning activities. In addition, social interaction among peers is a key element and prerequisite for collaborative learning (Kreijns et al. 2003). Dillenbourg (1999) suggested that the way learners interact with each other should be taken into account when designing a collaborative learning activity in order to promote productive interactions and regulate cognitive conflict within the group (Njenga et al. 2017). Therefore, knowledge building and the interaction approach are crucial for productive collaborative learning. Hence, this study is used to illustrate the application of three proposed indicators related to knowledge building and the interaction approach, which are then used to evaluate the alignment between a collaborative learning design and its enactment. The three indicators include the alignment of the range of activated knowledge, the alignment of the degree of knowledge building, and the alignment of the interactive approach itself. The alignment of the range of activated knowledge denotes the frequency with which the same target knowledge is activated in both the design plan and enactment process. The alignment of the degree of knowledge building represents the comparison between the level of collaborative knowledge building that designers expected and that which students actually achieved in their co-construction. The alignment of the interactive approach denotes the extent to which learners interacted with each other in the manner that was planned in terms of interactive strategies, role assignment, and exchanged information types. The following illustrates the algorithmic interpretation of the three indicators described informally above.

The alignment of the range of activated knowledge computes a ratio designed to compare the amount of overlap and non-overlap in knowledge elements between what was intended to the activated and what was actually activated. Possible values range between 0 and 1. The higher the alignment value, the more the target knowledge is the same as that which was actually activated during enactment. It can be calculated using the formula proposed by Tversky (1977), as shown in formula (1). Tversky (1977) proposed this formula to measure the similarity of two sets by comparing the common and distinctive elements (or, from a different perspective, the similarity of two objects based on the commonality of their features). Therefore, in this study, this formula was applied in order compute the alignment of activated knowledge between the collaborative learning design and its enactment.

$$ S=\frac{f\left(A\cap B\right)}{f\left(A\cap B\right)+0.5\ast f\left(A\cap B\right)+0.5\ast f\left(A-B\right)} $$
(1)

where S denotes the alignment of in activated knowledge. A denotes the amount of target knowledge in the design plan. B denotes the amount of activated knowledge during the enactment process. f(A∩B) denotes the amount of commonly activated knowledge in the design plan and enactment processes. f (A-B) denotes the amount of knowledge that only appeared in the design plan. f (B - A) denotes the amount of activated knowledge during the enactment process that was not in the target. If 0 < S ≤ 0.3, the degree of alignment is considered low. If 0.3 < S < 0.8, the degree of alignment if considered medium. If 0.8 ≤ S ≤ 1, then the degree of alignment is considered high.

The alignment of the degree of knowledge building is measured by the similarity between the target knowledge map and the knowledge map generated by each group. The target knowledge map represents the level of knowledge building that designers expected in connection with their design. The knowledge map generated by each group represents what group members actually co-constructed during collaborative learning. The alignment of the degree of knowledge building can be calculated using formula (2).

$$ G=\frac{\left(R+W\right)-\left(D+Y+F\right)}{Z+W} $$
(2)

where G denotes the alignment of the degree of knowledge building. R denotes the score of the proposition chains on a group’s knowledge map that were correctly matched with the target knowledge map. Here, a proposition chain is conceptualized as the set of all propositions along the longest path on the knowledge map (Yao et al. 2006). The correct, broken, missing, and wrong proposition chains are determined through comparing with the target knowledge map. W denotes the score of the new and correct proposition chains during enactment. D denotes the score of the broken proposition chains during enactment. Y denotes the score of the missing proposition chains during enactment. F denotes the score of the wrong proposition chains during enactment. Z denotes the total number of proposition chains in the target knowledge map. If 0 < G ≤ 0.3, the degree of alignment is considered low. If 0.3 < G < 0.8, the degree of alignment is considered medium. If 0.8 ≤ G ≤ 1, the degree of alignment is considered high.

The alignment of the interactive approach includes the alignment of interactive strategies, role assignment, and interactive information types. The interactive strategies represent the interactive forms and rules. Role assignment represents the division and responsibilities of each group member. The interactive information types indicate how to represent information during collaborative learning. These three aspects represent the interaction approach together and it can be calculated using formula (3).

$$ K=\frac{\sum_{i=1}^3\frac{f\left({\mathrm{D}}_{\mathrm{i}}\cap {\mathrm{P}}_{\mathrm{i}}\right)}{f\left({\mathrm{D}}_{\mathrm{i}}\cup {\mathrm{P}}_{\mathrm{i}}\right)}}{3} $$
(3)

where K denotes the alignment of the interactive approach. D1 denotes the interactive strategies in the design plan. P1 denotes the interactive strategies during enactment. The interactive strategies include online discussions, puzzle solving, brainstorming, and so on. f(D1 ∩ P1) denotes the number of common interactive strategies included in both the design plan and collaborative learning processes. f(D1 ∪ P1) denotes the total number of interactive strategies included either in the design plan or enactment processes. D2 denotes the role types included in the design plan. P2 denotes the role types included in the enactment process. The role types include the group leader, monitor, summarizer, and so on. f(D2 ∩ P2) denotes the number of common role types included both in the design plan and enactment processes. f(D2 ∪ P2) denotes the total number of role types included in the design plan and enactment processes. D3 denotes the interactive information types in the design plan. P3 denotes the interactive information types during the enactment process, which should be obtained by analyzing the discussion transcript. The information types include semantic knowledge, goal descriptions, contexts, examples, questions, answers, management instructions, and relevant information. f(D3 ∩ P3) denotes the number of common interactive information types included both in the design plan and enactment processes. f(D3 ∪ P3) denotes the total number of interactive information types included either in the design plan or enactment processes. If 0 < K ≤ 0.3, the degree of alignment is considered low. If 0.3 < K < 0.8, the degree of alignment is considered medium. If 0.8 ≤ K ≤ 1, the degree of alignment is considered high.

An illustrative example of the method

This example illustrates how to evaluate the alignment between a collaborative learning design and its enactment. The topic of this example is how to improve Li Fang’s learning motivation for physics. The collaborative learning objective was to acquire knowledge about the underlying conceptions, theories related to motivation for learning, and strategies for improving learning motivation. This is just an illustrative example of the approach applied to data from one collaborative learning group doing a single session.

First, the collaborative learning plan for the first round was designed, and it is shown in Table 2. Before online collaborative learning, the researcher explained the learning goals, tasks, interactive approach, and assessment method in detail to the group. In addition, learning resources were also provided for all participants of the group to facilitate online collaborative learning.

Table 2 The design plan of online collaborative learning for the first round
Full size table

Second, the online collaborative learning for the first round was conducted. The group of three students who were located in different rooms participated in the online collaborative learning session for two hours. The discussion transcripts were automatically recorded through our online collaborative learning platform.

Third, the discussion transcripts were analyzed through the IIS-map-based analysis method. Figure 2 shows the target knowledge map for this collaborative learning task. Two coders independently coded the discussion transcripts through our analytical tool. Figure 3 shows the coding results. Then, the knowledge map of this group was automatically generated through our analytical tool. Figure 4 shows the knowledge map generated for this group.

Fourth, we evaluate the alignment between the collaborative learning design and its enactment through the three alignment indicators defined above and the interaction path graph. With respect to the alignment of the range of activated knowledge, as shown in Fig. 2 and Fig. 4, the green nodes indicate the commonly activated knowledge, the red nodes indicate the knowledge that only appeared in the design plan, and the yellow nodes indicate the activated knowledge during enactment. It is very clear that f(A∩B) = 21, f(A-B) = 14, and f(B-A) = 4. Thus, the alignment of the range of activated knowledge was equal to 0.700 based on formula (1). With regard to the alignment of the degree of knowledge building, the score of the proposition chains in Fig. 2 was 61, and the score of proposition chains in Fig. 4 that were correctly matched with Fig. 2 was equal to 47. The score of the new and correct proposition chains was equal to 4. The score of the broken proposition chains was equal to zero. The score of the missing proposition chains was equal to 14. The score of the wrong proposition chains was equal to zero. Thus, the alignment of the degree of knowledge building was equal to 0.569 based on formula (2). In terms of the alignment of the interactive approach, Table 3 compares the interactive approaches between the collaborative learning design and its enactment. As shown in Table 3, the common interactive strategies in the design plan and its enactment only include online discussions. Thus, f(D1 ∩ P1) is equal to 1. The total interactive strategies in the design plan and its enactment included online discussions. Thus, f(D1 ∪ P1) was equal to 1. The common role types in the design plan and enactment included a recorder. Thus, f(D2 ∩ P2) was equal to 1. The total role types included in the design plan and enactment included a recorder. Thus, f(D2 ∪ P2) was equal to 1. The common interactive information type in the design plan and its enactment included semantic knowledge, goal descriptions, examples, and management instructions. Thus, f(D3 ∩ P3) was equal to 4. The total interactive information types included in the design plan and enactment included semantic knowledge, goal descriptions, contexts, examples, questions, answers, management instructions, and relevant information. Thus, f(D3 ∪ P3) was equal to 8. Consequently, the alignment of the interactive approach was equal to 0.833 based on formula (3).

Table 3 The comparison of the interactive approach between the CSCL design and enactment
Full size table

In addition, the interaction path graph was drawn to further identify the misalignment (see Fig. 5). It is found that the actual interaction path was different from what the designer expected. For example, one group member proposed the solution directly without analyzing the problem of Miss Li when completing subtask 1. In addition, the group members believed that the major problem of Miss Li was related to psychological problems or laziness. Then, they attributed it to improper learning motivation. When the group members completed subtask 2, one group member proposed suggestions for Miss Li directly without analyzing the characteristics of physics.

Fig. 5
figure 5

The interactive path graph for the first round

Full size image

Fifth, we reflected on the misalignment based on the three indicators and the interaction path graph to optimize the collaborative learning design. It was found that there were 14 knowledge nodes that were not activated during enactment. There were 14 missing proposition chains during enactment. There were four types of information that were not mentioned. In addition, there were several new paths that were not consistent with the design plans. Therefore, the optimization strategies were proposed based on these misalignments, as shown in Table 4. The optimized collaborative learning design plan for the second round can be found in Table 5.

Table 4 The misalignment and the optimization strategies
Full size table
Table 5 The design plan of online collaborative learning for the second round
Full size table

Sixth, we conducted a second round of collaborative learning to validate the optimization strategies. The results indicate that the values of the three indicators were improved. The alignment of the range of activated knowledge, the alignment of the degree of knowledge building, and the alignment of the interactive approach were 0.857,0.833,and 0.917, respectively. Figure 6 shows the knowledge map generated by a group in the second round.

Fig. 6
figure 6

The knowledge map generated by a group in the second round

Full size image

The empirical study

Participants

Participants were recruited through posters on a university campus. In total, 75 undergraduate and postgraduate students voluntarily participated in the study. There were 8 males and 67 females with an average age of 23 years of age. Their majors of study included politics, educational science, psychology, English, Chinese, law, and physics. The gender imbalance of our sample reflects the gender composition of this university, with approximately 80% being female students.

The experimental procedure

The purpose of this experiment was to validate the proposed method of evaluating the alignment and optimization strategies. The experimental procedure is shown in Fig. 7. The first step was to design collaborative learning plans, including defining collaborative learning goals, drawing a target knowledge map, designing collaborative learning tasks and interactive approaches, preparing collaborative learning resources, and designing assessment methods. The evaluation study adopted a quasi-experimental design in which participants were assigned to the first round of collaborative learning and/or the second round of collaborative learning semi randomly. The study was not a part of an ongoing course, and all of the collaborative learning plans were designed by the authors. A pretest was conducted before students engaged in any collaborative learning activity to evaluate their relevant prior knowledge. The pretest consisted of multiple-choice questions, true-false questions, and short-answer questions, with a total score of 100 possible points. Participants were randomly assigned into 40 groups of three students each, which participated in their group work online within different time slots. There were 20 groups in the first round and 20 groups in the second round. There were 47 students who only participated in the study once, 11 students who participated twice, and 17 students participated three times. Students who participated more than once participated in a different task with different associated knowledge each time. And for each collaborative learning activity, the 3 students in the first round and the 3 students in the second round were different students. Furthermore, the students in the two rounds had equivalent prior knowledge as measured by the pretest.

Fig. 7
figure 7

The experimental procedure of this study

Full size image

At the beginning of the experiment, researchers introduced the study and explained the collaborative learning goal as well as the different tasks to participants. Then, each group participated online in group work for about two hours using the online collaborative learning tool (see Fig. 8). During collaborative learning, no assistance was provided unless participants could not log in to the online system. In the first round, each group completed one unique collaborative learning task listed in Table 6. In total, 20 groups completed 20 collaborative learning tasks in the first round at different time slots for one month. Then, the data from the first round were analyzed according to the aforementioned method. Based on the analysis results, the collaborative learning design plans for 20 collaborative learning activities were refined and optimized. One month later, another 20 groups conducted the second round of collaborative learning at different time slots and completed the same set of collaborative learning tasks as the first round. Each group completed one unique collaborative learning task for about two hours. The students who participated more than once took a different task with a different grouping. After collaborative learning, each group uploaded the group products through the online collaborative learning platform. Finally, the alignment and group products of the second round of collaborative learning were evaluated and compared with the first round.

Fig. 8
figure 8

The online discussion screenshot

Full size image
Table 6 List of 20 collaborative learning tasks
Full size table

Results

Pretest results of prior knowledge in the first and second rounds

To ensure that the participants who conducted online collaborative learning in the first and second rounds had equivalent levels of prior knowledge, a pretest was conducted before the experiment. Table 7 shows the results of the pretest for 40 groups in the first and second rounds. All data were distributed normally. The paired t-test results indicated that there were no significant differences in the prior knowledge for the 40 groups between the first and second rounds.

Table 7 The results of pre-test in the first and second rounds
Full size table

Alignment between design and enactment in the first round

Table 8 shows the results of the alignment of 20 groups in the first round. The range of activated knowledge achieved the highest alignment (M = 0.664, SD = 0.115), followed by the interactive approach (M = 0.635, SD = 0.208), and the degree of knowledge building (M = 0.335, SD = 0.181). In terms of the alignment of the range of activated knowledge, three collaborative learning activities achieved high consistency, and the remaining 17 activities achieved medium consistency. With respect to the alignment of the degree of knowledge building, 9 collaborative learning activities achieved low consistency, and the remaining 11 reached medium consistency. Regarding the alignment of the interactive approach, only one collaborative learning activity achieved low consistency, 14 activities reached medium consistency, and the remaining five met the criteria for high consistency.

Table 8 The results of the alignment in the first round
Full size table

Identify misalignment and refine the collaborative learning design

After the first round of collaborative learning, the misalignment between the collaborative learning design and its enactment was detected through the three indicators and interaction path graph. First, it was found that there was misalignment in terms of the range of activated knowledge between the CSCL design and its enactment. This part of the knowledge was not mentioned and activated during the enactment. This phenomenon occurred in almost every group. The main reason was that participants forgot what they had learned or did not acquire the target knowledge. This result also revealed that the initial design plan was defective in terms of the design of learning resources. There were not sufficient learning materials in the first round of collaborative learning related to some necessary prior knowledge for the task.

Second, the present study also found that there was misalignment in terms of the degree of knowledge building between the collaborative learning design and its enactment. For example, the missing or incorrect proposition chains often appeared in the actual knowledge maps. The findings resulted in inconsistency between the target knowledge maps and the actual knowledge maps. The main reason probably was that participants had misconceptions or failed to link prior knowledge with new information. This finding revealed that the initial design plan was defective with respect to the collaborative learning task, learning resources, and assessment method.

Third, this study also found that the interactive approach between the collaborative learning design and its enactment was not consistent. Some of the students did not perform the assigned roles and interactive strategies during collaborative learning. In addition, there was a dearth of structured interactive strategies, role assignment, and clear interactive rules in the initial design plans. This might be the reason why there was misalignment concerning the interactive approaches. Furthermore, the present study also found that the actual interactive topics and paths were not in line with what teachers designed through the interaction path graph.

To improve the alignment and design quality, 20 design plans of collaborative learning activities were optimized. We take the 13th collaborative learning task as an example. This collaborative learning task engaged students in discussing how to solve well-structured problems and ill-structured problems and how to improve problem-solving skills. Table 9 shows the optimization strategies employed along with illustrations.

Table 9 The misalignment and optimization strategies
Full size table

Alignment between design and enactment in the second round

As shown in Table 10, the alignment between the collaborative learning design and its enactment in the second round was higher than in the first round. The range of activated knowledge achieved the highest consistency (M = 0.912, SD = 0.070), followed by the interactive approach (M = 0.907, SD = 0.070), and the degree of knowledge building (M = 0.837, SD = 0.149). Concerning the alignment of the range of activated knowledge, only one collaborative learning activity achieved medium consistency, and the remaining 19 activities achieved high consistency. Regarding the alignment of the degree of knowledge building, 6 collaborative learning activities achieved medium consistency, and the remaining 14 had high consistency. With respect to the alignment of the interactive approach, only 2 collaborative learning activities achieved medium consistency, and the other 18 achieved high consistency. In addition, it was found that the reliability of the three indicators for the 40 groups achieved high reliability based on the intraclass correlation coefficient (r = 0.831, p = 0.000). Moreover, there was a significant relationship between the range of activated knowledge and the degree of knowledge building (r = 0.912, p = 0.000). There was also a significant relationship between the range of activated knowledge and the interactive approach (r = 0.561, p = 0.000). Therefore, these three indicators jointly represented the alignment between the collaborative learning design and its enactment.

Table 10 The results of the alignment in the second round
Full size table

Difference in alignment between the first and second rounds

To further analyze the difference in the alignment between the first and second rounds, a paired t-test was conducted, as shown in Table 11. The results indicated that the differences in the alignment of the range of activated knowledge between the first round and second round (p = 0.200), the differences in the alignment of the degree of knowledge building between the first round and second round (p = 0.186), and the differences in the alignment of the interactive approach between the first round and second round (p = 0.077) were normally distributed. There were significant differences in terms of the alignment of the range of activated knowledge (t = 9.336, p = 0.000), the degree of knowledge building (t = 11.250, p = 0.000), and the interactive approach (t = 6.502, p = 0.000). Therefore, the alignment of the second round was significantly improved after optimizing online collaborative learning design plans.

Table 11 The results of paired t-test
Full size table

Difference in group performance between first and second rounds

In this study, group performance was measured by the activation quantity of each group’s knowledge map, namely, the sum of the activation quantity of each node in the knowledge map. The algorithm has been validated in previous work (Zheng et al. 2012). To examine the difference in group performance between the first and second rounds, we tested the normality of the distribution of scores. Since all data about group performance were normally distributed (p = 0.174), a paired t-test can be adopted to examine the difference between groups. The findings indicate that there was a significant difference in the group performances between the first and second round (t = 6.864, p = 0.000). The group performance in the second round (M = 408.45, SD = 185.40) was higher than that in the first round (M = 138.52, SD = 74.12). Therefore, the improvement of the alignment between the design plans and collaborative learning processes was associated with a significant enhancement in group performance in online collaborative learning.

Discussion and implications

This study designed and implemented 2 versions of each of 20 online collaborative learning activities and evaluated the alignment between a collaborative learning design and its enactment based on the DCR approach for both versions. The results indicated that the three indicators and interaction path graph together offer an evaluation of the alignment between a design and its enactment that provides valuable insights for optimization of the designs. It was also found that the alignment of the second round of collaborative learning was significantly higher than that of the first round. In addition, the improvement of the alignment was associated with improvement in group performance.

The evaluation of alignment

The present study proposed an innovative method to evaluate the alignment between a collaborative learning design and its enactment. This new method emphasizes analyzing the design deficiencies and refining the collaborative learning design through evaluating the alignment between the design and its enactment. The evaluation of the alignment contributes to how well the collaborative learning design achieved its intent and how to guide the design improvement.

The present study revealed that the alignment of the range of activated knowledge, the alignment of the degree of knowledge building, the alignment of the interactive approach, and the interaction path graph were very effective for evaluating the alignment between a collaborative learning design and its enactment. The range of activated knowledge and the degree of knowledge building represent the breadth and depth of the collaborative knowledge building, respectively. The interactive approach represents how learners interact with each other during collaborative learning. An interaction path graph represents the routes of the designed collaborative learning and the actual collaborative learning activity. Dillenbourg et al. (1996) stated that collaborative learning is a process of collaborative knowledge building through peer social interactions. Therefore, the three indicators and the interaction path graph are able to evaluate the most important elements of collaborative learning. Furthermore, the evaluation of the alignment provides a new perspective on supporting collaborative learning through the improvement of the design quality.

Optimizing collaborative learning design

After the evaluation of the alignment between the collaborative learning design and its enactment, it was found that the collaborative learning design in the first round was defective to some extent, and the measurements pointed to specific deficits that needed to be addressed. These specific deficits connect with findings in the literature that offer guidance on what types of support affect which processes and outcomes. Thus, the results of the alignment measurements in connection with published literature informed us of specific types of adjustments that would be likely to improve the alignment between the processes and outcomes we wanted to see and what we actually saw. To that end, the following strategies were adopted in the second round of collaborative learning.

First, cognitive scaffolding for recalling and understanding target knowledge was provided in the second round. Previous studies indicated that providing cognitive scaffolding was helpful for activating domain knowledge (Pattalitan 2016; Vogel et al. 2017) and improving knowledge acquisition skills (Raes et al. 2012). Cognitive scaffolding contributed to promoting knowledge building through eliciting explanations, high-level elaboration, and activating prior knowledge (Demetriadis et al. 2008). Therefore, cognitive scaffolding including learning materials, prompts, prior knowledge, and questions were delivered to participants through the online collaborative learning environment in the second round. With the help of cognitive scaffolding, it was found that the alignment of the range of activated knowledge and the degree of knowledge building in the second round were higher than those in the first round.

Second, metacognitive scaffolding was also delivered for all groups through the online collaborative learning environment in the second round to promote collaborative knowledge building. Previous studies indicated that metacognitive scaffolding was very helpful for promoting knowledge building (Zheng et al. 2019) and improving knowledge gains (Eshuis et al. 2019; Kramarski and Dudai 2009). The present study provided metacognitive scaffolding through prompts and guidelines to guide learners to set learning goals, make plans, monitor the collaborative learning process, and evaluate and reflect on group products. The results indicated that the alignment of the degree of knowledge building in the second round was higher than that of the first round.

Third, scripts about how students should interact with each other were provided for all participants to promote productive interactions. Scripts aim to structure the collaborative learning process to promote group interactions (Dillenbourg and Tchounikine 2007). Previous studies indicated that providing scripts was very useful for enacting an assigned role (Avcı 2020; De Wever et al. 2008) and promoted learning outcomes in the CSCL context (Mende et al. 2017). The scripts in the present study included structured guidelines, prompts, and questions. For example, instructions for how to conduct online discussion, brainstorm, and solve puzzles were provided for each group. The guidelines about the role assignment were listed in detail to guide participants to participate in line with the expectations. The clear interaction rules and prompts were specified in the second round of the collaborative learning. The results indicated that the alignment of the interactive approach in the second round was higher than that in the first round.

After optimizing the collaborative learning design plans, the results indicated that the alignment in terms of the range of activated knowledge, the alignment of the degree of knowledge building, and the alignment of the interactive approach were significantly improved. It was also found that group performance also significantly increased in the second round. Therefore, the proposed optimization strategies contributed to maximizing the alignment and improving the design quality.

The main contribution of this study

The main contribution of this study is to advance a novel method of optimization of a collaborative learning design through evaluating the alignment between an ideal vision for a collaborative learning design and the reality of its enactment. Here we elaborate on what is novel in this approach.

First, this method is designed to allow researchers to quantitatively evaluate the alignment between a collaborative learning design and its enactment. It provides a clear comparison between what teachers designed and what students enacted. The extent of the alignment can be easily calculated through the three indicators. The alignment can also be observed through the interaction path graph. As a whole, this method offers a more intensive quantification of alignment than previous methodologies more standardly used in CSCL, such as for DBR.

Second, as Meijer et al. (2020) indicated, the potential of a collaborative learning assessment method mainly depends on whether or not it can pinpoint what specifically is interfering with meeting the collaborative learning objectives. Our method is very promising for identifying particulars of the misalignment between a collaborative learning design and its enactment. In this way, the method links collaborative learning design with collaborative learning analytics to evaluate the alignment and analyze misalignment.

Third, this method provides new insights for reflecting on a collaborative learning design and its enactment. If collaborative learning objectives are reasonable, but the enactment fails to live up to the expectation, then it is necessary to reflect on why the collaborative learning design does not align with its enactment and the reasons for the misalignment to improve the design quality. The analytics that are core to this approach aid in this reflection. Thus, this approach might stimulate more intensive reflection on design in the field of CSCL.

Fourth, this method is a data-driven analytical approach to systematically inform design decisions over time. Thus, this innovative method provides strong decision support to guide the design efforts of teachers, enabling their decision making to be data-driven. Teachers can make scientific decisions and optimize future collaborative learning designs based on the analysis results of the alignment. The quantitative analysis results might also be used to inform guide and assist students during collaborative learning. This method also provides more details about collaborative learning processes and their connection with outcomes rather than only measures of learning achievements.

Finally, this method contributes to developing technological knowledge through analyzing the reasons for misalignments and design deficiencies. The resulting knowledge has the potential to contribute to generalizable knowledge about collaborative learning design more broadly.

Implications for designing and implementing collaborative learning

Collaborative learning design is very important for improving students’ learning performance and teachers’ professional skills. The results of this study showed that optimizing a collaborative learning design can contribute to group performance. Based on the findings of the study, design and enactment implications were proposed as follows. Above we discussed the approach as contextualized within a specific design effort. Here we discuss it more generally, discussing the process in a decontextualized manner, made precise in Table 12, and then discussing potential broader impacts of the work. As we have discussed above, the framework includes five basic design elements, namely, collaborative learning goals, tasks, interactive approaches, learning resources, and assessment methods. This framework also indicates the key considerations in a collaborative learning design.

Table 12 The design framework of collaborative learning activities
Full size table

This study has important implications for teacher professional development. As indicated, training teachers as learning designers can promote educational innovation (Asensio-Pérez et al. 2017). The collaborative learning design and its enactment are two interdependent processes. Teachers can learn from the successes and failures of what they have designed to improve CSCL design. Teachers can also reflect on the enactment misalignment to refine future collaborative learning designs. The alignment study will also help teachers to be aware of the deviation and prepare possible solutions for the next round of learning design. In this way, teachers’ instructional design skills might be improved, thus leading to a higher level of professional ability.

Moreover, misalignment can in some ways be expected as a normal part of the design and enactment process. There will always be an extent to which details of the enactment offer an element of surprise. The full extent of what learners do and discuss and learn cannot be predicted in advance. Learners can generate new knowledge based on their understanding of the subject matter. However, the new knowledge cannot be designed or estimated in collaborative learning design plans ahead of time. In fact, misalignment is allowed to some extent only if learners can achieve collaborative learning goals. However, too much misalignment indicates that the collaborative learning design was ineffective. With the proposed approach, this uncertainty can be accommodated. It is invited to occur, as it will, and the methodology provides tools to meet it an address it appropriately.

Finally, this study proposed that future studies on collaborative learning should focus on two key themes. The first theme is developing more in depth knowledge about collaborative learning design. This can be achieved through examining how different design decisions impact collaborative learning processes and outcomes through this new lens. Researchers and practitioners can also compare the effects of different choices in collaborative learning design components. The second theme is enabling more widespread research evaluating alignment between the collaborative learning design and its enactment. Intelligent tools for evaluating alignment automatically would be an excellent direction for future research in order to improve the efficiency of the approach even further. Moreover, the metrics for analysis of the misalignment and design deficiencies could be developed even further.

Conclusion

In conclusion, methodology for designing collaborative learning activities is extremely important but frequently neglected in research studies. This study adopted the DCR approach to evaluate the alignment between a collaborative learning design and its enactment for 40 collaborative learning sessions altogether. The results indicated that the new method was informative and holds potential for identifying and evaluating alignment. It was found that the three indicators, namely, the alignment of the range of activated knowledge, the alignment of the degree of knowledge building, and the alignment of the interactive approach can effectively evaluate the alignment between the design plan and enactment process. In addition, the interaction path graph is very helpful for identifying the misalignment between CSCL design and enactment. The results also indicate that the degree of alignment in the second round was significantly higher than that in the first round after refining the collaborative learning design. The findings further revealed that improving the alignment can indeed improve group performance. Furthermore, this study also sheds light on how to improve design quality through analyzing the misalignment and design deficiencies.

This study was constrained by the following limitations. First, the sample size was small and only 20 groups participated in the collaborative learning activities in the first and second rounds, respectively. Future studies will expand the sample size to engage different students in conducting two rounds of online collaborative learning. Second, the duration for each online collaborative learning activity was approximately 2 h. In the future, more complex collaborative learning tasks will be designed to extend the duration of activities. Third, this study only examined the alignment from the perspective of the acquired knowledge and interactive approaches. Future studies will explore the alignment concerning learning engagement, behaviors, and durations. Finally, the collaborative learning design framework needs to be examined and refined in future studies.