Keywords

1 Introduction

Collaborative learning has attracted much attention in the field of education in recent years. Collaborative learning “is a situation in which two or more people learn or attempt to learn something together” (Dillenbourg 1999). Numerous studies have revealed that collaborative learning can lead to critical thinking (Garrison et al. 2001), shared understanding (Roschelle and Teasley 1995), and good social relationships (Johnson and Johnson 1999). In order to produce effective collaborative learning, five basic elements have been proposed by Johnson and Johnson (1999), namely positive interdependence, face-to-face promoting interactions, individual accountability, interpersonal skills, and group processing. Among these five elements, face-to-face interaction can be transformed into synchronous interaction or asynchronous interaction if supported by computers or mobile devices.

Previous studies have indicated that social interaction plays a very crucial role in collaborative learning (Kreijns et al. 2003; Stahl et al. 2006). Typically, there are two approaches to social interaction. The first is the socio-cognitive approach, which focuses on individual development in the social interaction context. The second is the socio-cultural approach, which focuses on the causal relationship between social interaction and individual cognitive change (Dillenbourg et al. 1996). The common trait of these two approaches is that individual cognitive development is based on social interaction. Moreover, the degree of interaction should be determined by the extent to which social interactions influence an individual’s cognitive process (Dillenbourg 1999).

Generally speaking, building collective knowledge, making shared understanding, and creating significant artifacts are fundamental activities in collaborative learning (Stahl 2008). Group members can share information, build common understanding, make artifacts, construct knowledge, and create knowledge through social interaction during collaborative learning. Among these activities, collaborative knowledge building has gained more interest since Scardamalia and Bereiter (2003) proposed knowledge building to the field of education for the first time. The importance of collaborative knowledge building is widely recognized nowadays. Collaborative knowledge building in a collaborative learning context refers to how group members collaborate to construct knowledge and to learn in groups (Chan 2012). Collaborative knowledge building emphasizes collective and increasing responsibility for building a community’s knowledge (Scardamalia and Bereiter 2006).

In order to examine how group members co-construct knowledge in collaborative learning, researchers have attempted to adopt various kinds of methods to analyze the process and outcomes of collaborative learning. The commonly used analytical methods include the conversation analysis method, the social network analysis method, and the content analysis method. The following section will illustrate each of these methods in detail.

2 Literature Review

Different analytical methods have been adopted to analyze the process and outcomes of collaborative knowledge building. In order to obtain evidence of collaborative knowledge building, various kinds of data sources have been collected to triangulate the research findings. For example, group products, logs, posts, questionnaires, interviews, journals, as well as pre- and post-tests are the major data sources. Furthermore, researchers often take some time to collect data, ranging from several days to several months, even several years. Since studies often differ from research purpose or research question, the central concerns are also different. For example, some studies focus on analysis of the contribution of group members, some studies focus on analysis of quality of posts. In this section, the commonly used analytical methods will be illustrated in detail.

The conversation analysis method is often employed to analyze the process of collaborative knowledge building. Conversation analysis centers on how speakers and hearers collaboratively produce sensible ideas by talk-in-interaction (Koschmann 2013). Conversation analysis provides a new way to get a better understanding of social interactions in collaborative learning (Koschmann 2011). Turn-taking, sequence construction, and repair organization are the three basic elements of conversation analysis (Schegloff 1992). The analysis of how group members take speaking turns, construct adjacency pairs and sequences, as well as organize repairs in social interactions can shed light on the nature and processes of collaborative knowledge building. In order to promote productive interactions and improve ideas in collaborative knowledge building, turn design, sequence construction, and repair organization are essential to provide insights into how intersubjective meaning occurs among group members. In addition, the transcripts of conversation analysis include what is said, intonation, volume, pace, and timing (Koschmann 2013). Previous studies have adopted the conversation analysis method to understand group discourse or classroom discourse during collaborative knowledge building. Caswell and Bielaczyc (2002) examined the knowledge-transforming discourse in knowledge forums to understand the evolution of scientific knowledge during an investigation of islands. Zhang and Sun (2011) analyzed idea improvement in a knowledge building community by analysis of online discourse supported by a knowledge forum.

The social network analysis method is another commonly used approach when analyzing the pattern of collaborative knowledge building. Social network analysis considers social relationships as nodes and ties. Nodes represent actors and ties represent the relationships among the actors (Wasserman and Faust 1994). The major indicators in the social network analysis method include betweenness, bridge, centrality, centralization, closeness, the clustering coefficient, cohesion, degree, density, and so on (Wasserman and Faust 1994). These indicators are often used to analyze the participation and contribution pattern of collaborative knowledge building. For example, Zhang et al. (2009) employed the social network analysis method to examine online participatory patterns and knowledge advances so as to provide insight into collective cognitive responsibility. Hong et al. (2010 ) analyzed different network structures for participants and idea interaction in the knowledge society network.

The content analysis method is the most often used method when analyzing collaborative knowledge building processes. The content analysis method is conceptualized as “the research method that builds on procedures to make valid inferences from text” (Rourke et al. 2001). Usually researchers adapt the existing coding schemes or develop a new coding scheme to analyze how group members build knowledge. Many coding schemes were developed and adopted in previous studies. For example, Zhu (1996) developed a coding scheme to analyze meaning negotiation and knowledge building for a distance-learning course. Gunawardena et al. (1997) examined the social construction of knowledge in computer conferencing via the content analysis method. Pena-Shaff and Nicholls (2004) developed a thematic category system to describe online interactions in order to analyze and evaluate knowledge building processes during online discussions. Weinberger and Fischer (2006) developed a multi-dimensional framework to analyze argumentative knowledge building in a CSCL environment. However, most of these coding schemes focus only on speech acts, such as questions, replies, discussions, elaborations, explanations, arguments, reflections, and so on. There are several disadvantages to code transcripts for speech acts. First, how learners construct knowledge is often ignored if you only center on speech acts. Second, the contextual evidence cannot be obtained because coding assigns speech acts an isolated meaning (Suthers et al. 2010). Third, it is very difficult to code discussion transcripts into speech acts because the purpose of such speech acts are implicit (Zheng et al. 2012), therefore, the coding results will be very subjective. Finally, reliability and validity are major concern for the content analysis method. Strijbos et al. (2006) believed that unit boundary overlap affected the reliability and validity of the content analysis method. Therefore, the replication of coding schemes will be limited to other research settings.

To sum up, the existing analysis methods have been employed to serve a different research purpose. However, analysis of the level of collaborative knowledge building from the perspective of knowledge and relationships remains lacking. The present study proposes an innovative analysis method that can analyze the process and outcomes of collaborative knowledge building in collaborative learning. The following section will describe this new method and the empirical study in detail.

3 The IIS-Map-Based Analysis Method

The proposed IIS-map-based analysis method is based on the information flow approach, which considers a collaborative learning system as an abstract information system. This innovative approach focuses on information flows within a collaborative learning system. The information flows are generated by group members during collaboration. The functionality of the collaborative learning system is to collaboratively build knowledge, skills, methods, emotions, attitudes, as well as values by group members. The present study aims to verify that the IIS-map-based analysis method can analyze the process and outcomes of knowledge building both in face-to-face and online collaborative learning.

The main theoretical foundation of this new approach is that knowledge building is closely related to information processing (Wang et al. 2011). Mayer (1996) believed that learners need to select relevant information, organize information, and integrate information with prior knowledge when they construct knowledge. Osborne and Wittrock (1983) also reported that integrating prior knowledge with new information can lead to making meaning and constructing knowledge. Therefore, we argue that the nature of knowledge building is to process information implicitly, including encoding and decoding information (Zheng et al. 2012). The information flow is the output and constructed by group members during collaboration. Thus, the interaction among group members involves sharing information flows, making meaning, and constructing knowledge. The information flow is visible, while knowledge is invisible and needs to be externalized. So information flows can be analyzed directly in order to provide insights into how learners co-construct knowledge during collaborative learning. The following section illustrates the three steps of IIS-map-based analysis method.

First, draw the initial knowledge map based on the collaborative learning objectives and tasks. The domain knowledge can be represented by the initial knowledge map, which can be drawn based on the selected norm. The nodes on the initial knowledge map denote knowledge and the edges denote mutual relationships of knowledge.

Second, code and segment information flows based on the following format:

$$ <{\text{Time}}> <{\text{IPL}}_{i} > < {\text{Cognition level}} > < {\text{Information type}} > < {\text{Representation format}} > < {\text{Knowledge sub-map}} > $$

Here, time denotes the start time of the information flow; IPL i denotes the information processing of different learners; the cognition level includes discerning, remembering, understanding, and applying; the information types include contexts, objectives, knowledge, facts and examples, management instructions, relevant information, and off-topic information; the representation format denotes text, graph, table, sound, video, animation, and body language; the knowledge sub-map denotes part of initial map. The information output flows of group members can be coded based on this format and mapped onto the knowledge sub-map. In addition, rules of segmenting information were developed based on analyzing the large number of samples. The rules specify that information flows will be segmented when the learner, or cognition level, or information type, or knowledge sub-map changes. However, if the representation format changes, information flows will not be segmented because each information flow can be represented by multiple formats.

Third, compute the attributes of information flows and generate the final knowledge map. We assume that some attributes of information flows can predict group performance. The following section will illustrate these attributes one by one. Traditionally, group performance is measured by pre-test and post-test. However, the process of knowledge building is ignored through pre-test and post-test. Therefore, the process-oriented method is called for so as to provide insights into how group members build knowledge together. The IIS-map-based analysis method puts emphasis on the process of collaboratively constructing knowledge. Furthermore, the level of collaborative knowledge building can be automatically calculated by this innovative method.

4 Research Hypotheses

The present study assumed that the following attributes of information flows can predict group performance.

  • H1: The activation quantity of the final knowledge map can predict the level of knowledge building.

  • H2: The average activation quantity of the final knowledge map can predict the level of knowledge building.

  • H3: The standard deviation of activation quantity of the final knowledge map can predict the level of knowledge building.

The activation quantity of the final knowledge map can be calculated by Eq. 1.1. We assume that the activation quantity of the final knowledge map can measure the level of co-construction of knowledge by group members.

$$ A = \sum\limits_{i = 1}^{N} {\sum {\frac{F * \log (d + 2) * r}{\log (n * (D - d + 2))}} } $$
(1.1)

where d denotes the number of activated edges; D denotes the total number of edges; n denotes the categories of edges that are not activated; both F and r are adjustable parameters; and N denotes the number of knowledge in the final map. For more details see Zheng et al. (2012).

The average activation quantity of the final knowledge map can be calculated by Eq. 1.2.

$$ \bar{A} = \frac{A}{N} $$
(1.2)

where A denotes the activation quantity of the final knowledge map and N denotes the number of the nodes in the final knowledge map.

The standard deviation of the activation quantity of the final knowledge map can be calculated by Eq. 1.3.

$$ S = \sqrt {\frac{{\sum\nolimits_{i = 1}^{N} {\left( {A_{i} - \overline{A} } \right)^{2} } }}{N}} $$
(1.3)

5 The Empirical Study

The purpose of this empirical study is twofold: first, it aims to validate the IIS-map-based analysis method as a means of analyzing knowledge building. Second, it aims to examine whether the activation quantity of the final knowledge map can predict the level of collaborative knowledge building. The following section will illustrate the participants, collaborative learning tasks, experimental procedure, and data analysis method in detail.

5.1 Participants

The participants were recruited from one university in Beijing. They all majored in education, psychology, computer science, and educational technology. Some 497 undergraduate students volunteered to participate in this study—85 % of them were female. They were randomly divided into 153 groups of three or four participants. Among these 153 groups, 121 groups conducted face-to-face collaborative learning and 32 groups conducted online collaborative learning.

5.2 Samples

The samples from the study were from knowledge maps not the participants. Each group generated one knowledge map subsequent to collaboration. Therefore, 153 knowledge maps were generated in this study. Hence, it comprised 153 samples.

5.3 Collaborative Learning Tasks

The collaborative learning tasks covered five topics, including how to understand curriculum objectives, the application of consumer behavior theory in microeconomics, the application of knowledge transfer theory, the theory of graphs in data structure, and problem solving strategies. These five tasks included four kinds of knowledge, namely concepts, principles, processes, and facts as well as examples of comprehensive knowledge. For each collaborative learning task, the real-life learning context was provided to participants so as to stimulate interest in collaborative learning. The task assignment depended on participants’ subject domain. Assignment of tasks was as follows: 30 groups completed a task regarding how to understand curriculum objectives; 30 groups completed a task about the application of consumer behavior theory; 31 groups completed a task about the application of knowledge transfer theory; and 30 groups completed a task about the theory of graphs in data structure. These 121 groups conducted face-to-face collaborative learning at different time slots. In addition, 32 groups completed a task about problem solving strategies. These 32 groups conducted online collaborative learning in different labs via instant message software. A research assistant was available only if groups needed help concerning experiment procedure. All of the participants only took part in the experiment once.

5.4 Experimental Procedure

The experimental procedure included the following steps.

First, researchers designed collaborative learning tasks based on the objectives of collaborative learning. The subject domain knowledge needed to be instructed in advance. The purpose of collaborative learning was to strengthen what the participants learned earlier as well as generate new ideas.

Second, the initial knowledge map was drawn based on the collaborative learning tasks. Five knowledge maps were drawn according to the five collaborative learning tasks in the study. The items of pre-test and post-test were also designed ahead of time based on the collaborative learning objectives and tasks. The test items of pre-test and post-test were identical.

Third, participants were recruited by advertising on distributed posters on campus. Before collaborative learning, all of the participants were randomly assigned one group of three or four. Then, they took the pre-test for about 15 min.

Fourth, participants conducted collaborative learning for about 2 h. The breakdown of groups was as follows: 30 groups focused on understanding curriculum objectives; 30 groups focused on the application of consumer behavior theory in microeconomics; 31 groups centered on the application of knowledge transfer theory; and 30 groups focused on the theory of graphs in data structure. The whole face-to-face collaborative learning process of each group was recorded by video. In addition, 32 groups conducted online collaborative learning by instant messaging software. The discussion transcripts were automatically recorded by the software. The post-test was taken immediately after collaborative learning so as to ensure the validity of the experiment. It took about one year to collect the data from the 153 groups.

Fifth, researchers coded and segmented all the data based on the above-mentioned format and rules. At least two raters coded and segmented data of one group in order to assure reliability of the study. It took about one year to code and segment the information flows from the 153 groups.

Finally, the activation quantity of the final knowledge map was calculated using software. Each group generated one knowledge map. The knowledge maps were therefore different in terms of knowledge and relationships for each of the different groups. Thus, 153 knowledge maps were generated after analyzing all of information flows. In the next section we illustrate how to analyze the data using our software.

5.5 Data Analysis

The present study adopted an innovative analysis method, namely the IIS-map-based analysis method to analyze the discussion transcripts of 153 groups. An analytical tool was developed by us to draw the initial knowledge map, code information flows, and compute the activation quantity of the knowledge map. Figure 1.1 shows the initial knowledge map drawn via our analytical tool. Table 1.1 shows the discussion transcripts of one group. Each sentence can be viewed as one information flow. All of information flows in Table 1.1 can be coded based on the aforementioned format and rules, as show in Fig. 1.2. Furthermore, Fig. 1.3 shows the final knowledge map with the activation quantity.

Fig. 1.1
figure 1

The initial knowledge map

Full size image
Table 1.1 Fragments of discussion transcripts
Full size table
Fig. 1.2
figure 2

The fragments of coding information flows

Full size image
Fig. 1.3
figure 3

The final knowledge map with the activation quantity

Full size image

5.6 Inter-rater Reliability

Two trained coders independently coded the discussion transcripts of the 153 groups and assessed the test items of pre-test and post-test. The percentage agreement was used to calculate the inter-rater reliability. The results indicated that the reliability coefficients achieved values of 0.90 and 0.91 for coding discussion transcripts and assessing test items. All of the discrepancies were discussed face-to-face by two coders.

6 Results

6.1 Analysis of Knowledge Building in Face-to-Face Collaborative Learning Environment

Table 1.2 shows the descriptive statistical results for the group performance and the attributes of the information flows. Correlation analysis and linear regression analysis were conducted to test the hypotheses.

Table 1.2 The descriptive statistical results of 121 groups
Full size table

H1 assumed that the activation quantity of the final knowledge map can predict the level of knowledge building. The results indicated that the activation quantity of the final knowledge map was significantly positively related to group performance (r = 0.487, p = 0.000). Moreover, the results of the linear regression analysis indicated that the activation quantity of the final knowledge map can predict group performance (adjusted R 2 = 0.230, β = 0.487, t = 6.077, p = 0.000). The activation quantity can explain 23 % of the total variance. These findings revealed that the activation quantity of the final knowledge map is the significant predictor for the level of knowledge building. Thus, H1 was supported.

H2 assumed that the average activation quantity of the final knowledge map can predict the level of knowledge building. The results indicated that the average activation quantity of the final knowledge map was positively associated with group performance (r = 0.263, p = 0.004). In addition, the results of the linear regression analysis indicated that the average activation quantity of the final knowledge map can also predict the group performance (adjusted R 2 = 0.061, β = 0.263, t = 2.969, p = 0.004). Therefore, the average activation quantity of the final knowledge map can only explain 6.1 % of the total variance. Thus, H2 was also supported.

H3 assumed that the standard deviation of the activation quantity of the final knowledge map can predict the level of knowledge building. The results indicated that the standard deviation of the activation quantity of the final knowledge map was not related to group performance (r = 0.099, p = 0.280). This means the standard deviation of the activation quantity of the final knowledge map cannot predict the level of knowledge building. Therefore, H3 was not supported.

6.2 Analysis of Knowledge Building in Online Learning Environment

Table 1.3 shows the results for the online collaborative learning environment. The findings indicated that the group performance was positively related to the activation quantity (r = 0.369, p = 0.038). However, the average activation quantity (r = 0.305, p = 0.089) and the standard deviation of the activation quantity were not related to group performance (r = 0.265, p = 0.142).

Table 1.3 The descriptive statistical results of 32 groups
Full size table

The findings of the linear regression analysis revealed that the activation quantity can predict group performance (adjusted R 2 = 0.101, β = 0.361, t = 2.086, p = 0.04). Therefore, the activation quantity can explain 10.1 % of the total variance. So H1 was supported in online collaborative learning environment. However, the average activation quantity and the standard deviation of the activation quantity cannot predict the level of knowledge building. Therefore, both H2 and H3 were not supported.

Based on the aforementioned results, the activation quantity of the final knowledge map was the best predictor for the level of knowledge building. Therefore, the activation quantity of the final knowledge map can be adopted to predict the level of knowledge building in the future.

7 Discussion

The findings of this study demonstrated that the activation quantity of the final knowledge map can significantly predict the level of knowledge building. In addition, the IIS-map-based analysis method can also analyze the knowledge building process in collaborative learning. The main reason for this was that the activation quantity of the final knowledge map represented the dynamic features of interaction as a whole, while the other two attributes cannot reflect the complex knowledge structures.

The study viewed collaborative learning as an information system. The information flows of the collaborative learning system were the central concern. The interaction information set was the sharing information set which helped learners to acquire domain knowledge. Jonassen (1999) believed that information was very necessary for learners to obtain knowledge and construct knowledge. Therefore, the present study focused on analysis of information flows during collaborative learning, which provided insights into how group members co-constructed knowledge. Thus, the level of knowledge building could be measured by the attributes of the information flows, namely the activation quantity of the final knowledge map.

The IIS-map-based analysis method is different from the previous approaches in several aspects. First, the sample was a knowledge map generated via output information from group members. The author believes that the knowledge was relatively objective and stable, while the learners varied regarding prior knowledge, personalities, and personal characteristics. Thus the research on the knowledge map can be replicated in other contexts. If we selected participants as the sample, it is very difficult to replicate the results in other educational contexts. Therefore, this innovative approach is more scientific than other approaches that focus on learners’ characteristics. Second, the IIS-map-based analysis method focuses on the knowledge map. The nature of this method is to map information flows onto the knowledge map by natural language. The knowledge map serves as the reference when coding and segmenting information flows. The knowledge map consisted of different knowledge and their inter-relationships represent the level of collaborative knowledge building. Third, the IIS-map-based analysis method is more scientific and has a stronger predictive power for the level of knowledge building. The present study validated that the activation quantity of the knowledge map can predict the level of knowledge building. In contrast with previous studies that focused on speech acts during interactions, this new method focuses on the objective knowledge map and its features. Finally, the IIS-map-based analysis method demonstrates the temporal characteristics of interactions during collaborative learning. Previous studies also highlighted that temporal sequences played a vital role in collaborative learning (Stahl 2011).

However, the present study has several limitations. First, only the activation quantity of the knowledge map was validated to predict the level of knowledge building. A future study will explore other attributes of information flows that can strongly predict the level of knowledge building. Second, the sample size for online collaborative learning environment was very small. Follow up studies will examine this method by increasing sample sizes. Third, the present study only centered on analysis of knowledge building. How to analyze emotions, values, and attitude still need to be explored in future studies.

8 Conclusion

In conclusion, the present study revealed that the IIS-map-based method is an effective method that can analyze collaborative knowledge building in collaborative learning. This study also validated that the activation quantity of the knowledge map was an effective predictor for the 153 groups. The activation quantity of the final knowledge map can significantly predict the level of knowledge building.

There are many benefits to adopting the IIS-map-based analysis method. First, instead of the tests that only focus on results, this innovative analysis methodology is a process-oriented method that can analyze knowledge building processes. Second, knowledge building processes can be visualized by information sequences and knowledge maps. Third, the process and level of collaborative knowledge building can be analyzed and calculated by the IIS-map-based method. Fourth, mapping information flows output by group members onto a knowledge map can minimize the subjectivity of coding information into separate speech acts on a larger extent. Finally, the IIS-map-based analysis method can be replicable and applicable to various kinds of collaborative learning settings. Therefore, this study made a contribution to positioning this analysis method within the field of collaborative learning.