Collaborative learning is defined as an instructional method that allows pairs or small groups of students to work together towards a common goal; it has been advocated as having the potential to enhance active exchange, critical thinking, and achievement (Gokhale 1995; Johnson and Johnson 2003; Totten 1991). These authors emphasize several important aspects that characterize effective collaborative learning: positive interdependence, group and individual accountability, interpersonal skills, the ability to self-monitor and ensure consistent progress towards the goal, and the ability to discontinue patterns of behavior that impede the progress. Collaborative problem solving in a shared virtual space can result in a positive educational experience for students who would otherwise be isolated by time and space.

However, collaborative problem solving or learning, increasingly carried out in the complex technology-mediated communication environments, has its challenges for the participating collaborators, who must switch or split attention and engage in multiple media-induced tasks or task switches (Rosen et al. 2013). A closer examination of the constructs within complex environments for shared space has been suggested as the key to understanding the role of the computer for effective learning collaborations in technology-mediated, virtual shared spaces (Roschelle and Teasley 1995). Therefore, this current experimental study examines the effect of collaboration and multitasking on technology-based problem solving performance in the virtual spaces.

Conceptual framework

Collaboration and collaborative problem-solving

Collaboration can be viewed as a process by which we negotiate and share meanings relevant to the task at hand (Roschelle and Teasley 1995). In the broadest terms, collaborative learning is a situation in which two or more people learn or attempt to learn something together (Dillenbourg 1999). Kaye (2012) characterizes collaborative learning as being a secondary outcome of a task-oriented activity. However, collaboration for learning does not necessarily take place simply because students are co-present (Roschelle and Teasley 1995); good collaborative practice will depend on the development of reciprocity and cooperation among students (Chickering and Ehrmann 1996). Roschelle and Teasley (1995) discussed collaboration as “a coordinated, synchronous activity that is the result of a continued attempt to construct and maintain a shared conception of a problem” (p. 70). Talk is the primary resource in the process, allowing the production of shared knowledge, divergent understandings, and resolution of problem solving impediments (Roschelle and Teasley 1995) by the use of constructive dialogue. Collaboration in problem solving is shown to promote creative thinking skills and to reduce problem solving anxiety (Gokhale 1995). Coordination is essential to problem solving. Barron (2000) conducted a study to describe the types of interaction that promote coordination. He attributed differences in the performance of problem solving to the degree to which collaborators will have (1) shared task alignment, (2) joint attention for solution, and (3) a mutuality and reciprocity of contribution.

The skill set required to participate in teamwork and solve problems collaboratively is considered a precondition for success in many learning and working contexts (National Council of Teachers of Mathematics 1989; National Research Council 1996; Rummel and Spada 2005). The ability to define and solve problems is a highly-valued skill in the knowledge-based, interdisciplinary, and distributed work of today (Barron 2000). Salomon (1993) described two kinds of cognition in collaboration: off-load and shared cognition. Off-load cognition is characterized by a division of labor. Individuals pass on their responsibilities to peers. Shared cognition, on the other hand, is characterized by shared and interactive labor. Individuals are engaged in the same process collaboratively, resulting in the same or different outcomes. Salomon (1993) argued that shared cognition is likely to advance the participants’ competencies while the off-load cognition may in fact reduce the participants’ opportunities to learn.

Increasingly, communication technologies provide both synchronous and asynchronous technology tools that support a more abstract view of collaborative, active learning. Technology-mediated collaborative learning or computer-supported collaborative learning (CSCL) extends collaborative learning beyond face-to-face environments (Alavi and Dufner 2005; Scardamali and Bereiter 1994; Wegerif 2006) and going beyond that, supporting the flexible learning paradigms that were once dependent almost solely on email and computer conferencing (Collis and Moonen 2001). Computer-mediated communications for new learning paradigms are also associated with shifting philosophical foundations, from objectivist to constructivist views, in fields of learning theory and instructional design (Jonassen et al. 1995). Bruffee (1999) points out that collaborative learning in higher education is creating a need for reexamining the assumptions of knowledge, authority, and institutions within the social constructive framework. Gokhale (1995) posits that collaborative learning is most effective when the primary objective of a teacher is not transmission of information but the development of a students’ ability to learn.

Collaborative problem-solving and multitasking in virtual shared spaces

Collaborative problem solving naturally involves some level of multi-tasking or dual tasks, and becomes multi-layered and out of sequential order due to an increased amount of information processing and interactions (Cross et al. 2016; Kolfschoten and Brazier 2013; Lin 2013). The multitasking process in collaborative problem-solving becomes intensified further in a virtual technology-intensive environment due to the affordances of technologies. This is because the collaborators can take advantage of multiple technologies, for instance, mobile technologies, mobile apps, Internet screens, online game environments, to name a few, to switch easily and quickly between different tasks and modes of conversations (Gurvich and Van Mieghem 2015).

Research in general has shown that when people try to conduct several tasks at the same time or switch quickly between different tasks, that is, when they try to multi-task, they are cognitively overloaded and are less productive than if they would have focused on one task (Burgess 2000; Hembrooke and Gay 2003; Just et al. 2001; Lin 2009; Meyer and Kieras 1997; Ophir et al. 2009; Rosen et al. 2013). Clearly, the benefits of collaborative and problem solving can be upset by the potential multitasking and cognitive overload involved in the more complex problem solving process. Regardless, however, multitasking is prevalent phenomenon in the society, especially among the youth (Foehr 2006; Lenhart et al. 2010; Pea et al. 2012; Rideout et al. 2010; Rideout 2013, 2015). Some other studies have also shown that in certain circumstances, the learners may be more productive in a multitasking setting than in a quiet, sequential, or single-tasking environment (Andrade 2010; Poldrack and Foerde 2007; Lin et al. 2011; Lin et al. 2009). Therefore, understanding the cognitive load involved in collaborative problem solving is important in designing, scaffolding or facilitating technology-based collaboration and learning.

The current study

With the current study, we hope to obtain a better understanding of students’ collaborative problem solving performance in a complex setting. We were interested in finding out the relationships between collaboration, multitasking, and problem solving performance and how such relationships would develop over time. A natural form of model-based reasoning, the learning-dependent progression of working mental models, can provide a view of what a learner knows based on the state of cognitive structures during task-oriented problem solving (Ifenthaler et al. 2011; Ifenthaler and Seel 2005, 2013). An examination of the learning-dependent progression of cognitive structure over time allows recognition of changing patterns of reasoning which can be viewed as ranging from novice to expert cognitive learning strategies (Ifenthaler and Seel 2005; Jonassen 2000).

We were also interested in empirically documenting the degree to which collaboration influences students’ performance. We hypothesized that: (1) there is a relationship between multitasking, collaboration, and problem solving performance (Hypothesis 1); and that (2) collaboration is positively related to the overall problem solving performance (Hypothesis 2).

Methods

Participants

One hundred and four (N = 104) university students participated in the study. They were 63 female and 41 male students, who were enrolled in an intermediate-level teacher education course. Their mean age was 23.49 years (SD = 4.22).

Materials and instruments

The participants were asked to solve analytical reasoning (AR) problem tasks chosen from graduate record exam (GRE) tests. As shown in Table 1 below, AR problems require an understanding of a given structure of arbitrary situational relationships for subsequent deduction of new information from the given relationships for constraint-satisfaction (Kaufman et al. 2001). The AR problem questions for the study were assigned a difficulty rating, on a scale of 1 to 6, where 1 = least difficult (d1) and 6 = most difficult (d6), based on guidelines for analysis of content characteristics to the difficulty and discrimination of GRE problems (Chalifour and Powers 1989).

Table 1 Two examples of GRE reasoning tasks from the lowest to the highest difficulty levels, and solutions
Full size table

Multiple instruments were administered to test participants’ pre-dispositions and changes of dispositions over time. These include: (1) Verbal ability test (r = .96; split-half reliability; Amthauer et al. 2001); (2) Multi-tasking preference inventory or the Inventory of Polychronic Values (IPV; Bluedorn et al. 1999). The IPV has 10 items measured on a 7-point Likert response scale (anchored from strongly disagree to strongly agree) with higher values indicating a more polychronic or multi-tasking attitude. An example of a statement is: “When I work by myself, I usually work on one project at a time.” Bluedorn et al. (1999) reported Cronbach’s alpha = .822 and higher as evidence for construct validity for this instrument. The retest–reliability coefficient over a 2-month interval is .78 (Conte and Jacobs 2003); (3) Integrated communication technology learning (ICTL) instrument. It was measured on a 7-point Likert response scale (anchored from strongly disagree to strongly agree) with higher values indicating a preference for communication technology use. An example of a statement is: “I use Internet when I want to find out about something new.” The ICTL instrument included 15 items (Cronbach’s alpha = .605). (4) Formal-to-informal learning scale (12 items. 7-point Likert scale from strongly disagree to strongly agree. Cronbach’s alpha = .695; Mills et al. 2014); A sample statement includes “I learn new things by exploring them myself.” (5) Technology affinity scale (22 items. 7-point Likert scale from strongly disagree to strongly agree. Cronbach’s alpha = .624; Mills et al. 2013). A sample statement reads: “I communicate with my friends mostly via Short Message Service (SMS); and (6) Confidence, effort, motivation, collaboration, tools, and strategy inventory (5 items, Cronbach’s alpha = .692). This paper will only report results examining the participants’ problem-solving performance and their levels of collaboration between the collaborative multitasking (CMT) and collaborative single-tasking (CST) groups.

Design and procedure

The experiment was conducted using a web-based platform during a period of one semester. All the materials and instruments were uploaded and administrated through the online platform, which enabled tracking of the participants’ online activities and the time they spent on the activities. The participants were randomly assigned to one of the four conditions:

  1. 1.

    Individual and Multi-task (IMT; n1 = 26). The participants in this condition worked on the problems by themselves, but they were presented two problems to solve on a split screen each time (a multi-tasking situation).

  2. 2.

    Collaborative and Multi-task (CMT; n2 = 26). The participants in this condition each were randomly paired with another participant to solve the problems together. They were also presented two problems to solve on a split screen each time.

  3. 3.

    Individual and Single-task (IST; n3 = 26). The participants in this condition worked on the problems by themselves and they were presented with one problem to solve each time.

  4. 4.

    Collaborative and Single-task (CST; n4 = 26). The participants in this condition each were randomly paired to work with another participant, but they were presented with one problem to solve each time.

The experiment was conducted in two phases. During the first phase, the participants were asked to complete a group of surveys, including a demographic data questionnaire, the verbal abilities test, the multi-tasking preference inventory, the integrated communication technology learning scale, the formal to informal learning scale, and the technology affinity scale. During the second phase, each participant was assigned the corresponding tasks and assessments based on his or her login and condition. The participants were assigned seven tasks in 7 weeks (one task per week with an increasing difficulty level), and they were measured using seven measurement points. Figure 1 shows the study design with the four conditions and eight data points (including seven measurement points).

Fig. 1
figure 1

Longitudinal research design including eight measurement points (IMT individual and multi task, CMT collaborative and multi task, IST individual and single task, CST collaborative and single task; O x measurement of states and performance, 1 individual, C collaborative, MT multi task, ST single task, d1 low task difficulty, d2 medium task difficulty, d3 high task difficulty)

Full size image

After solving each of the tasks, the participants were asked to type in (1) the solutions to the tasks; (2) confidence in the accuracy of the solutions; (3) levels of motivation; (4) the problem solving strategies applied; and (5) an estimated time on task (although the time they spent on the task was also stamped in the platform). Additionally, the participants in the two collaborative conditions were asked to note their degrees of collaboration and the methods they used to collaborate (e.g., chat or email).

Analysis

Initial verbal ability and multi-tasking preference scores were calculated for each student. Each participant’s task solution scores were determined at each measurement point. Additional measures were calculated, including confidence in accuracy of the solutions, levels of motivation, the problem solving strategies applied, an estimated of time on task, the estimated degree of collaboration, and the method of collaboration.

Results

The effect of multi-tasking and collaboration

We computed a repeated-measure MANOVA with the intensity of collaboration at five measurement points as a within-subjects factor, and experimental conditions (CMT, CST) as a between-subjects factor. MANOVA revealed a significant main effect of time on intensity of collaboration, Wilks’ Lambda = .782, F(4, 47) = 3.28, p < .05, η 2 = .218, and for time x condition, Wilks’ Lambda = .771, F(4, 47) = 3.49, p < .05, η 2 = .229. The sphericity assumption was met (χ 2(9) = 11.45, p = .25). The difference between measurements was significant, F(4, 200) = 2.43, p < .05, η 2 = .046.

A pairwise comparison of intensity of collaboration at each measurement point (MP) indicated significant differences between experimental conditions as follows: MP2, t(50) = 3.61, p < .001, d = 1.00; MP4, t(50) = 3.17, p < .01, d = .88; MP5, t(50) = 4.64, p < .001, d = 1.29 (see Table 2 for descriptive statistics).

Table 2 Means (standard deviations in parentheses) of intensity of collaboration over time
Full size table

Further, we found a significant interaction effect of time and condition on the intensity of collaboration, F(4, 200) = 2.88, p < .05, η 2 = .054. Figure 2 shows the interaction effect on the intensity of collaboration.

Fig. 2
figure 2

Interaction of experimental group × time on collaboration

Full size image

That is, the randomly paired participants who had two problems to solve together on the split screen (CMT) reported a significantly higher or more intense levels of collaboration than the randomly paired participants who had only one problem to solve together each time (CST). In addition, as the tasks became more difficult over time, the CMT participants reported a significant increase in the intensity of their collaboration while the CST participants reported a significant decrease in the intensity of their collaboration. To sum up, results showed that the participants who were confronted with multi-tasking activities outperformed the others. Accordingly, we accept Hypothesis 1.

Influence of collaboration on performance

The regression analyses results for the acceptance and use of the three examples of learning analytics systems (ALA) on problem solving performance are presented in Table 3 yielding a ΔR 2 of .319.

Table 3 Regression analysis predicting collaboration on problem solving performance
Full size table

Clearly, collaboration positively predicted the problem solving performance, indicating that the higher the perceived intensity level of collaboration, the higher the overall problem solving performance. Accordingly, we accept Hypothesis 2.

Limitations of the study and future directions

Limitations of the study include the following main issues. For one, the collaboration intensity measurements were based on the participants’ self-reported measures. Self-report measures have long been associated with concerns about response accuracy and validity (Bernard et al. 1984; Stone et al. 2000). Reflected in this study would be the participants’ possibly inaccurate perceptions of their intensity levels of collaboration with their partners at any of the seven measurement points, although the seven measures taken from seven problem-solving experiences during a semester may have mitigated the levels of inaccurate reports by the participants as a whole. Future studies can incorporate eye-tracking and psychometrical measurements to record collaboration intensity more objectively. For another, the collaboration was not clearly defined in this study. Future studies could differentiate different types of collaborations, for instance, using the concepts of off-load versus shared cognition, or division of labor versus shared labor as discussed by Salomon (1993) to capture how the participants collaborated with one another in the virtual environments. Last but not the least, it is worth highlighting that in this study, the participants were asked to either solve the problems by themselves or were paired to solve one or two problems each time. A paired collaboration is different from a group setting, where three or more people work on a problem or two together. Therefore, the results should only be viewed from the paired collaborative problem-solving perspectives.

Discussion and conclusion

This paper reported expanded literature, results, and discussions of the study reported at the international conference on cognition and exploratory learning in digital age (CELDA) (Lin et al. 2015). Due to the space limitation, this current paper focused on one aspect of a larger study and reported the participants’ perceptions of collaboration intensities and their problem-solving performance over the course of a semester during their task-oriented collaborative problem-solving processes in virtually shared spaces. The other aspects of the study are being developed in a paper presented at the AERA conference (Ifenthaler et al. 2016) as well as the other publication channels.

In this paper, we examined how technology-supported collaboration would develop over time, as well as the effect of collaboration on problem solving performance. The results of the study showed that: (1) the participants who were confronted with multi-tasking activities outperformed those who did single tasks; (2) as the task difficulty level increases over time (during seven measurement points during a period of a semester), the CMT paired participants increased their level of collaboration while the CST paired participants decreased their level of collaboration significantly; (3) the higher the levels of collaboration reported by the participants, the better the overall problem-solving performance was demonstrated by the participants. In general, students became increasingly more collaborative over time, and collaboration was a strong predictor for overall problem solving performance.

This result is somewhat counter-intuitive, since the literature in general has repeatedly shown that multitasking increases cognitive overload, resulting in a decrease in task performance in general (Burgess 2000; Hembrooke and Gay 2003). In our study, the paired participants with two problems to solve simultaneously in the split screens virtually and remotely from each other (CMT) outperformed the other groups who tried to solve the same problems, including the paired participants with one problem to solve at a time (CST) as well as the individual participants with one problem (IST) or two problems (IMT) to solve at a time. CMT participants also increased their level of collaboration and their level of problem-solving performance over the time (based on the seven measurement points).

We can speculate that when confronted with two problems to solve simultaneously, the paired participants were immediately confronted with sets of negotiations and logistical decisions to make, for instance, which problem to focus on first and who is going to do what. This might have increased their levels of discussion and collaboration. Additionally, since they were expected to solve two problems simultaneously, they would likely need to talk to each other more frequently and iteratively in order to solve both problems together. There might also be a sense of new excitement and challenges stimulating more interactions and collaborations, because these kinds of multitasking activities are usually not permitted in formal learning settings. Such new challenges in formal learning settings may have increased the participants’ levels of interaction and collaboration, resulting in better performance in general. In addition, the overall better performance in multitasking problem-solving over the period of the semester could be due to the fact that the paired participants have learned strategies to collaborate better in this multitasking setting. In comparison, the participants in the collaborative single-tasking condition may feel more of a need to focus on solving the problem at hand rather than interacting with each other and their more familiar process of collaboratively solving one problem at a time may not stimulate a higher level of collaboration or performance over time. Obviously, although such speculation is aligned to some earlier studies (Andrade 2010; Lin et al. 2011; Poldrack and Foerde 2007), further studies need to be conducted to validate and confirm these suppositions.

This study is significant in several ways. From a theoretical perspective, with the increasing complexities of learning environments being afforded by new technologies, it is important to examine aspects of collaborative learning and problem solving in flexible and multi-tasking environments. A lot is still unknown about the reasons behind the productivity or non-productivity of collaborative problem-solving processes. Methodologically, it is important to advance an understanding in this area of learning by testing hypotheses and conducting experiments to obtain results that may assist educators and learning technologists to advance understanding of how best to design and support a student’s ability to coordinate, collaborate, and problem solve in new distributed workspaces.

Helping students to “develop their capacities for productive engagement in collaborative problem solving is both an educationally and socially important venture” (Barron 2000 p. 433). The new spaces of study and work are increasingly virtual and visited by individuals who are distributed in time and place (Resta and Laferrière 2007). These technology-supported workspaces enable new models of flexible collaboration for learning and problem solving, although they could potentially increase cognitive overload as well. It is important to examine elements and dynamics of such workspaces to ensure smart learning environments for future learners.