How to improve collaborative learning with video tools in the classroom? Social vs. cognitive guidance for student teams

Abstract

Digital video technologies offer a variety of functions for supporting collaborative learning in classrooms. Yet, for novice learners, such as school students, positive learning outcomes also depend centrally on effective social interactions. We present empirical evidence for the positive effects of instructive guidance on performance and on learning of students who use web-based video tools during a short collaborative-design task in their history lesson. In an experiment with 16-year old learners (N = 148) working on a history topic, we compared two contrasting types of guidance for student teams’ collaboration processes (social-interaction-related vs. cognitive-task-related guidance). We also compared two types of advanced video tools. Both types of guidance and tools were aimed at supporting students’ active, meaningful learning and critical analysis of a historical newsreel. Results indicated that social-interaction-related guidance was more effective in terms of learning outcomes (e.g., the students’ history skills) than cognitive-task-related guidance. The different tools did not yield consistent results. The implications of these findings are discussed.

Video is one of the most popular forms of educational media across the curriculum and plays an increasingly important role in classroom learning (cf. The New Media Consortium 2008). However, effective video usage in the classroom does not proceed without challenges. Research has shown: If videos are presented in whole-class models and passively watched by students they tend not to encourage the same level of reflective-learning activities as printed texts do (e.g., Salomon 1984). Consequently, to be effective for learning, video usage in class must extend beyond classic teacher-centered presentation approaches. It should foster student activities instead: Creative learning in task contexts that incorporate collaborative knowledge construction in small groups (e.g., Goldman 2004), joint observation and inquiry (Smith and Reiser 2005), and the understanding of complexity (Spiro et al. 2007). How can such processes be supported in a real classroom setting?

From a technical perspective, active student learning can be supported by advanced video tools with specific affordances (e.g., zooms, hyperlinks) that encourage learners to relate visual information to other instructional materials, or arrange video sequences for further group discussion, analysis and joint reflection (e.g., Goldman et al. 2007; Pea et al. 2004; Zahn et al. 2005). From a social-constructivist perspective, suitable tasks and instructive guidance of collaborative processes framing the use of video tools can help students make productive use of specific technology affordances for learning. Yet systematic research addressing video as socio-cognitive tool for collaborative learning is very scarce (Schwartz and Hartmann 2007). Further inquiries are needed to dig deeper into the complex interplay of tool affordances, task demands, social interactions and learning outcomes in complex learning situations. With this article we aim to contribute to understanding this interplay by focusing on instructive guidance as a possible factor for learning with collaborative design tasks incorporating video tools in the history classroom.

The technology perspective: Video tools for collaborative learning in the classroom

Research in the field of computer-supported collaborative learning has provided ample evidence on how technology affordances can support students’ learning in general (e.g., Roschelle 1992; Roschelle and Teasley 1995; Suthers and Hundhausen 2003; Suthers 2006), and specifically for using digital video technologies to support a variety of socio-cognitive functions. Early research works investigated the educational value of films in arts education and found that filmic coding elements such as zooming in can facilitate individual students’ mastery of mental skills necessary to understand art works (Salomon 1979). Another way of using video was suggested by Spiro et al. (1992) who studied hypermedia technology affordances as support for multi-thematic exploration and cognitive flexibility in history and language arts education. In a similar cognitive-constructivist framework for the use of video in the social science classroom, video analysis activities with video tools have been investigated as supporting perspectivity and critical analysis of video content (Goldman 2004; Goldman et al. 2007).

Recent approaches have turned to comprehensively investigating video tools used for complex design tasks (similar to the learning through design approach, e.g., Kafai and Resnick 1996). Schwartz and Hartmann (2007) connect “putting video in multimedia context” into a “space of learning” for the use of video in social studies (also see Pea 1991). Their students create their own multimedia documents or arrange video contents in order to learn by explaining. Zahn et al. (2010a) have employed specific video tool affordances in design tasks to support history learning and critical analysis of historical documents. Starting from the concept of guided noticing and web-based video (WebDIVER ^TM) developed earlier by Pea et al. (2004; Pea 2006) for joint visual analysis and reflection, they theoretically outline how different advanced video tools (selective and integrative tools) differ substantially in their affordances and socio-cognitive functions for learning (Zahn et al. 2005, for a summary see Fig. 1). In experiments, they have tested whether and how different video tools influence collaborative epistemic activities (grounding, negotiation, comparison and interpretation processes) for students using those video tools during history learning. Results from these studies show that the affordances of specific video tools can better support learners’ interactions to make them more productive compared to those performed with simple technological solutions, resulting in improved learning outcomes (e.g., Zahn et al. 2010a). A field study further revealed that these differences persist in the real history classroom with 16-year old students (Zahn et al. 2010b).

Fig. 1

Summary of technology affordances and theoretical socio-cognitive functions of integrative hypervideo tools and selective video tools

Fig. 2

a and b Graphical user interfaces of the video tools used in the study: (a) selective video tool WebDIVER^TM (http://diver.stanford.edu), (b) integrative hypervideo tool Asterpix (http://www.asterpix.com/), no longer available

However, caution is still warranted in expecting these initial results to immediately apply to any classroom situation, for several reasons: First, the results are still limited to specific tools. Further systematic comparisons between different advanced video tool affordances (as summarized in Fig. 1) remain to be performed. Second, the middle school students investigated in the field exhibited insufficient collaborative design strategies, in particular, their planning activities were wanting (Zahn et al. 2010b). In line with this issue, there is further evidence showing suboptimal video use strategies during individual history learning (e.g., Merkt et al. 2011). Studies investigating how to guide students in order to optimize their video use strategies still have to be conducted.

Taken together, the effectiveness of collaborative design tasks with video tools for student learning at school is still a controversial issue. From earlier debates on influences of media and instructional methods on learning (e.g., Clark 1983, 1994, 2009; Kozma 1991, 1994) we know that learning cannot be expected to happen as a consequence of receiving information from video media, but occurs as a consequence of an “… active, constructive process whereby the learner strategically manages the available cognitive resources to create new knowledge by extracting information from the environment and integrating it with information already stored in memory” (Kozma 1991, p. 179). Also, from a recent discussion on constructivist-learning approaches (e.g., Kirschner et al. 2006; Hmelo-Silver et al. 2007), we learn that students need guidance allowing them to learn collaboratively in complex domains (Hmelo-Silver et al. 2007). CSCL research on constructivist learning has repeatedly shown, that collaborating students need support—in organizing, planning and conducting scientific inquiries (Edelson et al. 1999), in scientific argumentation, in accomplishing long-term scientific design projects (Kolodner et al. 2003)-and that we cannot ignore group dynamics in a classroom and their possible influences on productivity in student teams (e.g., Cohen 1994).

From these considerations the research question arises: how can video tools be utilized under favorable instructive conditions to support learning through collaborative design in class? Knowing that students need guidance does not yet answer the question of how to provide effective support. As a key to establishing more detailed answers to this guidance problem, and to finding strong solutions both technical and instructional, we argue for a detailed investigation on the origins of the problems students might face during collaborative design with video tools in class.

The socio-constructivist perspective: Why even smart groups can fail

Two major sources of problems can hinder productive learning through collaborative design with video tools in class (cf. Zahn et al. 2002): the complexity of design (cognitive task) and the complexity of collaboration in design (social interaction). Both types of potential problems may overburden student teams, leading to cognitive disorientation or superficial task performance, with the consequence of impeding learning successes. We will elaborate on these potential sources of problems below, and extrapolate two reasonable solutions to balance instructive guidance.

The complexity of the task: Cognitive demands of designing with video tools

Design tasks generally consist of creating and structuring content for an anticipated audience according to the aesthetic standards of the media involved. They include the setting of design goals and complex processes of knowledge transformation, as proposed earlier by related cognitive research (e.g., Bereiter and Scardamalia 1987; Goel and Pirolli 1992; Hayes 1996). Cognitive psychology has related writing and design acts to complex problem solving: Just as writing is a special case of problem solving with the rhetorical goal of creating a coherent text for a specific audience (Hayes and Flower 1980), designing is a special case of solving an ill-structured problem with the goal of designing usable visual or physical artifacts for others (Goel and Pirolli 1992). Understood in this way, the conceptual problem space of visual design is a very active one, full of uncertainties and open dimensions of design choices. Inexperienced or less knowledgeable students may experience problems of being overwhelmed by the task, and subsequently experience difficulties in learning.

The complexity of collaboration in design tasks: Socio-cognitive demands of designing with video tools

Collaborative design is an iterative process “…of actively communicating and working together in order to jointly establish design goals, search through design problem spaces, determine design constraints and construct a design solution” (Lahti et al. 2004, p. 351). Correspondingly, design activities relate to the levels of the design problem and group cooperation. When students in design tasks use complex and sometimes unfamiliar digital tools (in our case, video tools), they need to coordinate their collaboration by establishing a social problem space that is distributed over the cognitive systems of at least two people and a digital artifact. Based on this shared context, they negotiate their choices of design goals and their understanding of content, task schemas, genre knowledge, and task relevant strategies (as in collaborative writing, e.g., Lowry et al. 2004). In sum, collaborative design includes the management of both task interdependencies and the coordination of the multiple perspectives of the collaborators (Détienne 2006). Thus, the success of collaborative learning depends upon the social activities of organizing teamwork. These dependencies may create new coordination problems of a social nature that are universal in “distributed cognitive systems” involving multiple agents (Streeck et al. 2011). Problematic alignments in communication and social interaction may impede learning: Barron (2003) in her ground-breaking work on “why smart groups fail” analyzed in great detail why not all student groups engage in productive knowledge-building conversations.

Guiding student teams in learning through collaborative design—a challenge

How can these origins of potential problems in learning with collaborative design tasks be tackled with instructive guidance? We propose two aspects as central when developing instructive guidance: The first is guidance relating to the cognitive demands of design, which should provide adequate task schemas for success in design problem solving by student learners. The second is guidance relating to the socio-cognitive demands of collaborative design, which should support effective social interactions during group coordination and communication in design. This distinction is consistent with other CSCL research studies, such as Fischer et al.’s (2002) conceptions of content-specific vs. content-unspecific aspects of instructional support, or Weinberger et al.’s (2005) epistemic vs. social scripts for learning groups.

Instructive guidance emphasizing the cognitive demands of design can be based on the Hayes and Flower (1986) model (Lehrer et al. 1994), and Bereiter and Scardamalia’s (1987) writing approach (Stahl et al. 2006). We refer to this type of support as “cognitive task-related guidance”. In a complementary fashion, guidance emphasizing social interactions focuses on pro-social behaviors like coherent communication, partner responsiveness and management of cooperation as suggested by small group research (Nastasi and Clements 1991; O’Donnell and O’Kelly 1994), including aspects of team formation, socializing, coordination monitoring, and reflecting on team processes and outcomes. We refer to this type of support as “social interaction-related guidance”. There is a large body of research on small student group productivity that we cannot comprehensively review here (e.g., Cohen 1994; O’Donnell and O’Kelly 1994). This research reveals the cognitive and social-emotional benefits of learning in groups or teams (e.g., enhanced academic achievement, motivation, cf. Nastasi and Clements 1991; meaning-making, negotiating meanings, grounding, cf. Stahl 2006). This research also reveals that successful learning depends centrally on group dynamics within student teams who function more effectively if they discuss and reflect on their own group processes (Cohen 1994; Nastasi and Clements 1991; Webb and Palincsar 1996).

So we ask whether it would further improve learning outcomes if we simply added cognitive task-related and social interaction-related instructive guidance in our case of learning through design with video tools in the classroom. Yet if we guide students so thoroughly, we run another risk: Instead of being under constrained by minimal guidance (the problem of discovery learning), students could feel extremely restricted by too much guidance. Students could feel overwhelmed by extensive instructions on design and social interactions, or become bored by elaborate accounts of what to do and not to do and how to do it before they really start doing anything. And at the extremes, students could end up becoming insecure and dependent on instructional support for their performance. This situation would, in turn, impede students’ creativity and self-determined learning, and finally—as a backfire effect—could run contrary to the educational goals of an authentic design task altogether. Similar critiques have been levelled about scripted, or “cookbook” science labs that do not sufficiently foster building scientific understanding through inquiry processes, argumentation, and other practices of science (e.g., Hofstein and Lunetta 2004; Monteyne and Cracolice 2004)—and about “over scripting” computer-supported collaborative learning (e.g., Dillenbourg 2002).

Hence guidance in collaborative design must be carefully balanced for students, and should tackle only those aspects of collaborative design where guidance is really needed. Empirical results from systematic comparative studies may yield clues for how to best establish such a desirable balance of guidance. But the few comparison studies to date (e.g., Weinberger et al. 2005) do not specifically address the affordances of video tools. Our empirical work aims to meet this research need. More specifically, we ask the following research questions:

• 1) Does social interaction related guidance or cognitive task-related guidance lead to better performance and learning when students perform design tasks in class?

• 2) Do video tools with different affordances lead to differences in performance and learning when students accomplish such design tasks in class?

• 3) Do effects of guidance and tool interact?

• 4) Which differences in collaborative processes can explain possible differences in performance and learning?

To answer these research questions we conducted an experiment on history learning and historical skills development by using video tools in a design task. The study is part of a larger research program on using video tools for history education and we accordingly apply a well-proven experimental setting from prior research.

Experimental study

In the present study, we compare two forms of guidance (factor 1) using two different types of video tools (factor 2) to examine their influence on student performance and learning. We also explore the role that specific collaborative processes in student teams play as possible mediators.

Based on related research by Barron (2003) on learning in student groups and by Weinberger et al. (2005) on scripting of online peer discussions, we predict that for research questions one and four that social interaction-related guidance will lead to better performance and learning than cognitive task-related guidance, and that these benefits can be related to improvements in social interactions during collaboration. Based on distinctions between video tool affordances (Zahn et al. 2005), we predict for questions two and three that differences between video tools will be found, but the directionality remains to be discovered. The reason for the non-directional hypothesis is that both video tools in use—although different in their affordances as summarized in Fig. 1—are, nonetheless, advanced socio-cognitive tools and the limited research literature accounts for effectiveness in both cases (Zahn et al. 2005, 2010a), although direct comparisons between these two cases have not been made in prior studies.

Method

Participants

One hundred and forty eight students (68 dyads, four triads; 81 male, 65 female, two no answer) from four different German high schools located in Southwestern Germany participated in the study. Their mean age was M = 16.2 years (SD = 1.0). Prior to the study we obtained written consent from the students’ parents and the school administration. Not all data were available for the whole sample; therefore the Ns vary across analyses (see Tables 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10). Due to technical problems, video-recorded interactions of 18 student teams were lost. For two teams among these plus another team design products were not available also due to technical reasons. Independent sample T-tests comparing those teams with all data available with those where the respective data was lost with regard to pre- and posttest scores, or transfer task performance, showed no significant differences for design performance, ps ≥ .32, and video data, ps ≥ .15, respectively.

Table 1 Coding scheme and examples of the coding procedure tapping students teams’ interactions during the design planning phase (Step 2 in the experimental procedure)

Table 2 Coding scheme and examples of the coding procedure tapping students teams’ interactions during the design action phase (Step 3 in the experimental procedure)

Table 3 Coding scheme and examples of the coding procedure tapping task relevant communication content during the design action phase (Step 3 in the Experimental Procedure)

Table 4 Means (M) and Standard Deviations (SD) of students’ choices in the question tapping their understanding of the task (treatment check)

Table 5 Means (M) and Standard Deviations (SD) for the multiple choice test tapping history knowledge acquisition and indicators of the transfer task tapping history skills acquisition

Table 6 Means (M) and Standard Deviations (SD) for the quality indicators of design products (Team performance in design)

Table 7 Means (M) and Standard Deviations (SD) for coding of students’ answers to the “Next steps” question (Team performance in design)

Table 8 Means (M) and Standard Deviations (SD) for the absolute time devoted to design planning (Step 2 in the experimental procedure) and percentages of time devoted to sub-categories of design planning (Collaboration processes)

Table 9 Means (M) and Standard Deviations (SD) for absolute time devoted to design action (Step 3 in the experimental procedure) and percentages of time devoted to sub-categories of design action (Collaboration processes)

Table 10 Means (M) and Standard Deviations (SD) for the percentages of talking time devoted to different contents during design action (Collaborative processes)

Experimental design

The study was conducted in a computer classroom set up at our research institute. Classes accompanied by their respective teachers came to the institute on regular school days as part of their regular history curriculum. Upon arrival they were randomly grouped into dyadic teams and randomly assigned to one of the four experimental conditions in a 2 × 2 study plan. The first factor Guidance (social interaction-related vs. cognitive task-related guidance) determined which type of instructive guidance was provided to support the collaborative accomplishment of a visual design task: guidance either emphasizing the cognitive aspects of the design task (e.g., setting a design goal, planning a design concept, tailoring information for an audience), or guidance focusing on well-functioning group collaboration (e.g., developing cooperative and pro-social norms for discourse practice). The second factor Video Tool determined whether the students worked with a selective video tool (WebDIVER, Pea et al. 2004) or an integrative hypervideo tool (Asterpix) as their design tools (see Fig. 2a and b): With the selective tool, learners’ cognitive/collaborative analysis is heightened by their ability to zoom into and out of digital video sequences, and arrange digital video sequences for discussion and reflection. With the integrative hypervideo tool, the collaborative ability to insert new knowledge artifacts into an existing digital video is heightened by hyperlinks relating visual information to other materials. All other circumstances were kept constant across conditions.

Learning task and learning goals

For the purpose of the experiment we employed a learning task that we had previously developed to study computer supported history learning with digital video tools in the classroom (e.g., Zahn et al. 2010b). In this task, students are asked to work on a newsreel about the Berlin blockade in 1948, so that it can be published, e.g., on a website of a virtual history museum. They were asked to analyze and comment on the newsreel so that future visitors of the website could develop a good understanding of both the content and the style of the newsreel as a propaganda instrument. To accomplish this design task, the students could use a collaborative video tool. Designing visual content for a web page of a virtual history museum provides students with an activity framework for comparison and re-organization of knowledge. The learning goal—and a special challenge for the students—is thereby to understand that the newsreel is not only “showing” a history topic (Berlin 1948), but that the newsreel itself is a history topic (i.e., a newsreel as an historical means for propaganda). This goal is aligned with criteria for the use of audiovisual and film sources in history education in German school education (Krammer 2006; Schreiber 2007) derived from Schreiber’s (2008) competence-structure model for historical thinking. The model specifically emphasizes skills to apply historical methods as an important goal in history education, precisely, the skill to “de-construct historical narrations” such as a text or film source. Skilled students—according to the model—are able to analyze historical films by interpreting their surface features (filmic codes and style) and their deeper structure in the respective historical context (content, target audience, message, author’s intentions). Such analytical abilities are widely accepted beyond the German educational system as basic history skills (e.g. Lorence 1983). We therefore refer to them as a “history skill” in our Measures and Results sections below.

Materials and tools

For the design task a video was selected that belongs to the pool of designated materials for history lessons provided by German media centers for teaching, which covers a topic from the 10th grade curriculum: post-war Germany and propaganda. The video used in the visual design task is a digitized version of an historical newsreel originally produced by the Allied forces (US/Great Britain) and shown to the German public during the Berlin blockade in 1948. It covers news information about the airlift established in 1948 by the Allied forces when Russia tried to cut off Berlin from traffic of goods. It consists of 95 single pictures and lasts 5 min. The video used in the skills transfer task measuring history skills is a modern 65-second TV-Clip by the German Green Party (Buendnis 90/Die Gruenen) from the 2006 nationwide election in Germany. This video was selected because analyzing ads like this is also a topic in 10th grade curriculum and the ad itself was of high quality. The texts used in the experiment contain 350–1500 words each. The content of the texts provides detailed information on three sub-topics: accounts of the historical context of Berlin in post-war Germany, information on media history and newsreels in post-World War II Germany, and a short introduction on film theory. Guidance was implemented in text-based form within the computer environment used for general task instruction. The texts differed between conditions in their descriptions of how one should best proceed to solve the given design task. The video tool used for computer-supported learning in the visual design task was either WebDIVER (see Fig. 2a) or Asterpix (see Fig. 2b). WebDIVER is one of the software programs developed in the DIVER Project (http://diver.stanford.edu) at Stanford University. Asterpix is a commercially available hypervideo tool. It is based on the idea of enabling users to select areas of interest and place graphical hyperlinks into a source video.

With the functions offered by WebDIVER, users can select either a temporal segment or a spatio-temporal sub-region of a video by mouse-controlling a rectangular selection frame (acting like a camera viewfinder) to “pan” and/or “zoom” into view only that subpart of a video that they wish to feature, and then interpretively annotate their selection via a web interface. Each movie clip and its associated annotations are represented in a panel, and a remix of the video clips and annotations can be played. Asterpix was a Web 2.0 tool (http://www.asterpix.com/, no longer available) with functions based on the hypervideo idea: Users could isolate sensitive regions within video materials and add links to other web resources, text commentaries, or pictures. The links could further be discussed by means of an integrated e-communication tool. Thus, users could include their own annotations and knowledge in a video and share them with others in a group or community (cf. Zahn et al. 2005). The socio-cognitive functions of the tools are summarized in Fig. 1.

Experimental procedure

A week before the students came to our lab, they filled in questionnaires that assessed their prior knowledge and other control variables (participants’ age, prior experience with computers in general and video software in particular, their history grades, or their dispositional interest in history). The experimental procedure in the lab lasted one and a half hours for all students and consisted of the following steps:

• Step 1 (preparation phase): The students read the overall instructions, which varied between conditions with respect to the Guidance factor. Then they read the history/media texts, and watched the video showing the historical Berlin-Blockade newsreel from 1948. They briefly practiced the use of the video tools to establish familiarity. The video tools varied between conditions according to the factor Video Tool.

• Steps 2 and 3 (collaborative design phase): In Step 2 (design planning), the student teams were asked to write down the content they would like to cover in their design products. Additionally, teams in the social interaction-related guidance condition were asked to develop social and cooperative norms for their design work by briefly writing down the communication rules they wanted to follow during task work and to consider the structure of the task with regard to a possible division of labor. Those students in the cognitive task-related guidance condition were asked to develop design norms by briefly writing their envisioned audience and which effects they wanted to produce with their adaptation of the source video. In Step 3 (design action), the student teams designed their products using the video tool of the condition that had been assigned to them.

• Steps 4, 5 and 6 (test phase): In Step 4, the students rated the quality of their own products and their teamwork. The students’ learning outcomes were measured in terms of history content knowledge and skills acquisition by a multiple-choice test and a transfer task assessing basic history skills (see below). Both self-assessment questionnaires and knowledge or skills tests were completed individually. Participants were thanked, released, and went back to their schools with their teachers. During the whole procedure, the teachers were present but not involved in the experimental procedures. The experimenter and research assistants were available for any questions or technology problems. To control for possible additional information provided when assisting students we coded this help seeking from experimenter or assistants separately in the videotaped interactions (see Tables 1, 2, 3).

Measures

Prior knowledge

To assess prior background history knowledge, and prior computer expertise or expertise in film and media production, a pre-questionnaire (self-assessment) and a multiple choice knowledge test were administered.

Treatment check

We asked participants to complete a recognition task on the instructions content to appraise whether the treatments were conceived of by participants as intended. Participants were asked to select a maximum of three alternatives from six statements repeating the task’s characteristics. To check whether participants had perceived the focus of the respective instruction, we presented three interaction-related task goals, e.g., “one of the most important aspects of the learning unit was good communication” and three design-related goals, e.g., “one of the most important aspects of the learning unit was to design for a target audience”. The two item groups showed convergent and discriminant validity, and were thus aggregated into two indicators.

Acquisition of history knowledge and history skills

To assess possible treatment effects on learning outcomes, a post-test was administered individually, consisting of (1) a multiple choice test measuring historical topic knowledge (topic: Berlin 1948) and (2) a transfer task tapping the skill of applying historical research methods (analyzing a historical film source). The post-test was adapted to educational standards in German schools (Ministry of Education, BadenWürttemberg, 2004) and based on the competence-structure model by Schreiber (2007), as well as widely accepted notions on basic history skills (see section on Learning Goals). The test has successfully been used in previous experiments (e.g., Zahn et al. 2010a), so we applied it to this study. The multiple choice test consists of eight items, with one or more correct answers per item (sample item: “At the beginning of 1946 Germany is… a) …a unified nation, b) …divided into four sectors, c)… divided into an Eastern and a Western part, d) …divided into 16 Länder”). The theoretical maximum score of this test was 13 points, and it had a relatively low internal consistency, Kuder-Richardson Formula for dichotomous items=0.61. Due to the various aspects of history content tapped by the test, however, we considered it suitable.

The transfer task assessing history skills acquisition assesses the students’ ability to analyze a historical video source by interpretation of its surface features (filmic codes and style), and by interpretation within the historical context (content, target audience, message, author’s intentions). The test was adapted from Schreiber’s (2007) suggestions and consisted of questions relating to a political TV-ad from the 2006 nationwide German government elections (duration 1 min). Two short sequences from the ad were given to the students (durations 4 and 9 seconds) and they were asked to answer the following open-ended questions for each sequence: Which film techniques were used in this sequence? What were the intentions for using them? Students were also asked to answer two open-ended questions with regard to the whole video ad: “Please characterize the target group of the TV-ad”; and “Please describe the main message of the TV-ad”. Two raters independently coded the students’ answers to the transfer task questions. For the coding procedure, coders considered a pre-defined default analysis solution created by an expert (first author). The solution comprised exemplary target groups of the TV-ad, characteristics of those target groups, film techniques used in the TV-ad (stylistic surface features such as camera, music, montage), as well as examples for correct deeper interpretations of such elements (e.g., close-up of a person’s face aims at creating emotional involvement). Based on this example, raters counted the numbers of named “target groups”, “target group characteristics”, “style features” and their “interpretations” that were plausible before the background of the default solution. The “elaborateness of the answers” was also rated on a 3-point Likert scale (1=simple, 3=elaborate). In sum, we coded the students’ answers for five indicators for the skills of students to analyze a historic film source (as an assessment of history skills acquisition). With regard to inter-rater reliability we used Cronbach’s alpha when data was assessed at the interval level and Cohen’s Kappa when assessed at the nominal level according to Asendorpf and Wallbott (1979). Accordingly, we used the aggregated rater codes for analyses when the data had at least interval level (counted aspects, Likert-Skale ratings) and used the codes of one rater when data were on the nominal level (exclusive categories), while checking whether results were the same for both raters. Inter-rater reliability was satisfactory for the number of target groups, Cronbach’s α = .77, target group characteristics, Cronbach’s α = .77, the number of style features, Cronbach’s α ≥ .91, and the elaborateness rating, Cronbach’s α ≥ .76. However, rater agreement for the number of interpretations of these style features was very low, Cronbach’s α = .10. Closer analyses revealed that the raters differed greatly with regard to how strictly they applied the coding scheme. For further analyses we decided to only use the coding of the rater who had applied the coding scheme in a very strict way.

Team performance in design

We assessed team performance in design by analysis of the products that the student teams had created during their task according to a method developed in prior studies (Zahn et al. 2010a): We analyzed the panels created with WebDIVER and the hyperlinked comments created with Asterpix. Data were obtained by coding and counting “video selections or hyperlinks with comments”, as well as “style features”, and “interpretations” named in the comments. In sum, we received quantitative (video selections/hyperlinks) and qualitative (style features and interpretations) indicators for team performance in design. Inter-rater reliability for style features and interpretations were satisfactory, Cronbach’s α ≥ .94.

As further indicators of team performance in design, we asked the teams to name the “next steps they would have performed, if they had been given more time to accomplish the task”. This was due to the tight timetable of our experimental procedure. From the students’ open-ended answers to these questions, two trained raters coded and counted the number of “planned content” items and “next steps” items, respectively. Raters also rated the elaborateness of these answers on a 3-point Likert scale (1=simple, 3=elaborate). Rater agreement was satisfactory, Cronbach’s α ≥ .81, so data from the two raters were aggregated.

Collaboration processes

As a record of the collaboration processes, student team interactions were captured with a webcam and a screen recorder. From the video data, design activities and communication contents were extracted. For the analysis we used a coding system developed from results of prior research (Zahn et al. 2010a, b) containing indicators for successful design problem solving, possible design problems and conversation content quality.

Design problem solving, possible design problems

We coded the talking times in Step 2 in the experimental procedure as “design planning” and in Step 3 in the experimental procedure as “design action”. Design planning consisted of the following categories: “task-related”, “collaboration-related” and “procedure-related” planning (see Table 1 for details and examples). Design action consisted of “task work”, “evaluation”, “technical issues”, “off task”, and “other problems or questions”, which were further refined into sub-categories (see Table 2 for sub-categories, details and examples). We computed the percentages of time devoted to these categories related to the overall talking times. Communication content: We refined our analysis by also coding the talk contents during design activities during the design action phase. We coded the amount of time students took within this category for talking about “the newsreel video ”, “the history topic” and “design” (see Table 3). We then computed the percentages of time devoted to these sub-categories relative to the overall talking time during the students’ “design action”.

All coding samples are summarized in Tables 1, 2 and 3. For computing rater agreement 20 % of the videos were coded by a second rater (cf. Trickett and Trafton 2009) and rater agreement was on average satisfactory in all categories, median of Cohen’s κ = .64.

Results

We first present results substantiating the comparability of our experimental conditions, and then results obtained from quantitative analyses of the products and the from the post-tests. Due to assumed interdependence of students working in one team, we determined dyads as the unit of analysis and used data aggregated within teams (cf. Kenny et al. 2006). The level of significance for all analyses was set to 0.05.

Comparability of the conditions

A 2 × 2 between subjects ANOVA with the factors Guidance and Video Tool revealed no significant differences between the conditions concerning participants’ age, prior experience with computers in general and video software in particular, their history grade, or their dispositional interest in history (all p > .10). The student teams also did not differ significantly between conditions concerning within-group composition related to age, gender, prior knowledge, history grade, or historical interest (all p > .10). In addition, student teams did not differ in their appraisal of the task, the appraisal of their teamwork or the amount of invested mental effort during task work (all p > .10), indicating that the participants’ overall positive attitudes towards task and performance were similarly high in the four conditions. In sum, the conditions can be considered comparable. However, historical knowledge showed a marginally significant interaction, F (1, 68) = 3.85, p = .05, partial η² = .05, showing that for students working with WebDIVER, those participating in the cognitive design-related guidance condition scored higher on the pretest (M = 10.23, SD = 2.55) than students in the social interaction-related condition (M = 8.22, SD = 2.20), t (34) = 2.53, p = .02. For students working with Asterpix, there were no significant differences. ANOVAs reported here were also run as ANCOVAs controlling for prior knowledge or interest in history when these covariates were correlated with the respective dependent variables, and are reported when they show different results.

Treatment check

The means and standard deviations of students’ choices in the question tapping their understanding of the task are shown in Table 4. An ANOVA revealed no significant difference between conditions concerning their scores in “design task” characteristics, F < 1, ns, but a significant difference for the “social task” characteristics for the factor Guidance, F(1, 68) = 15.51, p < .001, partial η² = .19. More “social task” items were chosen by students who had received social interaction-related guidance than by students who had received cognitive task-related guidance. Our text-based implementation of guidance by task instructions can thus be considered effective for eliciting the students’ awareness of the design problem in the intended way in all conditions—and the students’ increased awareness of the social demands of the collaborative design task in the social interaction-related conditions.

Acquisition of history knowledge and history skills

Scores on the individual multiple choice tests on knowledge about the history topic were aggregated for each dyad before analysis and revealed a total mean score M = 7.54 (SD = 2.46) out of 13 possible points (for other means and standard deviations see Table 5). We conducted a mixed 2 × 2 × 2 ANCOVA with the two between-subjects factors Guidance and Video Tool and the within-subjects factor Pre-Post-Test to control for the pre-test scores and to test for differences in the gain in history knowledge on the topic. The assumption of homogeneity of regression for conducting ANCOVAs was met, and prior history knowledge was correlated with the post-test score, r = .23, p = .05. The results showed a significant increase in history knowledge over time, F (1, 67) =34.80, p < .001, partial η² = .34. However, there were no significant differences between the conditions, F < 1, ns, and no significant interaction, F (1, 67) = 1.93, p = .17, indicating that the students in all conditions had developed a better understanding of the history topic.

The analysis of the transfer test results assessing acquisition of history skills revealed a total average of M = 1.31 (SD = 0.36) for “target groups”, M = 1.52 (SD = 0.52) for “target group characteristics”, M = 1.97 (SD = 0.74) for “style features”, M = 0.37 (SD = 0.23) for “interpretations” named in the answers of students, and M = 1.19 (SD = 0.47) for “elaborateness of the answer” (for all means and standard deviations see Table 5). ANOVAs revealed that three of the indicators were significantly higher in the answers from the conditions with social interaction-related guidance, than in the answers from conditions with cognitive task-related guidance: number of style features, F(1, 68) = 7.96, p = .01, partial η² = .11, number of interpretations, F(1, 68) = 4.36, p = .04, partial η² = 06, elaborateness of the answer, F(1, 68) = 4.11, p = .047, partial η² = .06. There was a signifficant main effect of the Video Tool factor for the mean number of target group characteristics, F(1, 68) = 4.38, p = .04, partial η² = .06. However, this main effect was qualified by a significant interaction, F(1, 68) = 4.32, p = .04, partial η² = .06., showing that only for social interaction-related guidance, students working with the selective video tool (WebDIVER) performed signifficantly better than students working with the integrative video tool (Asterpix), t (33) = 2.56, p = .02. Overall, effect sizes were of medium to large size. There were no further effects of the Video Tool factor, Fs < 1.1, ns, or any significant interactions, Fs < 1, ns. In sum, the learning outcomes in terms of history skills were better when social interaction was supported and with the selective video tool.

Team performance in design

The means and standard deviations of the scores concerning numbers of commented “video selections/hyperlinks”, “style features” and “interpretations” are presented in Table 6. ANOVAs revealed a significant main effect for the factor Guidance: The mean scores in all indicators were significantly higher for the products of student teams in the condition with social interaction-related guidance, than for those from student teams in the condition with cognitive task-related guidance, in terms of number of comments, F(1, 67) = 6.46, p = .01, partial η² = .09, number of style features, F(1, 67) = 4.78, p = .03, partial η² = .07, and number of interpretations, F(1, 67) = 4.63, p = .04, partial η² = .07. Hence, team performance in design was higher in the social interaction-related guidance conditions than in the other conditions. No further main or interaction effects were found.

The means and standard deviations of the numbers of different aspects that students indicated in answers to the “next step” question concerning their design work, and the means of the elaborateness rating of the answers, are shown in Table 7. 2 × 2 ANOVAs with the two between-factors Guidance and Video Tool yielded significant differences between the conditions with social-interaction related guidance and the conditions with cognitive task related guidance for both number of aspects F(1, 68) = 8.83, p = .004, partial η² = .12 and elaborateness of answers F(1, 68) = 4.66, p = .03, partial η² = .06. There were no further significant results, all F < 1. The students in the conditions with social-interaction related guidance indicated more items that they would have liked to include and gave more elaborate answers than the students in the conditions with cognitive task-related guidance.

Collaboration processes

Mean percentages of time and standard deviations for “design planning” and “design action” are summarized in Tables 8 and 9. 2 × 2 ANOVAs with the two between-factors Guidance and Video Tool yielded significant effects. Differences were found between the conditions with social-interaction related guidance and the conditions with cognitive task related guidance for all categories in “design planning”: Task-related planning F(1, 50) = 4.53, p = .04, partial η² = .08, collaboration-related planning F(1, 50) = 220.65, p < .001, partial η² = .82, procedure-related planning, F(1, 50) = 55.15, p < .001, partial η² = .52. Overall, the students in the conditions with social interaction-related guidance devoted more time to “design planning” than the students in the conditions with cognitive task-related guidance, F (1, 50) = 6.23, p = .02, partial η² = .11. A significant effect was found in the category “other problems or questions” in help-seeking from other teams F (1, 50) = 5.67, p = .02, partial η² = .10. The students in the conditions with social interaction-related guidance sought more help from other teams than the students in the conditions with cognitive task-related guidance. There was also a significant main effect for the Video Tool factor, F(1, 50) = 5.73, p = .02, partial η² = .10, indicating that students working with integrative video tool (Asterpix) spent significantly more time on “design planning”. Secondly, concerning the Video Tool factor, significant differences were found related to “design action” in the sub-categories: Watch newsreel video together F(1, 50) = 17.11, p < .001, partial η² = .26, and work on task together F(1, 50) = 26.98, p < .001., partial η² = .35. For the latter category we also found a significant interaction F(1, 52) = 10.49, p = .002, partial η² = .17. Bonferroni-adjusted post-hoc comparisons showed that students with cognitive task-related guidance worked more collaboratively on the task when working with Asterpix, p < .001, for students with social interaction-related guidance, however, there was no difference, p = .56. The differences between the conditions with the selective video tool (WebDIVER) and the integrative video tool (Asterpix) point in opposite directions: The students in the selective video tool conditions devoted more time to watching the newsreel video together than the students in the integrative video tool conditions. In contrast, the students in the integrative video tool conditions devoted more time to working on the task together than the students in the selective video tool conditions. No further significant effects were found as a result of this first step in the video analyses, all p ≥ .08.

For the analysis of the talk contents, the mean percentages and standard deviations of talking times devoted to the specific content categories “newsreel video”, “history topic” and “design” talk during “design action” are shown in Table 10. 2 × 2 ANOVAs with the two between-factors Guidance and Video Tool yielded significant differences between the conditions with social interaction-related guidance and the conditions with cognitive task- related guidance for newsreel video talk, F(1, 50) = 4.96, p = .03, partial η² = .09. The students in the conditions with social interaction-related guidance talked more about the newsreel video. Significant differences were also found between the conditions with the selective video tool (WebDIVER) and the integrative video tool (Asterpix) for history content talk F(1, 50) = 6.00, p = .02, partial η² = .11. The students from the conditions with the integrative video tool talked more about the history content than the students from the conditions with the selective video tool. No other effects yielded significance, all p ≥ .14.

Mediation analysis

To test whether the main effect of the factor Guidance can be explained by differences in the respective team interaction between the conditions we conducted mediation analyses using the process variables that were significantly higher in the SIG conditions as mediators, percentage of task-related planning, percentage of collaboration-related planning, and percentage of newsreel video talk . We followed the procedure proposed by Preacher and Hayes (2008) for estimating and comparing indirect effects of a mediator. This procedure estimates an unstandardized coefficient (b) for the indirect effect and tests its significance with a bootstrapping technique by estimating standard errors and confidence intervals. Analyses revealed that only the effect of social interaction-related guidance on the elaborateness of participants’ skills transfer test answers was significantly mediated by collaboration-related planning, b = .47, SE = .23, CI α = .05 [0.07; 0.96], rendering the direct effect on history skills transfer answer elaborateness, β = .30, t(55) = 2.28, p = .03, insignificant, β = −.21, t(55) = −0.71, p = .48. Furthermore, we found a significant indirect effect of social interaction-related guidance on the number of target groups named in the history skills transfer task mediated by their collaboration-related planning, b = .50, SE = .27, CI α = .05 [0.07; 1.11]. All other results of mediation analyses were not significant, with the confidence intervals including 0.

Discussion

Our experimental results provided evidence that contributes to answering the question of how to improve instructive guidance for student teams solving design tasks with the support of video tools, in this case, for acquiring history knowledge and skills. Results indicated that using either of the advanced video tools we offered was generally effective, but that differences in the types of instructive guidance we implemented (cognitive task-related vs. social interaction-related guidance) resulted in different collaborative processes and significantly different learning outcomes. First, the immediate products of the students’ teamwork were of better quality when students received social interaction-related guidance. Second, the scores of the students in a test assessing their historical skills (analysis of a historical film source) were also significantly higher with social interaction-related guidance. Concerning history knowledge about the topic (“Berlin blockade”), no differences and no trade-off effects in performance in a multiple-choice posttest emerged. Thus, the differences in design performance and the skills transfer test did not reflect sacrifice of any other learning outcome measures. Furthermore, this finding was not confined to a specific video tool used in our study: Results show that even given the conceptual differences of the video technologies (WebDIVER and Asterpix) described above, the overall benefits of supporting the challenges of coordination in the social problem space persist. We thus conclude that the students with social interaction-related guidance learned more than the students with cognitive task-related guidance. We further conjecture that, even given the different affordances for the two advanced video tools, social interaction-related guidance improved the quality of team interactions on a deeper content level. And this leads us to the question of how exactly that quality was improved.

To consider this question, we examined the results on our treatment check questions to tap into the students’ individual task understanding (answers to treatment check questions) and the student teams’ interactions (coding of collaboration processes). Concerning task understanding, we found that in the treatment check increased scores of the students in the social interaction-related guidance conditions concerning the social demands of the task, while scores concerning the cognitive demands of the task were equally high in all conditions. This result is interesting beyond the treatment check. It means that all students understood the design requirements of the task (consistent with the finding that all student teams worked successfully on the design tasks), but that they did not necessarily have a full understanding of the social interaction requirements (consistent with the finding that only the teams in the social interaction-related guidance conditions performed better). The guidance we provided in the social interaction-related conditions thus seems to have increased students’ awareness of the social demands of the collaborative design task. This was an added focus that did not come at the cost of understanding the design task. Due to this additional focus, student teams might have interacted in different ways and performed better as a consequence. If this were true, differences should be observable in the data on collaboration processes as possible mediating variables.

The student teams in the social interaction-related conditions devoted more time to design planning, and the differences were significant for “collaboration-related planning” and “task-related planning activities”. In these categories we coded conversations in a team relating to clarification of the task itself and its goals, including learning goals (e.g., “what are we supposed to do?”). We may infer that social interaction-related guidance (e.g., establishing communication rules and roles in the teams) thereby led students to more thinking about learning goals too, and consequently to a focus on relevant issues during design. This inference is however, only partly substantiated by the results of the mediation analysis. Collaboration-related planning was a significant mediator for students’ performance in some aspects of the transfer task measuring historical skills. Other differences in students’ interaction did not mediate the effect of social-interaction related guidance on learning outcomes. So we are cautious in offering an explanation for why the teams in social interaction related guidance condition performed better during the design action phase and acquired better history skills. For example, we did not measure other indicators of interaction quality (such as responsiveness to the partner) that may have led students to become still more focused on joint design activities, with a consequently better outcome.

Nevertheless, our findings provide an initial answer to the question of how instructive guidance can be balanced for middle-school students working with video tools in order to support skill-intensive collaboration processes in design tasks: It can be balanced by putting a focus on the social demands of a design task, instead of unnecessarily repeating its cognitive task-related design aspects. This finding is consistent with related research: First, studies on group learning in the classroom indicate that a social interaction focus is quite necessary for better group performance (Barron 2003; Webb and Palincsar 1996). Second, research studies on uses of collaboration scripts (e.g., Weinberger et al. 2010) reveal that social scripts work better than epistemic scripts. Weinberger et al. (2005) found in experiments that social scripts can be beneficial in online peer discussions with respect to individual knowledge acquisition, whereas epistemic scripts do not to lead to the expected effects. Despite the similarities, our contribution provides new results in revealing this, first, in the context of social interaction related instructive guidance instead of social scripting of specific interaction patterns by prompts, and second, in the context of video tool based design tasks for learning in history education in the middle school classroom. Weinberger et al.’s research examined online peer discussion environments (text-based and video conferencing systems) at a university level.

Our results on video tool effects and possible interactions are less clear. We found no significant effects of the video tools on learning outcomes and design products, which we consider surprising. We found no effects in task understanding either, which we consider unsurprising. Concerning effects on team interactions, results yielded significance in “design planning” and “design action”: student teams working with the integrative tool (Asterpix) devoted more time to “planning” and to “working on the task together,” whereas student teams working with the selective video tool (WebDIVER) devoted more time to “watching the film together”. An interaction effect yielded significance, indicating that during the design action phase, the student teams in the cognitive task-related guidance condition worked more collaboratively with the integrative video tool than the student teams working with the selective video tool. For the social interaction-related guidance condition, results from using the different video tools did not differ. The positive tool effects of the integrative video tool on team interactions in the student teams confirm the idea of mediating tool functions (implicit guidance of collaboration by digital tools) and converge with related empirical research described earlier. They show that such effects do exist for video tools, complementing earlier comparative research where positive mediating functions of an advanced video tool (the selective video tool) as compared to a simple control condition were found (Zahn et al. 2010a, b). Yet, the results are confined to processes of team interactions. There were no effects on team performance and learning outcomes - against our expectations. So, the video tool differences in effects can be considered rather weak. We suggest an explanation of these weak effects in terms of two considerations. The first is the fact that two similarly advanced tools were used, and the second is that strong and consistent impacts of instructive social guidance were found. Students might have used the tools in very goal-oriented ways when they received social interaction-related guidance. So the video tools, since they were similarly advanced (relative to a simple video player and word processor), did not make a great difference. From the interaction effect we found, we additionally infer that positive video tool effects on collaboration only surfaced when social interaction guidance was sub-optimal, as was the case in the cognitive task-related guidance conditions. Maybe only in this task-related guidance condition could a positive tool effect of the integrative video tool improve collaboration, and then it did. This is, however a tentative interpretation, which requires further research for appraisal.

When drawing scientific conclusions and implications for school practice from our overall results, we need to reflect on the following issues: In the study, we created a computer-supported experimental setting at our institute, to enable us to draw causal conclusions. Student experiences were limited to a short-time visual design task for a regular history lesson, which is quite different from larger scale, learning-by-design projects (cf. Lehrer et al. 1994) performed over several weeks with the necessary help and group support given by teachers. Thus, our results cannot at this time be generalized to large-scale, long-term projects. And although the setting was very similar to the students’ regular classroom situations in their schools (real classes and teachers, regular curriculum topic, regular lessons), it was not their real classroom. We set up the study in our research institute, not at the students’ real school. We could thereby not pick a random sample from a defined population and the students may not be typical of other 16-year olds. So, further field studies will be necessary to confirm our results. However, we have compared our results from this experiment with the results from an earlier field study in a real classroom situation with a comparable sample of students, and with the same short task and test items. Results revealed general gains in topic knowledge (pre- to post-tests) similar to those obtained in the field study. No indication of influences of the artificial experimental situation (positive or negative) were found.

Hence, from our findings, we conclude that students of the age group investigated here (16-year-olds) could profit from guidance for effective social interaction by establishing a social problem space. This might be the case because students seem to be less able to activate effective ways of social interaction in a team from their everyday school experiences. This interpretation is consistent with earlier research on hypertext design for learning claiming that a focus on the design process is more important for learning than a strong product orientation (Bereiter 2002), and with the results reported above on social scripts (Weinberger et al. 2005, 2010). Can we therefore conclude that social-interaction related guidance is always better? From a practical perspective, this issue would be important for teachers if they could focus on social interaction processes in their guidance of students’ collaborative task work with video tools in real lessons. From the scientific perspective, our findings would contribute evidence about potential effects of media on learning by revealing causal mechanisms influencing cognitive and social processes when students interact with media (Kozma 1994). While promising, a strengthening of our conclusions concerning the advantages arising from the social interaction-related guidance examined here will require further scientific analysis across a broad range of collaborative learning environments.