1 Introduction

1.1 Historical perspective on collaborative learning

The idea of introducing collaborative learning approaches in the classroom has been proposed well before networked computers. Based on earlier research on social psychology (e.g. Alport 1924) researchers in the 60s in the US and England were committed during the Vietnam era to democratizing education by eliminating what were perceived then as socially destructive authoritarian social forms (Bruffee 1984). This ideal was also supported by experimental findings that confirmed advantages of group learning (e.g. Abercrombie 1964), as it was found through extensive experimentation in medical schools, that students acquired good medical judgement faster by working in small groups rather than working individually. It was the networked computer however that has given new potential to these old ideas. While the computer in the classroom has been seen originally with scepticism, as considered inhumane and anti-social, the networked computer has changed this view. The networked computer has new affordances: it maintains shared representations, it supports and substantiates communication in various forms, it allows access to external information sources. So it creates favourable conditions for active and collaborative learning. As a consequence, a new area of research and practice emerged, that of computer-supported collaborative learning (CSCL). In this new field, theories, research methods and tools have been produced in recent years (see Stahl et al. 2006, for a historical overview). One particular trend of research involves the design of technology that supports collaboration. Since early CSCL projects, however, the development of technologies was coupled with study and development of new pedagogies and restructuring of class practices. For instance, in one of the earliest examples, the Computer Supported Intentional Learning (CSILE) project (Scardamalia and Bereiter 1996), later Knowledge Forum, technologies were developed together with innovative pedagogies to restructure classrooms as knowledge-building communities. It was understood that technologies shape practices and are informed by use. As Stahl et al. (2006) observe: artefacts become tools “in the ways in which they are oriented to and made relevant by participants in directed practice”. So design of CSCL technologies need to be coupled with studies of these emergent practices. These studies follow either the dominant experimental paradigm (mostly in laboratory settings) or the ethnomethodological tradition in the classroom (Jeong and Hmelo Silver 2010).

1.2 Human support for CSCL

The technology developed in the CSCL field focused first on knowledge building through collaborative interaction or group meaning making (Stahl et al. 2006), so the emphasis was on providing adequate tools and representations for the learners. It was later on that the teacher drew the attention of the community and technology was devised specifically for teacher support. A thorough study on human support in CSCL is done by Lund (2004). In this work the following kinds of support were identified for the teacher: (a) Pedagogical support aiming at the students’ learning, (b) Social support focusing on the social relations between the students, for maintaining their motivation, (c) Interaction support, aiming at ensuring that students participate, are responsive to each other, without overlapping in their contributions, (d) Managerial support focusing on task design, completion, and monitoring and (e) Technical support aiming at detecting operational and technical difficulties. It should be observed however that this study assumes distributed and asynchronous setting, and it should not be taken for granted that it is also applicable for synchronous collaboration in the class. For the later case there is relatively limited research. Lakkala et al. (2005), reporting on teachers’ role in technology supported collective student inquiry studies in Finish schools, identified the great challenges facing the teachers in such settings: new technological tools, new pedagogical models, emphasis on collaboration instead of students individual achievements. These observations coincide with comments by Krajcik et al. (2000) that the teachers’ new role involves helping the students to become knowledgeable about content, skilled in using inquiry strategies, proficient in using technological tools, productive in collaborating with others. A term used to describe the teacher’s role is that of “orchestrator of learning activity” to express this responsibility to make students aware of what their goals are, how they should be interacting, potential technological concerns, and define the time-frame for the exercise (Prieto et al. 2011). This structure, that takes often the form of a script, is used to enhance the experience for learners by supporting collaboration and creating opportunities for the construction of knowledge (Kobbe et al. 2007). It is this new role of the teacher, the focus of the research reported in this paper. In particular, we examine various tools that have been proposed to support this new role, and report on the findings of a wide scale study of teachers using alternative tools. The emphasis on this paper is that of describing the social practices (Hakkarainen 2009) observed when teachers were faced with the daunting challenges of the technology enhanced collaborative class. Teacher support tools were identified as key priority in this research field by many researchers (e.g. Dillenbourg 2009), so the reported here research is hoped to contribute to better understanding of the teacher in this setting.

1.3 Technology enhanced CSCL class

Many technologies have been used for supporting teaching in the classroom. In a recent overview paper (Dillenbourg et al. 2011a, b) these were related to the usability and affordances of various traditional and innovative technologies. Traditional tools, like blackboards, overhead projectors, video and TV sets, teacher PCs connected to projectors, seem to be challenged by new tools, like interactive whiteboards, new awareness devices and one-to-one technologies for the students. In this setting, groupware applications are introduced that allow communication and sharing of resources, promoting active participation, collaboration, inquiry and knowledge building. For organization and sharing of material and activities, online Learning Management Systems have emerged, that include tools for synchronous and asynchronous collaboration (Lonn and Teasley 2009). Management of classes in technology enhanced classrooms necessitate Classroom Management Systems, which allow teaching staff to manage students’ computers, facilitating sharing of resources and communication and support of group activities, see as a typical example I-MINDS (Soh et al. 2008). In addition, student assessment tools have been used for processing of submitted student work, such as processing of multiple choice tests, or systems that allow real-time response (Crawford et al. 2008; Roschelle 2003), while Intelligent Tutors have been proposed (Crawford et al. 2008; Scheuer et al. 2012), to be embedded in learning management systems or class management systems.

In the specific case of computer supported group activities in the classroom, there is a new challenge to define appropriate tools to support the teacher. The students using their computers are often engaged in high-density activities in short time, demanding teacher attention and support. These activities of different groups take place in parallel, producing large volumes of potentially interesting data. The teacher, as coordinator of the class activity, should be able to monitor the progress of the different groups during the process.

Monitoring of individuals is not enough, since the new entities, the groups, appear, and they should be monitored and their progress evaluated and supported. Development of tools to support teacher is of high importance, and remains one of the issues that need to be addressed during the integration of CSCL research results into the classroom (Chan 2011).

In this paper we report on a study of teachers who supervise computer supported group activities in the classroom. This study in its main part followed the ethnomethodological approach (Stahl 2006), producing a descriptive account of teacher activity in a specific technology enhanced class. This was combined with a data driven grounded approach that attempted to discover patterns in the data, e.g. comparing patterns of teacher behaviour when a tool that supported different social planes, e.g. individual, group, class (Dillenbourg and Jermann 2010) was used. In more detail, the study involved the following phases: In a preliminary phase, the design of the study was informed by the findings of a literature review that identified common trends in existing tools. The need to support monitoring at different social levels was identified first. Other findings included recording of the activities of students and awareness mechanisms based on quantitative and qualitative indicators that represent the state of groups and the whole class, use of various representations of the activity, and experimental use of intelligent teacher assistants. Based on these first findings, a number of field studies were conducted next that involved teachers and students. In the first phase the teachers did not use special supervision tools, while in the second phase new tools were designed and used by the teachers. The major difference among the first and the second phase was that in the latter, awareness on the group state was provided to the teachers through various representations. This affected the observed teacher behaviour. In the final part, the findings are discussed with the perspective to improve our understanding of the orchestrator of collaborative class in order to support this role.

1.4 Research questions on teachers supervising group classes

Group activities mediated by computers produce rich digital traces that have become a valuable resource for research on collaborative learning. For this reason specially-created tools have been built for the collection and analysis of group interactions (Dillenbourg et al. 1995). A plausible hypothesis can be that if such data have already been effectively used for understanding phenomena of collaborative learning by researchers, similar representations could, under certain conditions, be used by teachers who supervise group activities, playing the role of teacher awareness support tools.

In order to test this hypothesis we need first to define the representations that can be produced from data of recorded group activities that are suitable for the teacher. These should be defined in terms of temporal availability (e.g. during or after the activity), the types of representations afforded (e.g., charts, animation, video, etc.), their capacity to support reasoning about the process, and the social level that they concern (e.g. individual student, group, or the whole class).

Assuming that the teacher has available a large volume of data generated during group activities in the classroom, we are going to investigate some key aspects relating to their relevance to teacher tasks.

  1. (a)

    We are going to study the typical teacher actions during supervision of group work within the classroom in order to understand interaction with various groupings formed during the process (Dillenbourg 2009). For example, does the teacher need to understand the activity of all groups or some of them and in what level of detail? Understanding of the progress of a single group, necessitates following the activity of all its members or some of them? How is awareness of class progress shaped and how is it maintained?

  2. (b)

    Further on, we need to identify time related requirements of the teacher. What information needs the teacher to have access to in timely manner in relation to the progress of the activity for individuals, groups and the class? What representations should be produced that can be understood by the teacher in the short available time? For example, does the teacher need to represent, in addition to the current state of group activity, how this has been produced?

  3. (c)

    In addition, we need to find out the level of detail of the available data that will be provided for each social level and what support may be provided for navigation, from overview information to more detailed one, in the light of time and other constraints in the classroom.

The context of this research is the following: The main objective is to study the role of the teacher supporting small group activities in a class. The members of groups, dyads in general,Footnote 1 collaborate through interconnected computers (one student per PC). Typically they are engaged in problem solving tasks that require dual shared space, one in which they jointly produce a diagrammatic representation (concept map, flow chart, argumentation graph or other form of diagram), and one for text based communication. The group activities have a time span of typical classes: i.e. a few minutes to a few hours.

In this context, we are going to focus on investigating the above stated questions in light of past experience. So the main issues to be studied are related to typical teacher actions and used representations, the time relevance of these representations and the provided navigation support. Next we briefly review the relevant research findings so far, as they are reported in studies of similar character.

2 Relevant research: Studies involving teachers in technology enhanced classes

In this section we review research related to supporting the teacher in computer mediated group class activities. It was found from early studies that effective collaborative activities need teacher guidance and support. (Guzdial et al. 1997; Lund 2004; Dimitracopoulou 2005). This was particularly evident when the students were distributed in multiple parallel groups whose activity evolved with different pace. It was found that the guidance requested by the students had to be delivered promptly and be explicit, relating to the problem solving activity, the technology and the collaboration (Guzdial et al. 1997; Lakkala et al. 2005; Lund 2004). It was proven difficult for the teacher to respond promptly to such requests, to identify the group with the direst need and to estimate quickly the kind of support needed. As already discussed, the early studies focussed mostly on a-synrhronous, distributed studies and it was not until recently that the co-located group activities drew the attention of the research community. A number of studies have focussed on experimental prototypes supporting collaborative learning, using mostly AI techniques. These were not always explicitly aiming at supporting the teacher, as the feedback produced was often directed to the learners; however it is plausible to assume that these technologies would be a valuable aid for the teacher too. In the findings of these studies some common trends emerge that are related to requirements concerning teacher support for the tasks of monitoring, coordination and evaluation of group activities in the class.

2.1 Review of state of the art

There have been many attempts to build tools to support collaborative learning activities. A review of automated techniques used to provide feedback, especially for the case of argumentation tasks has been recently published by Scheuer et al. (2012). In this section, some typical examples are discussed. In particular the presented tools are related to groupware applications that support collaborative activities through dual space interaction (shared diagrammatic representation space and if needed a text communication space): Cool Modes (Pinkwart et al. 2002), COLER (Constantino-González and Suthers 2002), FreeStyler (Hoppe and Gaßner 2002), Digalo (Glassner and Schwarz 2005).

The first case was Cognitive Tutor (McLaren et al. 2005; Harrer et al. 2006) a support system that was integrated with Cool Modes (Hoppe et al. 2000), an application that is used by students to develop joint diagrammatic representations. To support feedback that could be provided to the teacher or directly to the students, different groups’ solutions were combined in a single graph through a tool called Behaviour Recorder (BR). Live student data were used for this purpose. The objective was to use this combined solution to support the intelligent tutor that would make suggestions about feedback to individual groups. This prototype however was tested in the lab, and no data exist on empirical studies involving its use in authentic class conditions.

COLER Coach (Constantino-González and Suthers 2002, 2007; Constantino-Gonzalez et al. 2003) was developed to produce advice, based primarily on comparison of the individual and collaborative work of students and on measures of students’ contribution to the jointly developed diagram, as an extension to COLER (Constantino-González and Suthers 2002), a collaborative application that was used in database design activities with Entity-Relationship diagrams. COLER Coach operates by finding structural differences between the students’ evolving individual and group solutions and identifying opportunities to suggest actions based on those differences. It also evaluates collaborative features of student interaction, such as lack of participation by a particular student. Contrary to the Cognitive Tutor, it is not based on a model built from live student data. Rather, it operates by dynamically analyzing differences generated through comparison of the individual and group solutions. Evaluation of COLER Coach was also performed in the lab, involving expert judgement and user opinions.

Contrary to the previous two cases that implied replacing the teacher, ARGUNAUT Moderator Interface (De Groot et al. 2007) aimed at supporting the teacher. This tool was combined with Digalo (Glassner and Schwarz 2005), an argumentation tool and FreeStyler (Hoppe and Gaßner 2002), a diagram building tool. Moderator was used in the supervision and coordination of electronic discussions enabling the teacher to monitor the activities of parallel groups (de Laat et al. 2008; Wichmann et al. 2009), with various representations as charts, snapshots, notifications (alerts) for important events. Various studies (e.g. Asterhan and Schwarz 2010; Asterhan and Eisenmann 2011; Schwarz and Asterhan 2011) have been recently performed on the subject of teacher support in CSCL, which involved the Moderator. In these studies the number of groups was relatively small, 2 to 5 groups, and the number of group members was 3 to 7, the task being argumentation. In this setting it was found that teachers can provide proper guidance for productive engagement in the synchronous collaborative activities of many groups in the same class. Adequate support for the teacher is very important in this case.

2.2 Common tools requirements from review

The studies discussed, lead to certain observations regarding the requirements of the teacher supporting collaborative activities: the social entities monitored, the representations used, the teacher’s actions that were supported. The main points, related to our research, are briefly summarized next.

  1. 1)

    During group activities in the class teacher’s awareness was supported as follows:

    1. 1a)

      In order to monitor the activity of the class, a variety of representations were built that facilitated understanding of groups’ activity: (i) Overview representations such as charts were used to summarize interaction data through indicators (e.g. the indicator of members’ participation in group activity in Moderator). (ii) Comparative representations, such as side by side presentations of diagrams that facilitated the comparison between solutions, used in Cognitive Tutor and COLER Coach. (iii) Alarm messages were generated automatically from interaction data (Moderator), and messages concerning student participation in group activity in COLER Coach.

    2. 1b)

      In order to monitor group activity group’s shared workspace and dialogue space were mainly used. (i) In the shared space, diagrammatic representations of either problem solving or reasoning helped the teacher to understand the current state of the group (Moderator, COLER Coach). (ii) The dialogue space was used to support awareness of negotiation and reasoning. (iii) The representation of the dialogue in a diagrammatic way supported identification of interesting episodes, like conflicts, that helped the teacher follow closely the reasoning of the participants (Suthers 2005) and decide if intervention is needed.

    3. 1c)

      In order to monitor group progress, the comparison of individual group activity to the activity of the class was proposed. This was either by (i) direct comparison of groups (COLER Coach) or (ii) by automated comparison between groups and combined class solution (Cool Modes Tutor). This approach aimed at creating feedback to students through an Intelligent Tutor. Often the comparison involved a suggested solution that was compared to group solutions, an approach however not relevant to not well structured problems.

  2. 2)

    The timely information of the teacher was an important requirement. This was achieved either by quantitative representations such as charts depicting indicators, like number of interactions of participants, and by qualitative representations like snapshots of shared work spaces. These were continuously updated, in order to keep the teacher aware of the evolving activity. On the other hand, automatically generated alerts helped draw the attention of the teacher to incidents requiring special attention. At this direction intelligent tutors were used: Cool Modes Cognitive Tutor, Moderator (more specifically the so-called deep reasoning loop), and COLER Coach. The real time feedback and guidance provided to the students was based: (a) on processing of interaction data and (b) on background knowledge used to produce rules for feedback (e.g. Moderator). The quality and relevance of the feedback and guidance provided in this automated way was significant but did not seem to be comparable to that of an experienced instructor (Constantino-Gonzalez et al. 2003). Intelligent tutors do not seem to be able to replace the teacher, since they cannot handle all possible cases (idem) as no sufficient expertise exist for each particular subject to establish all necessary rules (McLaren et al. 2005), so in general all these AI approaches (Scheuer et al. 2012) have rarely managed to find their way to the class.

  3. 3)

    Regarding the detail of the information presented to the teacher, the studies discussed here did not refer to it as a critical factor for teacher awareness. This was associated with the kind, duration and complexity of the activity. For instance, modifications in the shared workspace were usually sparse and limited; therefore periodic snapshots of each work and dialogue space seemed to give adequate information to the teacher, without requiring continuous monitoring of them. In a study that focussed on the understanding gained by teachers from systematic inspection of group activity (Lund and Baker 1999), teachers who studied the data after the class with no time constraints, managed to reconstruct students’ reasoning by inspecting intermediate stages of the activity, with no need to use more detailed available data. On the other hand, in a study involving use of Moderator in the class (Wichmann et al. 2009), the teachers did not manage to acquire a thorough understanding of the activity, as they seemed to be mostly informed about just student’s participation, and as a result it was not possible to give feedback relevant to the task.

2.3 Common trends

The common direction in these studies was to support teacher awareness with various representations of the recorded interaction data, and groups’ products. Interaction data were used for real time monitoring or for alerts generation. Individual group product development was presented using snapshots of the workspace. The comparison of groups’ products and group behaviour was used for the evaluation of group’s progress, estimation of the activity progress in class level and teacher’s feedback to groups. Intelligent Tutors are a common trend in the reported research. However, automated teachers do not seem to be able to replace the teacher, since more experience and study of the field is needed to establish necessary rules in the class context.

3 Method of study

Based on the findings of the studies discussed in the previous section, a research agenda was defined that involved a series of field studies which implicated students and teachers, with focus on teachers and their support. The studies were organized in two phases, one with limited support to the teacher and the second with group activity data available to the teachers in order to study how it affected their behaviour. The research methodology followed is that of large scale design-based research (Collins 1992) with involvement of various educational institutions (two universities and four high schools) and teachers of different educational levels that supervised group activities. The studies focussed on specific research questions, related to teacher support. The studies took place in authentic class conditions. The activity in the classes was recorded in various ways (video, screen capture, observer notes, and interviews), following an ethnographic approach.

3.1 Overview of classroom studies

The first phase of studies, in which the teachers were not supported by tools, or the teacher tools used did not capture group activity data, involved two scenarios: (A1) in which the teacher supervised group activity with no supporting tools and (A2) in which the teacher used a class management tool that monitored individual students. The studies of scenario (A1) involved students working in dyads in small and medium size classes and of limited duration (Voyiatzaki et al. 2004). The studies of scenario (A2) involved dyads of students in medium size classes with activities of 30 to 90 min duration (Voyiatzaki and Avouris 2005; Voyiatzaki et al. 2006, 2008a, b; Voyiatzaki and Avouris 2009).

Based on the findings of phase A, a prototype was developed that supports the supervision of the activities of groups, capturing group interaction data. This was used in the study of the second phase (scenario B). In this case too the classes involved were of medium size with duration of up to 90′. Six classes supported by two teachers, were involved in these studies. See Table 1 for overview of the studies.

Table 1 Studies of phase A and B
Full size table

A coding scheme was defined to describe the observed actions of the teachers in these classes, related to the use of available teacher tool, the representations used and triggering events for these actions. The description of teacher actions helped identify typical behaviour and the tools that were used to support it. In all these studies the students used the dual space interaction tool Synergo (Avouris et al. 2004) for jointly building diagrammatic solutions to algorithmic problems, various collaboration scripts were used, which involved dyads of students with alternating roles. The group members communicated through a text-communication channel, as they were co-present but not sitting next to each other, so their communication was recorded. The group activity was coded through the Object-oriented collaboration analysis framework (OCAF) (Avouris et al. 2003). This describes the activity through a sequence of simple events, which contain information about the type of action that was performed to a diagram object, the person who performed it and the time that occurred. OCAF captures in uniform way both dialogue and shared space actions of group members and facilitates derivation of quantitative indices of the activity. Regarding the text communication, primary and derived indices were produced like the total number of messages, the average number of words per message, the number of rotations in the dialogue, the symmetry of the dialogue (i.e. the degree of participation in the dialogue), the number of questions, etc. Regarding the actions in the shared workspace indices were produced like the total number of actions, the symmetry in different types of actions, the number of rotations of partners in different types of actions, the symmetry in text modification actions, the number of items in the shared space at the start and the end of group activity. These indicators and others derived from them were used to provide evidence of the progress of activity and of the quality of collaboration. From analysis performed using such data (Kahrimanis et al. 2012), it was found that some indicators could be used as measure of the quality of collaboration. For example in the context of studies where groups solved an algorithmic problem, the number of text messages exchanged and speaker rotations was correlated to the quality of collaboration. It was also, discovered that while the symmetry between group members in text messages exchanged was not related to the quality of collaboration, the opposite was true for the symmetry in the actions in the shared work space. These quantitative measures were the input to the representations of group activity at run time which were produced and used in the tools of phase B.

3.2 Teacher training

An important aspect of the studies was the training of the teachers involved, who often had no previous experience with supervising group activities. For this purpose we used Synergo (Avouris et al. 2004), that includes a tool for presentation of captured interaction data. This has been used in the past for research purposes, but in this study it was used for training teachers to become familiar with the phenomena that appear when working with groups, which is a critical issue for real classroom settings (Chan 2011). The teachers at the end of a group activity were asked to inspect the log files and the produced representations and analyse them through this tool. These static representations for each group included: a) the diagram produced in the shared space, b) group members contribution to the diagram and to the exchanged text messages, c) the number of actions per group and per student and d) the evolution of indicators during the process. So teachers were asked to use these representations and make assumptions and evaluate the group activity. In addition, a playback tool was used to present to the teachers the group activity in the same way as seen from student workstations in simulated time. This was very realistic but its use was proven very time consuming, both during training and at run time, as discussed next.

In the next section we present the main findings of the studies.

4 Phase A study: Supervision without access to group data

Two scenarios were studied during phase A. In the first one the teachers had no access to supervising tools, while in the second one the teachers were able to monitor individual students with tools that however did not provide access to group data.

4.1 Phase A1: Without teacher tools

The first scenario was the simplest to follow, as the teacher supervised group activities by just moving around in the classroom, inspecting the screens of the students. The behaviour of the teacher was recorded and then analysed, in order to discover typical patterns of task execution.

It was observed that the teacher supervised the group activity by moving from student to student at random. It was a common phenomenon that a specific student was selected and more thorough monitoring of the activity followed that involved inspecting the student screen. There the teacher had access to the state of the work space and the text dialogue, and through that screen, indirectly the teacher had access to the group in which the specific student belonged. This pattern continued as long as there was no major disturbance in a group or request for help.

A disorder, i.e. a deviation from expected problem solving process, could be diagnosed by the teacher from either the state of the work space of some student, or because a student requested explicitly support. This deviation could be either a delay in the development of the scenario, or a too fast progress of it, without involving necessary negotiation among group members. Both cases were indications of problematic collaborative behaviour. When a disorder was diagnosed in a group, the teacher focused on the screen of one of its members. Starting point was the text communication. In this the supervisor sought episodes that may highlight the cause of the disorder, for example, phrases like “I do not know, do it yourself” or “leave it to me, I just take over to get done” showed failure of a team member, but also failure of the group to support each other, to manage time, and to coordinate the activity (Meier et al. 2008). Phrases like “What …I do not understand what you do …” “Leave it to me” were indications of lack of common understanding (idem).

If this information was proved not sufficient to appreciate the situation, the teacher then talked in person to the group member or discussed with all group members if this did not disrupt the whole class. When it was found that a specific group member was the cause of the disturbance, the teacher focused on the particular individual. The teacher then checked if similar phenomena appeared in other groups. If, for example, a notable delay was observed in one group, the teacher compared their progress to that of other groups. If the same pattern appeared in many groups, then some assumptions were made for the whole class and for the nature of the task or the instructions given. Then direct communication phase started in order to understand the cause of the disorder, and finally to give appropriate feedback at the right social level: the class, a specific group or an individual.

4.2 Phase A2: Teacher using class monitoring

Next, a further study was conducted according to scenario A2, in which the teacher supervised the activity using a class monitoring tool that provided information over the activity of each individual student. The tool used was NetOp school.Footnote 2 Through this class monitoring tool the teacher was able to attend each student’s desktop. Also overview of the entire class was provided through thumbnails of students’ desktops. So group activity was only indirectly observed through individual students’ workstations, but it was not possible to access data related to group interactions, as they were not recorded. Environments like the monitoring tool of this study, are not specially intended for group activities, they provide rich monitoring capabilities but are not able to create representations that depict information about group interaction, such as charts of relative contribution of group members, etc.

In our study the teacher monitored usually one student activity at a time. Thirty percent of the groups were monitored. The display of a group member often gave a rather incomplete view of the group. To get the full group state the teacher had to go through the screens of all group members, however even in this case there was no facility for navigating the dialogue and the history of group activity. Since the tool itself did not support representations of group memory (Hoppe and Gaßner 2002), the teacher was not able to navigate the history of group activities. So the teacher had to resort to her own recollections of past group activity. In general, in the study of scenario A2 (Voyiatzaki and Avouris 2009) the recorded diversions in student activity were similar to those of scenario A1. The teacher in this case, monitored the activity of the class through the displays of random individual students, without however the need to approach the student workstation. Group activity was presented indirectly through the screens of individual group members. If the data of the individual screen was not sufficient to be aware of the group activity, then the teacher focused on the text communication history and moved to desktops of other group members. It should be stressed that the teacher could not navigate the group dialogue history or pan in the work space without taking control of the individual student’s screen. This would have had the effect of stopping student’s participation in the group activity, thus disrupting the group. As in the previous case study, the teacher compared the activity of different groups before taking any action, something that was facilitated by the monitoring tool. In case the available data was not enough, the teacher came into direct contact with members of the group in person. Depending on the findings from the available data, the teacher proceeded with intervention and feedback either to the whole class, a specific group or an individual.

4.3 Common observations during phase A studies

In both studies of this phase, similar patterns of actions were observed. The teacher focused on certain individuals and had access to their screens. Class understanding was a result of synthesis of teacher’s view of individuals. Comparison of the displays of different group members helped draw conclusions on the progress of individuals and groups to which they belonged. When investigating the cause of a disturbance, face to face communication with group members often supported teacher awareness and decisions on intervention. However, continuous monitoring of all group activity was not possible; this was the main cause of misunderstandings for the teachers. For improving the appreciation of the process, the teacher had to resort in face to face communication with group members, even though in some cases this communication resulted in further disturbance to group activity. These limitations were dealt with in the case study of phase B, discussed next.

5 Phase B study: Use of supervision tool

5.1 Context of study

In studies of phase B the teacher was supported by a monitoring tool that provided group data. A prototype was used that was developed for this purpose; the Synergo Supervisor (Voyiatzaki et al. 2008b), adapted version of a tool designed to support researchers for offline analysis of collaborative activities. The design of this prototype was based on findings of the literature review and the studies of phase A. In this environment, the activity is presented at the teacher at class level, each specific group or individual group members. The metaphor used was that of a pilot cockpit (Dillenbourg 2005). This allowed monitoring of the activity of the class, identification of signs of emerging phenomena, navigation on different aspects of classroom activities, focusing on different social levels, finally guiding the teacher to intervene, the tool shares many common characteristics with other similar teacher support tools, like the Moderator, discussed in Section 2.

Synergo Supervisor monitoring screen is split in two parts: the Supervising area and Focus area (see Fig. 1). The Supervising area contains overview of the activity of the class, constantly available to the teacher. The Focus area changes according to the actions of the teacher, providing a variety of information, as described next.

Fig. 1
figure 1

Synergo Supervisor monitoring screen

Full size image

In the Supervising area a miniature representation of the students and the groups formed is shown (see Fig. 1). Using this view, the teacher can focus on specific group or individual. In the same area, alarms may be displayed through which the teacher is immediately informed about events that occur in the groups. In addition, there is a graphical representation of groups’ activity through selected indicators, e.g. in Fig. 1 a chart of the number of chat messages and actions of the 6 groups of a class is shown.

The content of the Focus area varies, depending on the social level of the monitoring activity (class, group, individual):

  1. (i)

    When the teacher focuses on the class, charts of selected indicators relating to the activities of all groups and their evolution in time are shown. It may also include comparative representation of qualitative and quantitative data of group activity through which it is possible to obtain a quick overview and compare their progress. An example of comparative representation of groups’ activity is through thumbnails of the diagrams of all groups.

  2. (ii)

    When the focus is on a specific group, the area contains the presentation of the activity through typical excerpts of the diagram in the workspace and of the text communication, through which it is possible to review the reasoning, negotiating process, conflicts, disagreements, delays, indifference or misunderstandings of group members and collaboration failures. It is also possible to review questions, clarifications, etc. exchanged with the teacher in the past, as well as interventions and annotations of the teacher. In this area it is possible to include charts of quantitative indicators of group interaction as well.

  3. (iii)

    Finally when the focus is on a single individual student, then this area includes excerpts of the workplace where the contribution of the student is shown, individual contribution to text communication, highlighting individual questions, ideas, messages exchanged, etc.

In the following, we discuss observed teachers’ behaviour when supervising group activity using this tool.

5.2 Teacher behaviour in study of phase B

In the study of phase B, in contrary to that of phase A, the group activity was presented to the teacher in various ways, through the tool used. Each group had its own identity, which was shown in the Supervising area, including information about its members. Associated group interaction data and group history were easily accessible for inspection by the teacher, while the activity was taking place. The teacher had also access to synoptic views depicting the class but also could easily navigate to more detailed information, e.g. concerning actions of an individual. Monitoring of a group was done through quantitative indicators and snapshots of the dual spaces (shared workspace and chat). Thirty percent of the groups were monitored through the tool in the classroom.

In Fig. 2a flow chart of typical teacher behaviour is shown based on the observations and coding of teacher actions while monitoring the class group activity. In steady state, the teacher monitored the whole class, using the Supervising area information. Class monitoring involved comparing groups through indicators like, for instance, the number of entities in shared work space. Then, mostly randomly, the teacher would move to a specific group in order to see the current status of its activity. This view includes graphs of selected indicators, current snapshot of work area, chat activity (see state 3 in Fig. 2).

Fig. 2
figure 2

Flow chart of teacher activity in phase B study

Full size image

This steady state would continue, until the appearance of a disorder which was indentified either by the teacher during monitoring or through alarms (Class.alarms)Footnote 3 (state 4). For example in one case the solution to the given problem required approximately 10 entities in the shared space and the teacher noticed through the graphs that a group had introduced just two entities while others had entered 9–10 entities, then she assumed that there was some disorder in this group. The attention of the teacher may be drawn to a specific group by the tool, if a relevant rule had been violated and an alarm triggered.

Then the teacher would enter into the disorder investigation phase. The investigation usually started with exploration of the dialogue (state 5). This was proven to be the richest area for rapid detection of incidents, despite the fact that text communication was often elliptical and situated (Herring 1999), and thus not easy to follow under time pressure. Exchanged messages such as “What are you doing ”,“ I do not understand!” “Don’t erase it!”, “I do not know anything, I give up!” “Why don’t you answer?” led the teacher to assumptions about the cause of the disorder. In some cases the teacher further investigated the graphs that highlighted the contribution and role of group members (state 6), such as the number of messages exchanged, and the contribution to the shared space.

The dialogue could reveal whether the disorder was associated with the collaboration or the task problem solving. If it was detected that the students had problems with the task, the teacher proceeded with navigation of the states of the diagram built in order to understand the cause of the difficulty, (state 7, in Fig. 2, snapshots navigation) or playback (state 8, problem solving process navigation) which provides more detail of the problem solving activity, including the dialogue and shared workspace actions. In an incident, one partner did indicate an error and as a result, another group member erased part of the diagram in order to make the suggested corrections. If not clear verdict could be produced by inspecting the available data then the teacher would proceed with inquiring group members for clarifications (state 13). This process of investigating the group activity may lead to a number of further actions: When the teacher realised that the disorder may be associated with a particular member of the group, e.g. with a problematic behaviour or significant lack of knowledge, then the teacher focused further on that person (state 12), and if necessary proceeded to direct communication with this particular student (state 14). By comparing the progress at the class level (state 9, Class.snap) or by looking in charts indicating similar behaviour in most groups, in some cases the disorder appeared in the same way in other groups. In this case the teacher came to the conclusion that it was the task or the script that was problematic rather than the specific group. Based on the findings of the exploration phase and the consultation with individual or group, the teacher proceeded with the adequate feedback at the class, group or individual level (states 15, 16, 17, respectively).

6 Discussion

In this section we compare and discuss the observed teacher behaviour in the studies of phases A and B, in order to identify the effect of the group data that was used in phase B. It was found that the teacher monitoring group activity in the classroom went through four stages of actions in both studies. After introducing the activity and giving the instructions to the class, the students got involved in the group activity and the teacher monitored their groupwork using the available tools. This phase was termed as steady state monitoring. During this phase the teacher did not identify any disorder and simply followed the class, randomly selecting and focusing on groups or individuals. The second phase (a result of a deviation of the plan) was that of investigation of a disorder. In this case, the teacher focused on a group, whose unpredicted behaviour was observed, attempting to identify the cause of the disorder. This may be attributed to either an individual, a specific group or even the script and the task, so that an intervention was needed at the level of the class. In order to identify the cause of the disturbance, the teacher often had to go into a third phase that included direct contact with some groups or some of their members either face to face or through the supervision tool. Finally the fourth phase involved feedback provided to groups, individuals or the whole class. This cycle was repeated many times during group activity.

In case studies of phase A, the teacher could focus on the individual, which was the discrete entity in the class. The next focus level was the entire class. Focus on a specific group when it was possible (A2), was done though the screens of the members of the group. In the steady state it was observed that the teacher supervised the class by looking at the screens of individual group members, either by approaching their screens or through the class management system, when available.

In the same study, during disorder investigation phase, the teacher explored the dialogue space (Group.Chat) in the screen of the student that was investigated, in order to find clues of the cause of the disorder. Then the teacher moved to more detailed view to explore the process (Group.Proc). In scenario A1, when no tools were available, the teacher had to communicate in person in order to understand the group activity, take often control of the workstation and navigate the history of the activity. This situation was visible to other members of the group, so the temporary pause of activity of a member of the group was known and understood by all the partners. In scenario A2, the teacher followed the progress through the class management tool, in order to investigate details of the group activity. The most abstract representation that was available, the screen thumbnails, was used for monitoring all the groups in the class. In some cases the teacher resorted to taking control of the workstation of a member of the group disrupting the group activity and the flow of group work.

In case study B, during steady state, the teacher monitored the class through indicators of the groups (Class.graph), which was the most abstract available representation, . When he chose to focus on more detail, he could choose a random group. The group was monitored through the overview representation (Group.overv) that contained the main information of its activities until then.

At the disorder investigation phase, it was observed that the teacher used all available data concerning the activity of the group, having as a starting point the text communication space. This space, like in Study A, remained an important source of information that helped the teacher build a first impression of the situation investigated. Navigation in the dialogue could quickly identify interesting verbal exchanges. Unlike phase A, the teacher had available compact representations of group activity (Group.graph, Group.snap). In them the teacher could navigate through time and identify significant episodes in group activity, and monitor group’s “tempo” (Ligorio and Ritella 2010).

When not enough information was available to investigate the deviation, the teacher moved on to change the point of view to a different social level. In study A from the individual moved to the group and in study B from the group to the person. In all studies the teacher proceeded to comparisons between individual group members in studies of phase A, or among the groups in studies of phase B.

The teacher moved to the third phase of direct communication when there was not enough information or there was not sufficient time. In all cases, where the communication was face to face, this was visible to all group members, as they were co-present. If an electronic message was sent, privacy was maintained, especially when the teacher addressed a person, but even in this case some disruption to the group was caused because a group member changed the focus of its attention.

The significance of the reported study lies on the findings that can lead to some interesting conclusions related to future design of supportive supervision tools for the teacher. In the following there is a discussion on the findings of the reported study referring to the typical teacher actions during supervision of group work in the classroom, and to requirements concerning time availability, level of detail and representations of group data.

7 Summary of observations on teacher tools

7.1 The social entities in the classroom

The teachers in our studies dealt with the difficult task of maintaining awareness of the activities of various social entities in the classroom. In the same physical and social space there were three levels of social organization (Dillenbourg et al. 2009) with which the teacher had to interact: the class, the groups and the individuals. When there was not enough information available in one of these levels, the teacher had difficulty in following the phenomena that took place in the class (Voyiatzaki and Avouris 2009). In collaborative activities, beside the individual student who is the main entity, the group should be recognised as a discrete entity, too. Group is not just the sum of its members, and group data should not be considered as a sum of the actions of the individuals that comprise the group (Stahl 2006). Therefore, in supervision tools each group needs to have its own identity, its own profile with its own characteristics that are related with its members, its history of activities, and its interaction data.

7.2 Summary of teacher actions during supervision

Here, we summarize our observations on teacher actions. An ordinary action of the teacher is to monitor all social levels of the class. A time consuming and complicate action is the exploration of phenomena that appear in all social levels in the classroom. A typical case is the investigation of disorders. During this action, the teacher is navigating in available data, he is moving between current status and past events, he is changing views of data navigating in various representations, in order to identify and investigate important events. Additionally he is comparing group states using various representations looking for disorders, and identifying their causes. The teacher is communicating with all available means with all entities in the classroom. Finally, he gives feedback based on the awareness that he has acquired from these actions.

7.3 Monitoring

Monitoring is a common action of the teacher during group activities to create a general perception of the activities in the class, and can be interrupted when a disorder appears. The teacher monitors different entities using various representations, but he prefers monitoring abstract visualizations of meaningful data. Switching monitoring among social levels and among various representations need to be supported.

In our studies the teacher switched from the whole class view to the different entities, i.e. the students screen views in phase A and the groups views in phase B, where group data was available.

The teacher should be supported to move between the three social levels, during classroom activities. While he is moving physically from student to student in the classroom, monitoring their activities (e.g. through their screens), he should be supported to keep track of the other social levels.

During a class activity, the teacher builds awareness of the class state, from his insight on the states of different groups in the class. Conversely, when studying a specific group, he relates the activity in a group to events that happened in the classroom. This way the teacher builds a relation between the whole class and individual groups.

The teachers, as found in the studies, seem to create their own hypotheses for the development of the activity. The model is related to the way the given problem is expected to be done, i.e. the expected complexity and structure of the solution, the kinds of entities used, actions, etc. The scenario also includes assumptions about group collaboration patterns, roles etc. For example, in our case, intense activity was expected in the shared work space during the development of the solution, high communication activity during negotiation and investigation of its correctness, modifications in the shared work space in the phase of testing and revising of the solution. By monitoring the activity the teachers are interested in discovering deviations from this model. This can be facilitated by abstract views like relevant indicator charts. The triggering of alarms, which are generated automatically from these indicators, draw the attention of the teacher to deviations from the expected behavioural pattern, and has also been proposed in other types of collaborative activities such as developing arguments (Scheuer et al. 2012). These mechanisms can be an important contribution to a new generation of monitoring environments for the teacher and address state awareness and workflow manipulation (Dillenbourg et al. 2011a, b), or provide a higher level feedback to the teacher.

7.4 Exploring important events

Exploring important events is another typical activity of the teacher. This involves search for discrepancies, or outliers, between groups, i.e. navigate in group data and perform comparisons. Comparisons, in our studies, were facilitated when quantitative data was available in appropriate representations, like the charts of indicators of groups in the tools used in Phase B. These facilitated the comparison especially when the teachers were trained and familiar with the representations used. In general, it is difficult to build quantitative representations of descriptors of group activity, that are qualitative in nature, like the depth of reasoning, quality of collaboration, quality of the solution to the given problem etc. It was found that the teachers who participated in our studies attempted to link empirically phenomena with quantitative indicators. For example, it was observed that some teachers compared indicators like the small number of interactions, high degree of asymmetry, small number of inserted entities in the shared space, large number of entity deletions in the shared space, etc., to class average values, and related these empirical interpretations of the observed patterns to group problems. Such observations, if confirmed by systematic studies on specific scenarios of activities can be incorporated into the monitoring tool to support the teachers in the future.

It was observed that in some cases close monitoring of group activity was necessary during investigation of a disorder. This was done by following the action (in phase A studies) or using the fast playback facility of the supervisory tool (phase B). This was proven an important process for increasing teacher’s awareness, which however necessitates increased time resources. A more efficient approach seems to be the use of highlights of the activity in the form of sequence of snapshots of the shared work space. As pointed out by teachers during the studies, the changes made to the shared space, related to the complexity of the activity and the object. A sequence of snapshots of the shared workspace, at short intervals, is sufficient to help the teacher follow the progress of the activity, in order to focus on important episodes.

This cannot be generalized however, but it is recommended for activities with simple diagrammatic representations whose significant differences are evident in miniatures such as those made in the studies mentioned here. The monitoring of collaborative activity through snapshots of the shared workspace can be an adequate representation when it is possible to navigate to interesting moments, so that the teacher can identify quickly the most important episodes of cooperation and conflict resolution.

7.5 Temporal dimension

With regards to the temporal dimension of activity development, the teacher wished to know the current state and the sequence of events that lead to the current state for all three social levels monitored. Regarding the whole class, the state of activity until the present time, was obtained through a) graphs illustrating the evolution of selected indicators and b) snapshots of the shared space of the groups. Regarding each group, this was obtained by following a) the dialogue and b) sequence of snapshots of the shared space which were navigable along the time dimension. Similar approaches have been used in other studies with different types of collaborative activities (De Groot 2010). Other useful means for following the development of the activity were suitable charts of indicators monitoring the current situation of the group. Regarding the individual student, more detailed monitoring of the student’s contribution in the dialogue provided information on the reasoning of the individual, and her collaborative behaviour, in conjunction with her contribution in the shared workspace.

7.6 On representations used

In relation to the representations used, many representations have already been presented. It has been observed that charts were powerful means of abstracting the evolution of the activity, as they allowed a comparison of quantitative indicators of the groups, as mentioned above, but they also facilitated comparisons along the time dimension. Alarms, that are produced after calculation and comparison among interaction indicators, attract quickly and effectively teacher’s attention too.

When focusing on a specific group, the teacher first inspected the current state of the activity through the current snapshot of the work space, through which it was possible to assess the progress of the group. This was related with the time evolution of the activity through graphs of indicators. This strategy in studies of phase B was facilitated by the group overview that contained different representations (Group.overv = snapshot, charts, dialogue). Focusing further, the teacher inspected the dialogue in the group communication space. Through this, it was possible to see the development of collaborative activities to the current time. The total process of the development was presented through series of snapshots presented together with the dialogue. At the class level, it was observed that the teacher compared the activity of groups, first comparing the current state of development and then doing the same comparison in earlier time points.

Regarding the level of detail of the data used by the teacher during the monitoring, this was associated with the specific teacher tasks, as discussed above. When a disorder was found exploration of the situation involved navigation in various representations and comparison of representations. By navigation the cause of the disorder was sought in past events and through different representations. By comparison, the cause of disturbance was sought in events that happened in the class and affected other groups and individuals.

7.7 On assessment strategies used

The teacher monitored the entire class and all the groups, at different levels of detail. When exploring the activity of a specific group, the teacher navigated through the dialogue and group activity in the shared space, first inspecting the current state of the solution and of the dialogue. The importance of obtaining an impartial view of the current state of the solution was evident in the cases when the teacher could not have this complete picture of the current situation since he followed the activity through one group member, as in study of Phase A (Voyiatzaki and Avouris 2009), in which case the teacher could not identify the reason of a disorder. Beyond the current state of activity, the teacher would seek past phases of the process, if there is such possibility, as in Phase B studies. In most cases it was observed that monitoring the process did not necessitate looking into the details of the recorded activity. It was enough to navigate through intermediate stages of activity in the form of snapshots of the shared space and the dialogue (Voyiatzaki et al. 2008a). Assessment of the progress of a specific group was effected through comparison with other groups. It was found that use of thumbnails of shared area (phase B), or group members’ desktops when no group data where available (phase A), were useful in this task (Voyiatzaki and Avouris 2009). The objective in both cases was to compare the current state of the groups. When it was possible to access historical data (phase B), the comparison was extended to earlier phases of the problem solving activity (idem). This approach is widely used as an assessment strategy, especially when the problem takes the form of a diagram that facilitates comparison (Constantino-González and Suthers 2002; McLaren et al. 2005).

In addition, charts of activity indicators were also used for quick comparisons between different groups. The result of these comparisons was to focus on outliers or groups with representative or interesting behaviours. When such concise representations were not available, as in studies of Phase A, the teachers tried to form a view empirically. So it was observed that the concise representations of group data were valuable means for conducting comparisons between groups and assessing their performance or reasoning about disorders (Constantino-González and Suthers 2002; De Groot et al. 2007). This finding leads to the use of group data with the aim to support evaluation during and after the activity (Kahrimanis et al. 2012; Meier et al. 2008).

8 Epilogue

8.1 Comparison to related work

Previous research in this area, as presented in Section 2, was mostly related to evaluation of new technology designed to support groups, which may be used by teachers in coordinating group activities in a class. The Moderator was designed and recently used in classes by teachers (Schwarz and Asterhan 2011). However even in this case, the size of the class and the number of groups working were more limited than in our case, thus the task easier for the teacher. The Coler Coach may well be used in a class to support the teacher, but in early studies (Constantino-González and Suthers 2002, 2007) it was evaluated retrospectively by an experienced professor who participated as an observer in the classroom, while the Cognitive Tutor was not evaluated in a class. On the other hand most of the research that dealt with human support in CSCL activities focused in distributed or asynchronous learning. In the earlier studies discussed, there were two trends: support of teacher awareness and provision of advice to support the system users, i.e. the teacher or directly the students. It was just the teacher support the focus of our research.

8.2 Main findings of the study

In our study, our aim was to follow teachers in authentic classrooms and identify patterns of actions influenced by the tools that they had at their disposal. This is a descriptive study, done in order to observe emergent patterns of behaviour. While the technologies used varied in the study, it was evident that new social practices emerged, influenced by the rather limited previous experience of the teachers involved and familiarity with the tools, as well as their teaching styles and course priorities. In technology enabled classrooms, in which the students use complex communication and collaboration tools, the teachers have to be empowered in order to meet the new challenges of such setting and integrate these new tools to their practice. They have to deal with the fact that they found it strange to have to monitor the progress of a lesson through a computer screen (at the teacher workstation) instead of moving from student to student. On the other hand they realized that they were provided with new information that created a new kind of “connection” with the class and the students. We observed that there was a continuous switch from screen monitoring to face to face interaction and thus a continuous switch between modes of operation and postures.

In terms of the representations used in the process, the original assumption was that the richest representations, in particular the student activity playback, will draw teacher’s attention. In addition, we expected that the teacher will identify diverging groups mainly through produced alarms, while use of more cognitively demanding representations, like graph plots of indicators will not be as frequent. However the technology is shaped by its use which does not follow always the designer’s expectations (LeBaron 2002). So, as discussed in the previous section, the teachers preferred condensed overview representations, and alarms that attract the attention to specific group activities. While the teachers were not able to propose the “most appropriate representations” it was found that they quickly familiarized themselves with the graphs of indicators available and seemed to be able to extract meaning from them and use them as a quick tool for identifying patterns of students behaviour. On the other hand, contrary to our initial hypothesis, highly detailed reproductions of the activity (like playback of students activity) was not used by teachers in the classroom, perhaps due to the fact that these representations were more demanding in terms of time, and the teachers felt that they did not wish to disconnect from their class activity for too long. This pattern however may be influenced by aspects like the size of the class in terms of number of groups and the time pressure factor, e.g. time allocated to the specific activity.

8.3 Future directions

The reported study involved activities with specific activity duration, size of group and class, type of activity, etc. So generalizing the findings beyond this class of activities is not easy. As the method of study used was rather descriptive, further research should be initiated for drawing conclusions applicable to a broad class of learning settings. Other issue that needs further investigation is related to methods for training teachers in computer supported group activities in the schools. In our case the teachers that participated in the studies had to be trained and familiarized with the representations in the tools and the new insights in groups’ process and reasoning, in an ad hoc manner, however more systematic approach on this aspect is needed. Acquaintance with collaborative learning requires changes in schools related with infrastructure, pedagogical and technical support and collaborative classroom culture (Chan 2011). Despite this inherent difficulty, the results of this research help to better understand the role of the teacher in this context and lead towards establishing requirements and specifications for a new set of tools that support this role. The use of supervision tools in the classroom should be seen as an extra support for the teacher offering quick overview of class activity through abstract views of groups’ progress and detailed insight in groups activities and reasoning, while keeping track of teachers interventions.

The devices used by the teacher, may nee to be further studied too. Some may consider incongruous a teacher to monitor the progress of a lesson through a computer screen (at the front of the lab) instead of moving among the students. So use of mobile technology (e.g. tablet PCs), may facilitate the teacher to combine monitoring interactions and the shared group space using technology with the more personal monitoring of the situation.