Collaborative group engagement in a computer-supported inquiry learning environment

Abstract

Computer-supported collaborative learning environments provide opportunities for students to collaborate in inquiry-based practices to solve authentic problems, using technological tools as a resource. However, we have limited understanding of the quality of engagement fostered in these contexts, in part due to the narrowness of engagement measures. To help judge the quality of engagement, we extend existing engagement frameworks, which have studied this construct as a stable and decontextualized individual difference. We conceptualize engagement as multi-faceted (including behavioral, social, cognitive and conceptual-to-consequential forms), dynamic, contextualized and collective. Using our newly developed observational measure, we examine the variation of engagement quality for ten groups. Subsequently, we differentiate low and high quality collaborative engagement through a close qualitative analysis of two groups. Here, we explore the interrelationships among engagement facets and how these relations unfolded over the course of group activity during a lesson. Our results suggest that the quality of behavioral and social engagement differentiated groups demonstrating low quality engagement, but cognitive and conceptual-to-consequential forms are required for explaining high quality engagement. Examination of interrelations indicate that behavioral and social engagement fostered high quality cognitive engagement, which then facilitated consequential engagement. Here, engagement is evidenced as highly interrelated and mutually influencing interactions among all four engagement facets. These findings indicate the benefits of studying engagement as a multi-faceted phenomenon and extending existing conceptions to include consequential engagement, with implications for designing technologies that scaffold high quality cognitive and conceptual-to-consequential engagement in a computer-supported collaborative learning environment.

Previous research on Computer-Supported Collaborative Learning (CSCL) in inquiry environments suggests that there is potential to foster deep-level engagement (Blumenfeld et al. 1991; Hakkarainen and Sintonen 2002; Järvelä and Salovaara 2004; Renninger and Shumar 2002, 2004; Veermans and Järvelä 2004). The interactive features of technologies, such as simulations and modeling tools, afford opportunities for learners to deeply engage with key content ideas and scientific practices (Arvaja et al. 2007; Harasim 1993; Krejins et al. 2002; Stahl et al. 2006; Suthers 2006). However, there is limited empirical research addressing this issue (Järvelä and Hadwin 2013). In CSCL settings, the extent to which collaboration is productive in ways that lead to conceptual understanding depends on high quality engagement in shared activity.

Collaboration and technology tools are not a panacea that ensures deep-level engagement. Group work raises challenges for maintaining engagement, including off-task conversation, exclusive focus on directions or procedures, and difficulty coordinating multiple perspectives (Barron 2003; Järvelä and Hadwin 2013; Roschelle and Teasley 1995). Technology tools raise their own set of challenges, including initial time invested in gaining familiarity in how to use the tools, as well as students’ superficial exploration of software features particularly when tools are not designed for novice learners (Quintana et al. 2004; Soloway et al. 1992). Ultimately, we do not have a good understanding about the quality and contextualized nature of group engagement within CSCL environments (Dillenbourg et al. 2009).

Extant research and operationalization limit our understanding of deep-level engagement in CSCL contexts. Existing studies have operationalized engagement as a single facet, yielding a narrow view of engagement and the interaction among these facets (i.e., behavioral, emotional, cognitive) (Fredricks et al. 2004; Ryu and Lombardi 2015). A multi-faceted take on engagement enriches our understanding of students’ classroom experiences. A second issue is that engagement has typically been evaluated at a single time point, with limited information provided about its evolving nature during task activity and over time. In addition, survey and observational measures of engagement evaluate individual learners, rather than shared engagement (Fredricks et al. 2004). Given the primacy of shared activity while collaborating and in the social construction of meaning (e.g., Roschelle and Teasley 1995; Suthers 2006), it is essential to advance these views to account for collective engagement. Finally, engagement has primarily been operationalized as general in regards to the task and classroom context, providing a decontextualized understanding of engagement. However, situational perspectives view engagement as negotiated within particular activity systems, comprised of the instructional opportunities that afford and constrain engagement in light of particular curriculum materials, pedagogical practices, and tasks (Greeno 2006). There have been some initial attempts at situating engagement in science disciplinary activity in whole class and small group contexts (Engle and Conant 2002; Gresalfi et al. 2009; Herrenkohl and Guerra 1998). This prior research has relied on discourse analyses to advance theoretical conceptualizations of these constructs. We are aligned with this perspective in articulating the need to operationalize engagement to account for context. Our research situates engagement within the collaborative group and disciplinary context, while extending the field empirically through the development of an observational protocol.

For the current research, we developed an observational measure to investigate how collaborative groups engage within a technology-mediated inquiry environment. We extend existing engagement frameworks to conceptualize and operationalize engagement as multi-faceted, dynamic, shared, and responsive to context. Drawing on this measure of collaborative engagement, we examine the range and variation of engagement quality across 10 groups, and then select two representative groups exhibiting high versus low engagement to closely explore engagement quality, the fluctuation in engagement quality during the lesson, and the subsequent interrelationships among the multiple facets of engagement, within a CSCL context.

This research is conducted in the context of middle school students learning about ecosystems (Eberbach et al. 2012; Jordan et al. 2013, 2014). This research is particularly timely as systems are key crosscutting concepts in current science standards (NGSS 2013), but remain challenging for learners because of the dynamic multi-level nature of systems (Hmelo-Silver and Azevedo 2006; Hmelo-Silver et al. 2007). One way to support learning about systems is to provide technology that allows learners to engage with systems phenomena—in this case, the Systems and Cycles curriculum includes simulations that make phenomena visible and a modeling tool that provides an opportunity for groups to discuss, integrate, and represent their understanding of ecosystems. These tools themselves may also pose additional challenges (Blumenfeld et al. 2006; Quintana et al. 2004; Soloway et al. 1992). Our interest then in deep-level engagement is oriented to the groups using ideas about ecosystems to solve an environmental problem. Building from Gresalfi’s notion of consequential engagement (Gresalfi et al. 2009), we use conceptual-to-consequential engagement to reflect how groups engage with ideas such that their application has consequences for solving a contextualized problem in a CSCL environment.

Engagement in CSCL environments

Consistent with CSCL theories, we consider engagement as a group process that is inextricable from its sociocultural context (Stahl 2013). This study conceptualizes collaborative group engagement as integrating behavioral, social, cognitive, and conceptual-to-consequential engagement. Engagement is central to understanding how to foster conceptual understanding because engagement mediates the relationship between motivation and learning (Blumenfeld et al. 2006). We view engagement as co-occurring with knowledge co-construction involved in sense making, with both being dynamically interrelated (Engle and Conant 2002). Through on-task persistence and effort investment (behavioral engagement), cohesive group exchanges (social engagement), joint regulation of deep-level strategy use and developing understanding (cognitive engagement), and application of disciplinary, conceptual and technological tools in solving authentic problems (conceptual-to-consequential engagement), engagement recursively influences the sustained co-construction of meaning.

Our guiding theoretical framework is consistent with Fredricks et al. (2004) in three primary ways. First, we consider engagement to be a multi-faceted construct that unites varying forms of engagement in meaningful ways as a “meta-construct” (Fredricks et al. 2004, p. 60). Throughout the manuscript we use the terms facet, dimension and form interchangeably. Bringing together behavioral, social, cognitive and conceptual-to-consequential is more consistent with learner’s experiences, given that individuals or the collective do not experience individual facets as isolated processes. Further, taking a multi-faceted view allows for a consideration of how these factors are mutually constituted, with the enacted quality of each having potential to influence the group’s enactment of remaining facets. Second, we assume that there are qualitative differences in the degree of engagement for each component. For example, cognitive engagement can range in quality from monitoring the shared task more superficially, while also encompassing monitoring for shared conceptual understanding. Finally, engagement is responsive to context, with the specific context encompassing the CSCL environment, acknowledging that this immediate learning context is necessarily nested within a larger classroom, instructional, and curricular context.

Drawing from this engagement framework has clear benefits; however engagement has primarily been used to investigate individual learners. Accordingly, we need to build from this conceptualization to account for the group as the unit of analysis. A consideration of collective engagement necessitates an examination of social interactions among students and the shared nature of their engagement. In addition, engagement has been measured as individual dimensions and at single time points, with limited information provided about the fluctuation in engagement or the interrelation among engagement facets (also see Ryu and Lombardi 2015). Moreover, CSCL environments have features that are likely to facilitate and/or inhibit engagement quality, such as use of technological tools that provide real-time feedback and opportunities to unpack scientific mechanisms, alongside the challenges of superficial self-regulated learning and strategy use (Blumenfeld et al. 2006), requiring a more contextualized accounting of engagement. In what follows, we define each dimension of engagement, and review research related to engagement in collaborative groups and/or conducted within CSCL environments.

Behavioral engagement

Behavioral engagement involves sustained on-task behavior during academic activity, including indicators such as persistence, effort, and contributing to the task (Fredricks et al. 2004). Previous measures have considered whether learners are on-task, attentive, and persistent (Lee and Anderson 1993; Lee and Brophy 1996). Within collaborative groups, behavioral engagement reflects a majority of members attempting to contribute to joint task work, with only intermittent disengagement by a few students. Individual learners who withdraw their participation from group discussion can undermine learning, due to lost opportunities for collaboration or by provoking whole group disengagement (Van den Bossche et al. 2006). Consistent with this definition, studies of student engagement within CSCL have primarily employed measures consistent with this engagement facet, measuring students’ participation given number of contributions (Lipponen et al. 2003), length of posts in online environments (Guzdial and Turns 2000), or whether contributions are more social (i.e., off-task) rather than around content ideas (Stahl 2000). We conceptualize on-task participation as necessary, but not sufficient, for high quality collaborative engagement. That is, students may attend to the task, without being cognitively or consequentially engaged (Blumenfeld and Meece 1988; Engle and Conant 2002; Lee and Brophy 1996).

Social engagement

Our inclusion of social engagement extends beyond the behavioral, emotional, and cognitive engagement distinctions made by Fredricks and her colleagues (Fredricks et al. 2004) to account for social interactions within small groups. Drawing from Linnenbrink-Garcia et al. (2011), we define social engagement as referring to quality of group socio-emotional interaction.¹ Quality social engagement involves respectful and responsive interactions among members of the group. Social engagement also reflects group cohesion, or evidence that the task is conceptualized as a team effort, rather than an as an individual activity. Finally, quality social interactions reflect equitable participation in which all teammates contributions are taken up (Barron 2003; Rogat and Adams-Wiggins 2015). Taken together, high quality social engagement enables groups to focus on jointly coordinating around a group product (i.e., cognitive engagement), rather than reacting to put-downs, or ignored or excluded contributions by central group members (Rogat and Linnenbrink-Garcia 2011; Rogat and Adams-Wiggins 2014). We use the terms socio-emotional interactions and social interactions, alongside social engagement throughout the current manuscript.

The inclusion of social engagement builds from the rich history of collaborative learning and CSCL research to account for group dynamics, as well as the richness that stems from cohesive and respectful interactions that facilitate shared sense making (Van den Bossche et al. 2006). This research highlights that groups often face difficulty finding common ground and may lack shared understanding (Dillenbourg et al. 2009). Negative social interactions can come to predominate group activity, and compete for limited attentional resources (Barron 2003). In worst cases, low quality social interactions can devolve into battles related to status differences and can promote inequity (Salomon and Globerson 1989). In contrast, research on learning in collaborative groups indicates that respectful, responsive, and cohesive interactions elevate the quality of joint task work (Engle and Conant 2002; Webb et al. 2006). Further, positive social interactions can facilitate higher quality cognitive engagement by helping ensure that feedback from monitoring was communicated well, supported joint and inclusive planning (Rogat and Linnenbrink-Garcia 2011). These interactions can also support behavioral engagement by helping to re-involve group members.

Cognitive engagement

Fredricks and her colleagues (2004) indicate that there are two primary conceptualizations of cognitive engagement, both in terms of investment in schooling (e.g., Connell and Wellborn 1990) as well as being a strategic and self-regulated learner (e.g., Pintrich and De Groot 1990). There is some degree of overlap between cognitive engagement and a conceptualization of psychological investment and motivation constructs. In addition, conceptualizations of self-regulated learning integrate motivational beliefs and learner’s intentionality in what constitutes high quality self-regulated learning (Pintrich 2000; Zimmerman 2000). Thus, we draw on the literature of self-regulated learning and learning strategy use as represented in the latter definition (Fredricks et al. 2004). Here, cognitive engagement reflects student involvement in planning, monitoring, and evaluation when accomplishing tasks (Pintrich and De Groot 1990; Zimmerman 1990). When students engage in planning, they intentionally set task-specific goals for how to go about solving the task and for their learning. They monitor developing understanding of content and skills integral for successful learning in activity, and adapt their use of particular learning strategies in response to that feedback. In addition, students can monitor their execution of the plan, and progress toward task requirements and set goals. Finally, effective self-regulators evaluate and reflect back on their content understanding and task performance following task completion.

We elaborate on this definition of cognitive engagement in two primary ways. First, we contextualize students’ regulation of cognition and tasks in joint collaborative group activity. This extension draws from recent research on the processes groups use to regulate their shared activity and reflects our focus here on collective group engagement (Volet et al. 2009). Integral to the consideration of cognitive engagement is the notion of regulation as socially shared. Socially shared regulation refers to multiple group members regulating and coordinating their joint activity (Vauras et al. 2003). Rogat and Linnenbrink-Garcia (2011) used the cognitive sub-processes from a self-regulated learning perspective to understand and elaborate the quality variation of collaborative groups engaging in shared planning and monitoring.

Second, we ground conceptualizations of quality differences in regulatory strategies within a technology-mediated context. Limited research has investigated how groups effectively regulate within CSCL environments (Järvelä and Hadwin 2013). We know that computer-supported learning can support and enhance students’ use of regulatory processes (Azevedo 2005). Socially shared regulation research has demonstrated the presence of frequent and at times extended use of regulatory processes within synchronous and asynchronous CSCL environments (Iiskala et al. 2011; Lee et al. 2015). Here, cognitive engagement with technological tools can be characterized by groups’ thoughtful and deliberate uptake of the affordances offered by the learning environment (e.g., predictions and goals for running simulations; model revision). However, groups exhibiting moderate to low quality cognitive engagement during planning or monitoring may demonstrate a focus on superficial features, such as brief planning discussions or a focus on color or neatness, with implications for challenges reaching consequential engagement via the technology tools.

Conceptual-to-consequential engagement

Our introduction of conceptual-to-consequential (CC) engagement provides an important extension to the forms synthesized in Fredricks et al. review (2004). CC engagement refers to making progress in solving meaningful problems through the use of domain-specific content and disciplinary practices as conceptual tools (Gresalfi and Barab 2010; Gresalfi et al. 2009). It involves making progress in critically considering the utility and impact of disciplinary content, strategies, or tools relevant to a larger task context (e.g., driving question, problem or project). Consequential engagement also specifies an active and agentic role for learners to justify identified solutions, particularly after having weighed and critiqued alternative solutions to the problem. In this way, consequential engagement builds from the connections and synthesis, as well as regulation, from cognitive engagement, toward a reflection of connecting to something larger.

Extant research has suggested that students’ connections between conceptual ideas and a broader context can be lower quality, shown by simple knowledge telling with limited connections (Scardamalia and Bereiter 1996; Chernobilsky et al. 2004), to moderate quality showcasing connections among content ideas (conceptual engagement or sense making), to higher quality linkages among content ideas with prior knowledge, everyday experiences, and/or the context of the larger problem (i.e., consequential engagement). Thus, we extend this construct by including conceptual engagement as part of a continuum that should culminate in consequential engagement. This inclusion works to better capture the quality range in which groups work with content ideas and disciplinary practices when problem solving.

Gresalfi and colleagues (2009) have argued that it is important to promote consequential engagement along with the practices of encouraging procedural and conceptual engagement. They stress the relevance of consequential engagement as it sets the stage for designing CSCL contexts, such as multi-user virtual environments and other computer-supported inquiry contexts. They note that contexts and practices that “emphasize making connections can only lead to robust learning when they are supported by tasks that create opportunities for students to grapple with the meaning and utility of content” (Gresalfi and Barab 2010, p. 301). Their research has primarily focused on providing rich examples to assist in conceptualizing the construct; additional research is needed to examine the extent and range with which groups engage consequentially. Specifically they focus on how consequential engagement addresses the ways that students can realize the opportunities provided by technological tool and associated classroom practices (Gresalfi and Barab 2010). We anticipate that high quality CC engagement with technological tools around domain content and scientific practices may promote the development of conceptual understanding in CSCL contexts.

Current study

This paper examines a multi-faceted, dynamic, shared, and contextualized conceptualization of engagement within a CSCL environment using our newly developed observation protocol. Toward this end, we initially explored quality variation in ten collaborative groups’ behavioral, social, cognitive and conceptual-to-consequential engagement using quality ratings. This was followed by coupling in-depth qualitative analysis and contrasting cases of two groups characterized by high or low quality engagement relative to the sample with the intent of describing engagement quality, the fluctuation in engagement quality during the lesson, and the interrelationships among engagement facets. In developing these cases, we prepared narratives that thickly described each engagement dimension in 5-minute intervals and visual representations of each group’s engagement ratings across a lesson. We also examined each group’s final explanatory model that was the subject of the observed lesson. A final analytic focused on case group comparison.

This research extends prior CSCL research that has examined student participation and group dynamics (i.e., behavioral and social engagement), with limited examination of higher quality forms of engagement (i.e., cognitive and conceptual-to-consequential engagement). Further, this research situates the study of engagement within a collaborative group and science disciplinary context by characterizing the quality of learning and regulatory strategies employed as a collective, and their application toward solving the larger unit problem using explanatory models within a technology-mediated learning environment. Finally, by drawing on this multi-faceted and evolving conceptualization, we are able to examine the interdependence among these dimensions over the course of the lesson to provide a fuller characterization of group engagement in face-to-CSCL. Our methodology combines the use of ratings operationalizing engagement as multi-faceted and collective, with qualitative analyses that aimed to examine the mutual relations among these facets over the course of activity.

• Research Question: How does a multifaceted, shared, dynamic and situated conceptualization of engagement serve as an observational tool for studying CSCL?

Method

Instructional context

Students participated as part of a technology-intensive curriculum designed to support 7th-graders’ learning about aquatic ecosystems (Hmelo-Silver et al. 2011). The curriculum was 6 to 7 weeks, spread over the academic year. The curriculum was divided into three units focusing on aquariums, ponds, and marine ecology. Each unit had a driving question in the form of a problem (Blumenfeld et al. 2006; Hmelo-Silver 2004). For this study we focused on the pond unit, with the driving question on investigating causes for fish death in a local pond.

Students collaborated in small groups to investigate possible causes of fish death and to develop an explanation that accounted for the evidence provided via the technological resources. Classroom instruction was a mix of whole class and small group activities organized around components-mechanisms-phenomena (CMP). CMP is a conceptual representation adapted from Structure-Behavior-Function theory (Vattam et al. 2011; Hmelo-Silver et al. 2007; see also Quellmalz et al. 2009). In brief, phenomena are the problems or patterns to be explained (here, the sudden fish death in the pond). Components are the individual entities in the system (e.g., fish), and mechanisms are characterized as causal explanations of how phenomena occur or how significant processes work (e.g., cellular respiration). The curriculum materials and technologies were designed to help students use CMP as a tool for systems thinking. For example, in the curriculum unit used here, the phenomena to be explained was the driving question. To explain this, the students investigated the mechanism of eutrophication in which fertilizer washed into the pond, caused an algae bloom, which depleted the dissolved oxygen and killed the fish. Fertilizer, algae, and fish are examples of components.

Technologies

Simulations, modeling tools and hypermedia were an integral part of the curriculum that promoted the usage of CMP as a conceptual tool to make sense of problems in the aquatic ecosystem. In particular, simulations provided opportunities for students to interact with mechanism and phenomena. Hypermedia provided background knowledge that was organized around functions and mechanisms in aquatic ecosystems (see Eberbach et al. 2012 for additional information on the simulations and hypermedia). Modeling tools provided occasions for students to integrate their CMP understanding by connecting across different system levels. Simulations and modeling tools have features that can facilitate the deeper engagement demonstrated by collective cognitive and conceptual-to-consequential engagement. In particular, simulations encouraged cognitive engagement in terms of planning and monitoring, as groups engaged in generating and testing their conjectures through several phases of running simulations related to the aquatic ecosystem. In addition, the simulation was intended to support groups’ exploration of the potential causes of fish death, affording conceptual connections to the unit’s driving question, facilitating CC engagement. The development of explanatory models through the EMT software promoted CC engagement, as it necessitated collaborative groups’ individual content connections, as well as the creation of a broader and elaborated explanation across the concept map in solving the critical problem of the cause of fish death. The planning of particular connections was not clearly a scaffold as part of the EMT software. Further, while both the simulations and the EMT software were similarly structured to support CC engagement, the conceptual-to-consequential connections during modeling with EMT required deeper-level connections than within the simulations to count as high quality.

Participants

Ten videotaped collaborative groups are the focus of this research. These groups were comprised of 36 students from the larger sample (N = 109). Students were grouped heterogeneously to represent mixed gender and ability. Each group consisted of three to four students. The two teachers involved in the project had been teaching science for more than 10 years. One teacher had 4 years of experience working with the technologies described in the study and the other had 3 years.

Measures

To examine the engagement of the videotaped groups, we selected ten observations of groups working on common tasks. These tasks included groups’ creation of their initial models with EMT in each of the 3 units, engaging in hypothesis testing with simulations, and the revision of EMT models at the end of the unit.

For the purposes of this study, we developed an observation protocol designed to evaluate collaborative group engagement using four dimensions. The quality ratings were grounded in theoretical conceptualizations and existing qualitative coding protocols, but modified for use in this collaborative, disciplinary, and technology-mediated context (elaborated below). Here, we operationalize and provide more detailed information related to each form of engagement (see Table 1).

Table 1 Engagement Ratings

Behavioral engagement

Behavioral engagement refers to the degree of the group’s on-task participation. Specifically, we examined evidence for the group’s shared attention on the task and contributions, active involvement in group activity, and persistence in the face of distraction or heightened challenge (Lee and Anderson 1993; Lee and Brophy 1996). Group members who engaged in off-topic conversation or distracted remaining team members evidenced low quality behavioral engagement (Van den Bossche et al. 2006).

Social engagement

Social engagement refers to positive socio-emotional interactions. Group interactions were differentiated in terms of quality given evidence of respectful, responsive, and cohesive interactions (Linnenbrink-Garcia et al. 2011; Rogat and Linnenbrink-Garcia 2011). In addition, high quality social engagement indicated all group members were equally involved in the task, rather than some group members’ contributions being excluded or ignored (Rogat and Adams-Wiggins 2014). Further, in conceptualizing negative socio-emotional interactions, we integrate Rogat and colleagues’ (2011; 2014) characterizations of one group member’s attempts to dominate group interactions (i.e., directive other-regulation) within a moderate quality rating. Directive other-regulators foster negative socio-emotional interactions given patterns of ignoring, rejection, and exclusion of fellow group members’ contributions (Rogat and Adams-Wiggins 2014; in press; Rogat and Linnenbrink-Garcia 2011) (also see Barron 2003; Eilam and Aharon 2003; Kumpulainen and Mutanen 1999).

Cognitive engagement

Cognitive engagement is measured with a focus on groups’ use of regulation and deep-level learning strategies (Fredricks et al. 2004; Pintrich 2000). Groups cognitively engage when they jointly regulate conceptual understanding of content or disciplinary practice, and their task activity. Given our focus on shared regulation, observations, we focus on group regulation rather than individual group member’s self-regulated learning. We employ a socially shared regulation theoretical lens with attention to cognitive sub-processes by investigating the above regulatory foci in relation to planning and monitoring (Khosa and Volet 2014; Molenaar and Chiu 2014; Rogat and Linnenbrink-Garcia 2011). Planning involves interpreting task directions and setting task and learning goals, designating task roles, eliciting relevant prior knowledge, as well as what steps the group will take to take to accomplish the task and modifications made to initial plans. Task monitoring refers to evaluating content understanding and strategy use, progress toward the task solution, group goals, or plan for completing the task.

Conceptual-to-consequential (CC) engagement

CC engagement on the shared task is meant to reflect group progress toward assigned task problems on a continuum of content connections that range from simple knowledge telling (Scardamalia and Bereiter 1996; Chernobilsky et al. 2004) to using science concepts and practices, related to the unit’s driving question, meaningful task problem, or relating to their real world experiences (Gresalfi et al. 2009). This form of engagement involves attempts made by groups to connect other sources of knowledge and experiences as conceptual tools, and establish content connections in the context of a meaningful problem.

For each engagement dimension, a rating of low, moderate, or high (range 1–3) was assigned to reflect quality of group engagement (see Table 1). Ratings were assigned for each form of engagement with the group being the unit of analysis. This measure afforded a focus on collective engagement, rather than the engagement of individual members of the group.

All videotaped observations were uploaded into qualitative data analysis software for observation and rating. Observations were segmented into 5-minute intervals, beginning when collaborative group activity was initiated (i.e., excluding teacher directions; whole class discussion). Ratings were assigned every 5 min and were accompanied with justifications. Each 5-minute segment was viewed and then paused to allow for the rater’s evaluation of each form of engagement during that time period. This interval was selected because previous measures of behavioral engagement have investigated the degree of on-task behavior in 5-minute segments (Lee and Brophy 1996). In addition, segmenting the videotaped observations using time segments allowed an examination of fluctuations in quality variation over time.

To achieve reliability, the observation protocol was initially piloted using video recordings of project data from collaborative groups that were not included as part of the final sample. During this phase, additional elaboration and detail were incorporated to clarify the quality designations for each engagement dimension. Examples were identified for use in a codebook by the primary coders. After achieving initial consensus on the piloted videos with the third author, the first author rated the full corpus of data. Any questions or clarifications during coding were resolved in the full group and/or with the third author, and any resulting modifications initiated recoding and revision of the engagement ratings. Reliability was obtained with a research assistant on 20 % of these videos following training by the primary coder and gaining consensus on a separate sub-sample of the coded data. Inter-rater reliability was assessed on the assigned quality ratings for all 5-minute segments for each included videotaped observation. An 86 % inter-rater reliability was achieved.

Achievement

In order to determine students’ initial understanding of aquatic ecosystem and how it developed over the course of the unit, we administered pre and post-test assessments. As part of the assessments, students were asked to draw what happens in an aquatic ecosystem. We focus on this measure since it is useful for evaluating student understanding of a combination of system measures. All drawings were coded along multiple dimensions to better understand how interacting structures and processes may affect increasingly complex systems thinking. We developed coding along the following dimensions: Structure-Behavior-Function (SBF) relations, Macro/Micro relations, Biotic/Abiotic relations and Extraneous Structures. Scoring criteria are summarized in Table 2 (see Eberbach et al. 2012). We also provide an illustration of how the coding was applied to student drawings in supplemental materials.

Table 2 Scoring Criteria for Student Drawings

Data analysis

Our goal was to examine engagement quality variation within this collaborative technology-mediated learning environment. Initially, we characterized the engagement quality of the whole sample. Toward this end, we calculated correlations among the four engagement dimensions (i.e., behavioral (BE), social (SE), cognitive (CE) and conceptual-to-consequential (CC) engagement). In preparing the data, we drew on the four assigned engagement ratings for each 5-minute time segment for all ten observations for the ten groups in the sample. Given that these time intervals are nested, and to avoid an overestimation of the correlations, we centered the values within each group for use in calculating the correlations. Importantly, this analytic step facilitated between-group comparisons related to quality of observed engagement and informed group case selection.

In a second set of qualitative analyses, we sought to construct a rich description of collaborative groups’ engagement quality when working with the technology tools, with three primary emphases. First, we aimed to differentiate low and high quality collaborative engagement using thick descriptions through the analysis of two groups. Second, we aimed to explore the interrelationships among engagement dimensions in regards to their reciprocal influence during group interactions. Finally, we sought to analyze how engagement quality and the interrelations among dimensions unfolded over the course of group activity during a lesson. We selected two groups representative of low and high quality engagement across forms of engagement using the engagement means. We opted to select collaborative groups at the end points of quality relative to the full sample since this research is exploratory, with the intent of better understanding quality variation observed when groups engage within CSCL contexts (Firestone 1993).

We focused our examination of group engagement on their work with the modeling tools. Groups created explanatory models for the causes leading to fish death in the form of CMP concept maps. After viewing the video describing the problem of dying fish, the groups constructed models on large posters based on their initial hypotheses. The paper models served as initial points of discussion for the electronic models created using the Ecological Modeling Toolkit (EMT; Vattam et al. 2011). Later in the unit, the groups revised their models and included evidence based on information gathered from multiple sources (i.e., simulations, hypermedia, curriculum materials, whole class discussion). EMT supported groups’ developing understanding of both individual mechanisms and the meta-level concepts related to complex systems (Vattam et al. 2011).

For the purposes of this case comparison, we selected one lesson for close examination of each group’s collective engagement. The selected lesson involved revision of the EMT models during the pond unit. We selected this task because it offered significant potential for facilitating CC engagement by affording opportunities for synthesizing and drawing connections among science concepts and sources of evidence from across the unit (such as information gathered from simulations, hypermedia, and other curricular materials) in ways that related directly to the larger unit problem. In addition, cognitive engagement was further facilitated given the focus on revision of the group’s explanatory model from earlier in the unit in light of additional content instruction and teacher feedback.

This qualitative analysis involved several phases. We returned to the videotape and re-viewed the group interactions. Prior to the observation, the engagement protocol, assigned engagement ratings for the selected lesson, and justifications were revisited. The observation that followed was conducted by the first two authors, with a focus on preparing a narrative that richly described the BE, SE, CE and CC engagement at each of the 5-minute intervals during that particular lesson. In a second phase, and to inform and deepen our interpretation of the group interactions, we examined each group’s final explanatory model for the pond unit that was the subject of the observed lesson. Here, we focused on the developed content connections as well as the causes of fish death proposed in the model. In a third analytic phase, and as an additional data reduction artifact, a graph was prepared to represent each case group’s engagement ratings across the modeling lesson. The visual representations supported our analysis of (1) the trend in engagement quality for each group, (2) fluctuations in engagement quality over the course of the group activity, and (3) the interplay of forms of engagement during phases of the lesson. Next, we revisited the narrative to ensure that the interpretations made across analytic artifacts were complete as well as cohesive. A final phase of analysis focused on case group comparison. Here, we contrasted the engagement quality descriptions in efforts to identify salient differences and to consider how the engagement dimensions were interrelated in fostering both lower quality and higher quality group interactions.

Once cases were developed and group comparisons completed, we examined individual group member’s unit pre and post-test achievement scores; group measures of unit achievement were not collected as part of this study. This measure of achievement provided a separable outcome to investigate whether higher quality engagement, using this multi-faceted, dynamic, shared, and contextualized conceptualization, has benefits for group members’ learning outcomes. In addition, we gain some validity information of our engagement ratings, beyond the concept map artifact, by considering whether high quality ratings were associated with higher individual achievement on the post-test. Some care should be taken in drawing strong conclusions regarding a relation between engagement ratings and achievement, given that the achievement data is at the individual level of analysis.

Results

Engagement quality across groups

To more broadly capture the engagement for the full sample, we examined the relations among forms of engagement as well as means and total scores for each group.

Correlations

An examination of the correlations among forms of engagement suggests that there were interrelations among all forms of engagement (Table 3). First, although behavioral engagement was correlated with social and cognitive engagement, it had only a moderate correlation with conceptual-to-consequential engagement. Consistent with research by Lee and Brophy (1996), this result may suggest that groups’ behavioral engagement alone may not have been sufficient to promote CC engagement. CC engagement was significantly related to cognitive engagement and moderately related to social engagement. In addition, we also observed that social engagement was highly correlated with behavioral engagement. This was supported by evidence from video footage that indicated multiple instances when groups were collectively engaged in off-topic conversations in several segments. During such episodes the groups did not actively work with the simulations or modeling tool to make sense of the assigned problem. It is important to recognize the high inter-correlations among the facets of engagement, which while suggesting relationships within a broader meta-construct, also indicates the need for some caution in considering whether these facets can be distinguish empirically.

Table 3 Correlations among engagement dimensions

Mean engagement scores

We examined the average engagement quality ratings for each form of engagement across groups to get an overview for the full sample. Table 4 shows the average score for behavioral, social, cognitive and conceptual-to-consequential engagement across lessons for each group. The final row also provides sample means.

Table 4 Engagement Averages for Groups

In general, these descriptive statistics illuminated between-group differences, suggesting substantial quality variation in engagement across the ten groups. The sample means for behavioral and social engagement were higher than for cognitive and consequential engagement, and this was a consistent pattern in the mean for all groups. Most groups demonstrated moderate quality behavioral engagement, with the sample mean for social engagement just over a 2. However some groups (such as 4–8) showcased low levels of social engagement. This implied that being behaviorally engaged was not necessarily a pre-requisite for the groups to collaborate on the assigned problem. The lower means for cognitive and consequential engagement may suggest that groups faced challenges reaching higher quality forms within this CSCL context. We drew on the means to identify groups 6 and 10 for follow-up case analyses, as they demonstrated the lowest and highest levels of engagement (respectively) relative to other groups. An examination of the pre-test scores administered before the start of the pond unit suggests that similar level of prior knowledge relevant to aquatic ecosystems.

All groups had access to the same set of technologies but nonetheless differed in their behavioral, social, cognitive and conceptual-to-consequential engagement when engaged with this CSCL context. This prompted us to take a closer look at interactions that highlighted the plans made by group members, the conceptual connections that were established and overall group coordination that facilitated such experiences and allowed us to examine how these engagement patterns related to how groups made use of the technologies.

Quality of collaborative group engagement: contrasting cases

Low Quality Engagement Group

Group 6, a three-member team (Ethan, Elton and James), had the lowest scores across forms of engagement relative to the sample. Overall the group’s low quality BE, SE, CE and CC ratings were apparent during their interaction with the modeling tool. Initial ineffective planning (CE) and a decision to work on the task individually (SE), seemed to provoke much of the ensuing low quality engagement across dimensions. In what follows, we first examine the quantitative engagement ratings, and how these unfolded over the lesson. Next, we provide a description grounded in our qualitative analysis, to characterize the observed engagement relevant to each dimension. Finally, we synthesize across the ratings and qualitative description to consider how the quality of engagement across dimensions mutually influenced one another to explain Group 6’s lower quality engagement.

Engagement ratings

Figure 1 shows Group 6’s engagement ratings over the course of working with the EMT to revise their concept map for the pond problem. The group started working with the modeling tool after 15 minutes of teacher-led whole class instruction. During this introduction, following initial brainstorming of causes for fish death, the teacher provided directions related to how to add components and phenomena to their model using the EMT toolkit. The teacher also provided task specifications regarding the number of components for inclusion and that the model should be completed as a group. Across the group activity, the ratings reflect largely low and moderate quality engagement. Figure 1 shows two different primary patterns. First is the lower quality engagement early in the activity (Time segments 2 and 3), with all four facets of engagement at a one rating in the third time segment. The last 10 min of group activity shows moderate CC, followed by moderate cognitive engagement, accompanied by moderate behavioral and social engagement.

Fig. 1

Group 6’s Engagement Patterns

Behavioral engagement

Group 6 frequently engaged in off-task conversations, reflecting a decision for only one group member to use the EMT software and add to the model at a time. As a consequence, two of the three members of the group disengaged for 70 % of the time during the modeling task. James was the group member who was largely responsible for adding to and revising their concept map. Although James occasionally sought his two group members’ input, for example during initial task planning (Time 1), this involvement was not sustained. For example, in the following excerpt, James worked independently while intermittently engaging in an off-task exchange with Ethan:

James: You tired? (Video shows him talking while adding components to the model).

Ethan: Yeah…And I didn’t even go to bed that late. What time do you go to bed?

James: It varies. If I can’t sleep I’ll like play on my iPod for like 10 min. And then go to bed.

Ethan: Do you have a bedtime?

James: No. Well, my parents want me to go to bed early like 8:30 or 9 if I’m sick. If I’m not then I get to stay up, say if I’m watching a movie like 10:30 or 11. But that’s on weekends.

Ethan’s contributions related to model development remained sporadic. This led to infrequent and inconsistent participation of group members who were only intermittently on-task during this activity. This general pattern that explains the first 20 min of group work shifts for the last five minutes of the lesson. In the final segment, all three-group members work on the explanatory model, perhaps due to an awareness of the short time remaining for completing the task. Overall the group displayed low to moderate quality behavioral engagement, demonstrated by on-task activity by a single group member at a time.

Social engagement

Social engagement during model creation and revision was primarily moderate. These assigned ratings of 2 reflect a climate characterized by low group cohesion. While there were a few initial attempts to initiate a group discussion, with efforts to solicit everyone's ideas, most of the time group members worked on the task independently and one at a time. Analysis of video footage suggested evidence of low cohesion as Ethan and Elton participated in off-task discussion while James worked independently. In addition, group members made multiple references to “I think” while proposing a group hypothesis on why the phenomena occurred, and references to “I’m going to” and “My turn” when planning what components to add during model revision. Further evidence for low cohesion was shown in the text entered into the final model (see Fig. 2). This use of “I” shows a focus on individual thinking, and a conceptualization of the work as individual activity, rather than collaborative.

Fig. 2

Evidence of Low Quality Social Engagement by Group 6

Another indicator of low quality social engagement was that even when group members’ contributions were solicited and/or shared, these ideas were not consistently taken up for discussion or further incorporated into the group model. For instance, group members ignored one another’s questions and ideas; rather than be responsive to a teammate, group members were observed simply adding their own disconnected contribution:

Ethan (reads from a sheet of paper): There also may be smoke from cars and fertilizers getting into the lake. Could there be acid rain in it?

(Elton and James do not respond. Elton steps away after he finishes typing.)

James: It could be a disease. Do we have that?

(Ethan and Elton do not respond to his question. He turns the laptop towards himself and starts typing.)

In general, our analysis of these socio-emotional interactions suggested an understanding that the task was individual rather than collaborative; interactions demonstrated parallel individual efforts to solve the assigned problem. Moreover, group members were not responsive during the few observed episodes of joint activity.

Cognitive engagement

Group 6 demonstrated low quality cognitive engagement. A primary source of this low level CE stemmed from initial task planning, when brief attempts left plans vague and incomplete, with the group not developing specific plans. Moreover, the initial planning suggested a misinterpretation of the task directions and purpose:

Ethan: Why do you think this is happening? (Referring to causes of fish death)

Elton: Low oxygen.

Ethan: Should I just list the reasons?

Elton: Well remember what we did yesterday with the evidence?

Ethan: Yes. (During this time James was writing on a sheet of paper. Ethan had the computer facing him while typing. Elton periodically looked at the computer.)

This example shows that the group’s planning remained unspecified. Ethan’s stated task goal for listing reasons suggests a simplifying and misinterpretation of the modeling task. The intended purpose of generating the list remains incomplete, as the group did not discuss possible interconnections between components that led to conditions resulting in the death of fish. It seems that Elton then made an attempt to respond to Ethan’s planning by connecting back to the previous lesson’s whole class modeling activity to shape the group’s plan. In the previous lesson the whole class developed a consensus model as an example to inform their own specific group model. However, Elton’s response only generally evoked the previous lesson, without providing any details or feedback about what to remember and draw from that experience to inform their group plan (i.e., low quality CE). Ultimately, the plan they developed remained focused on developing a list of reasons for low oxygen, rather than explaining fish death.

Following this exchange, the group began enacting their plan to generate a list of reasons for low oxygen. Subsequent group activity suggested that one consequence of the initial low quality planning resulted in subsequent low quality task monitoring:

Elton: You forgot to write an “a” here. (Pointing to a spelling mistake)

Ethan: In the pond there may be pollution or chemicals from the factory. I also think that the green mucky.... Go ahead Elton. (Slides over the laptop to Elton)

Elton: Ok, I’ll type.

Ethan (reads from a sheet of paper): There also may be smoke from cars and fertilizers getting into the lake. Could there be acid rain in it?

Here, the task monitoring was focused on spelling of components. Similarly, we observed low quality cognitive engagement as group members focused planning on who should type or add contributions to their model, rather than its content.

Taken together, low quality planning yielded activity that focused on listing factors responsible for low oxygen levels in the water, rather than developing an explanatory model for causes of sudden fish death. Specifically, Ethan and James identified multiple causes such as pollution, chemicals from the factory, and green mucky water. They also listed smoke from cars, fertilizers getting into the lake, and acid rain (see Fig. 3). During the last 5 min of the class period, the group shifted in focus, yielding some moderate quality cognitive engagement. The group worked on connecting different components such as “population of fish” to the “fish are dying” component (see CC engagement below). The group discussed the direction of arrows, aligning components so that they pointed towards the “fish are dying” component. They also maintained a superficial focus by discussing the colors of boxes. Low quality planning seemed to be a primary reason that the modeling task devolved into creating a list. Further, monitoring may have remained low quality as a consequence of this initial planning as well as not having a common understanding of the purpose of the modeling activity. Thus, group members could not apply criteria to monitor the quality of content connections or organizational structure.

Fig. 3

Group 6’s Explanation for the Fish Problem

Conceptual-to-consequential engagement

The group displayed low quality CC engagement while working with the EMT to uncover the factors leading to fish death. A fundamental challenge relevant to CC was that the group’s work on the modeling activity did not seem to connect to the larger unit problem explaining fish death in a pond, but remained narrowly focused on a single factor. Group 6’s joint activity demonstrated three primary indicators of low CC engagement. The group generated connections that were not substantiated by the available evidence or accessed via the conceptual or disciplinary tools. For example, individual group member’s added separate and disconnected possible factors that could have caused low oxygen such as smoke, pollution from the air, and the presence of fish disease. The available data sources did not direct attention to these factors listed by the group (i.e., information from simulations, data related to water quality, or hypermedia). Moreover, Group 6 did not provide rationales backed by evidence in their discussions.

An examination of the group’s final model provided further evidence for the primarily low quality conceptual-to-consequential engagement (Fig. 3). First, the group stated in the explanation box (located on the top left hand corner of the model) that they thought low levels of oxygen led to fish death. Low levels of oxygen were in the example provided on the handout that accompanied the simulation. It is notable that the group did not extend beyond this initial conceptualization of the modeling task. This was evident as components, such as carbon dioxide, nutrient run-off, and dead matter was connected to oxygen, rather than fish death. Connections established between components were superficial, as the explanation boxes simply reported simulation outputs and observed relationships without linking to the fish problem (e.g., “If the dead matter increases, oxygen would decrease” and “If oxygen increases, carbon-dioxide decreases”). Second, it appeared that the group also explored the possibility of alternative causes of fish death, such as decreased quantities of food and presence of carbon dioxide. However there was no evidence in the curricular resources that supported their reasoning that fish could have died due to these factors. A final indicator of their lack of coherent connections is that the group added multiple representations of the fish component with varying properties. Ultimately, Group 6’s concept map primarily focused an aggregation of individual connections that did not relate to the larger problem. Taken together, Group 6 did not seem to use the available conceptual, scientific, and instructional resources in consequential ways to solve the larger problem of fish death in the local pond.

Interrelationships among engagement dimensions

The separate examination of each engagement dimension was an important step in understanding the challenges faced by Group 6. In this final section, we consider how the mutual influence among dimensions explains Group 6’s lower quality engagement. Initial low quality planning decisions in the first 5 min seemed to set the stage for the remainder of group activity, and the predominant pattern of low-level engagement. Here, the group misinterpreted the purposes of modeling as a list of reasons, and only engaged in very brief attempts at task planning, which together posed significant challenges (i.e., cognitive engagement). Concurrently, low quality social engagement augmented the problem with group members working independently to add components, rather than jointly coordinating their task work. This continuing independent activity fostered low group cohesion. These initial decisions interrelated with disengagement for group members not currently contributing to the concept map (i.e., low behavioral engagement) as well as low-shared commitment or goals for working together. (i.e., low social engagement). Our analyses also suggested that the low quality planning provoked further challenge for subsequent cognitive engagement related to monitoring. Here, the group monitored themselves superficially without examining content connections or areas for improved organization. These three facets appeared to jointly diminish the potential quality of CC engagement. In particular, CC engagement was restricted to individual facts and reporting of information, without interpreting it in the context of the given problem.

High quality engagement case

Group 10 consisted of four members (Matt, Kylie, Joshua, and Maya) and showed the highest-level engagement relative to the sample. This high quality engagement reflected generally shared on-task activity and responsive positive social interactions. However, what differentiated this group and proved to be a hallmark of Group 10’s engagement was the maintained and consistently high quality cognitive and CC engagement when modeling. Specifically, high levels of CC engagement led to successful integration of information gathered from multiple data sources. This resulted in strengthening the group’s conceptual understanding of the problem.

Engagement ratings

The group showed high and moderate quality engagement on all dimensions across the lesson, with no assigned low quality ratings (see Fig. 4). High quality behavioral engagement was maintained across the lesson. The first half of the observation evidenced high quality social engagement, followed by the second half’s moderate levels. Cognitive and CC engagement ratings ran in parallel, with the exception of the last time segment. It is noteworthy that for four of the six observation ratings, Group 6 was rated a 3 for CC engagement.

Fig. 4

Group 10’s Engagement Patterns

Behavioral engagement

Video recordings showed that members of Group 10 displayed an overall high level of behavioral engagement. Given the lack of variability during the lesson, our description of on-task participation is brief. During the modeling task, all group members remained on-task and engaged in limited off-topic conversation. In addition to being focused and attentive, the entire group was involved in jointly working towards finding a solution that would help explain the problem of fish dying in the local pond. Finally, when alternative perspectives surfaced within the group, all group members continued to contribute to the shared product.

Social engagement

Group 10 displayed moderate-high level social engagement over the course of the lesson. The group’s social interaction can be characterized by Matt taking on a role in facilitating the group’s responses on their shared model. For example, it was common for Matt to introduce a concept or mechanism for inclusion in the model and present it to the group for discussion as to whether everyone agreed to integrate the concept. He solicited each group member’s opinion, even if it conflicted with his ideas. In this way, Matt set a tone for being respectful and responsive in interactions within the group, as well as one of equal participation:

Matt: Yes, yes okay because when the algae grew on the fish's skin, that's a possible way they could have died right?

Kylie and Joshua: Yes.

Matt: I agree with this. How about you Maya? Do you agree with it?

Maya: Yes.

Matt’s facilitation of group interactions was effective in that group members typically responded to his idea for inclusion in the model. Matt also fostered a sense of cohesion within the group, often referring to the collective and using “we.” In some instances, Matt employed the use of I when introducing his idea, but then returned to using “we”, suggesting some sensitivity to acknowledging the import of the collective or group. On a few occasions we observed tension among group members, which seemed to stem from Matt’s central role in generating ideas and making some edits to the group model without consulting others, reflecting moderate quality social engagement. This tension and difficulty with Matt’s perceived direction during group work resulted in some disrespectful exchanges marked by mimicking and ignoring. However, when group members Kylie and Joshua introduced concepts and mechanisms for inclusion, Matt remained responsive in discussing and integrating these ideas. Taken together, the group typically worked collaboratively on their model in ways that were inclusive of everyone’s ideas. This positive social engagement was primarily facilitated by Matt.

Cognitive engagement

Group 10 demonstrated high quality cognitive engagement marked by both high quality planning and task monitoring. The group engaged in high quality planning during the modeling task by taking a step back early in the group work to discuss the larger purpose and goal for their model. Their planning discussion focused on the purpose of the scientific practice of modeling, accompanied with specifics regarding what this model needed to explain. This planning was in response to monitoring when the group was discussing more general causes of fish death based on their outside knowledge:

Kylie: Cause the cleanliness of the water affect the fish. If the water’s dirty then the fish die. They…

Matt: Ok…may I explain something? This model (points to their model on the screen) is not every single possible way a fish can die. It only explains how the fish die in this particular…

Maya: No it’s just an idea.

Matt: So to tell you the truth the only possible explanations can be the amount of algae affecting the..

Maya: It’s the ideas of how they could have died.

Matt: Can I explain something now? She [the teacher] wants us to explain how the fish could have died now. Not what we thought before or the possible ways. Unless you think that the fish died as the water was dirty, after you see the evidence, then I will put it in.

Here, the group discussion coalesced around a plan that informed their subsequent activity. In particular, the group understood that the model needed to explain how these fish in the pond within the unit problem died, not all fish. The group maintained a focus on high quality planning by revisiting this previously established goal later during the task. A further indication of the quality of CE was that the group’s task monitoring was grounded in their initial understanding of modeling stemming from their initial plan:

Matt: To tell you the truth, in my opinion, even though chlorophyll and nitrates were present in our data, they are not really necessary. Wouldn’t you agree? Chlorophyll and nitrate, even though they are a part of the algae they are not really necessary to explain why fish died…It says it is washed into the rain. Does it say what effect it has on the pond? No…or why the fish died? So do we agree that we can take the two components out? (Referring to chlorophyll and nitrates)

Kylie: Well, we can take chlorophyll out.

Matt: What do you think? (Turns towards Maya)

Maya: We can take chlorophyll out.

Matt: Should we take nitrate out?

Maya: I don’t think so. Did we find anything important to nitrate?

The above task monitoring was beneficial since it refocused the group on the importance of explaining fish death when planning the specific components to include. Further, the group’s included relationships and components remained central to the larger problem of fish death as a consequence of this monitoring. Task monitoring also focused on checking that relevant evidence was included as justification and was drawn from available resources. Notably, their discussion maintained a focus on monitoring the development of explanations, not other superficial features. Here, the group invested efforts consistently across the class period to engage in revision and modification of their explanatory model in light of their task monitoring – an indicator of high quality cognitive engagement.

Conceptual-to-consequential engagement

Group 10 can be differentiated by their maintained high quality CC engagement. The group’s joint activity focused on explaining the larger unit problem. During the task, Group 10 grappled with and negotiated their understanding of varying explanations for fish death. Although early in the group activity they considered the role of the cleanliness of the water, the group shifted to consider the role of algae in decreasing levels of oxygen, with implications for the fish. When Group 10 introduced an explanation they evaluated the potential of the explanation to have caused fish death given available evidence gathered from the full range of data sources (i.e., video reporting the fish problem; information provided about temperature and quality of the water; fish necropsy reports). In addition, the group coordinated their efforts to consider how to best account for the evidence and include their evolving explanation within the model:

Kylie: Then, how does the algae affect the water if it’s affecting the fish?

Maya: It’s on the fish’s skin.

Matt: Well, it made the water look green but it didn’t affect the fish.

Kylie: Then that means that the algae affected the water.

Matt: Well the algae and the fish affected the water. The fish caused the smell and the algae caused the green.

Kylie: But you said that the fish affects the algae, so wouldn’t there be a line there? (Points to the components algae and fish on the screen)

Matt: No, I don’t think that the fish affect the algae. So maybe we should just get rid of this line all together? (Points to the line between fish and algae)

Kylie and Maya (together): No!

Matt: So what do you think about the connection between the fish and the algae?

Joshua: The algae affect the fish.

Matt: Yes, yes, ok because when the algae grew on the fish’s skin, that’s a possible way they could have died right?

Kylie and Joshua: Yes.

Members of the group justified their algae focused explanation based on the fish necropsy that reported that algae were found on the skin of the dead fish. Although they do not explicitly reference the necropsy report, the group questioned this hypotheses (that algae caused the fish to die) as lack of oxygen would have led to death to algae as well, which contradicted the evidence presented to them from the video where they saw abundant algal bloom on the water making it green in color. In addition, the group questioned the consequences of the behavior of algae that led to the phenomena. They justified exploring this line of thought based on the evidence gathered from the curriculum data. It is interesting to note how their interpretation of decreased oxygen led them to question the role of algae.

The group maintained high quality CC engagement by consistently integrating evidence gathered from multiple data sources - with implications for revising their developing explanation. Modifications were informed by peer feedback (see social engagement example) as well as new hypermedia resources, introduced mid-activity, which informed the role of nutrients. In the exchange below, Group 10 drew on the hypermedia in combination with other sources as grounds to shift their explanation to include nitrates as a cause for sudden fish death. In the exchange below, the group included nitrates as a component and discussed its mechanistic behavior in the context of the larger problem. This led the group to consider the likelihood that other factors may have led to the fish death:

Matt: I don’t think anything’s important to nitrate.

Maya: On the hypermedia?

Matt: All it says on the hypermedia is that it gets washed into the pond. It doesn’t really say what it does.

Maya: Let’s go through it once again. Click home. (Matt opens the hypermedia homepage)

Kylie: (reading from the screen) What is the role of nutrients?

Matt: Ok, here it is. (reading from the hypermedia) Living things use carbon and nitrogen to build and repair their bodies and carry out important processes…

Kylie: So wouldn’t the algae use the nitrogen to grow?

Matt: Ok, now that we found that we can add it [into our model].

Beyond Group 10’s maintained focus on explaining fish death, this excerpt highlights that they consistently worked to ensure their model could be justified using the evidence drawn from the resources. For instance, information gathered from the hypermedia along with experimentation with simulations led the group to disregard factors such as chlorophyll and refocus on factors such as nitrates and decomposing bacteria as pertinent to the problem. Matt indicated that although information about those specific components was presented as evidence, it was insufficient to tie it in to cause of fish death.

Group 10’s high quality CC engagement can also be detected in analysis of their final model (see Fig. 5). Their model extended beyond identifying relevant components to discussing mechanistic behaviors of those components in the context of the given problem. Group 10’s model presented the possibility that interaction between multiple components was critical to explaining the phenomena. The modeling process resulted in the inclusion of components such as decomposing bacteria, fertilizers, and nitrates. Interpretation of behavior of such components (based on explanations in the connecting boxes between components) indicated that the group portrayed their understanding of the eutrophication process that led to fish death. Overall, we observed consistent high quality CC engagement as Group 10 focused their model generation on explaining the broader problem using interrelations among components, as well as revising the model in light of their evolving understanding of the problem.

Fig. 5

Group 10’s Explanation for the Fish Problem

Interrelationships

Two primary interrelationships help explain the high engagement quality demonstrated by Group 10. First, this group showed consistent high-level behavioral and social engagement. It seems likely that because the group maintained on-task participation, they were able to maintain a shared focus on improving their conceptual artifact (the EMT model). Further, under Matt’s facilitating role within the group, group member’s contributions and perspectives were respected and considered for inclusion in the explanatory model. Here, both BE and SE seemed to be a critical undercurrent for reaching high levels of CE and CC. For instance, the group’s participation and responsive interactions allowed them to develop and endorse a shared group plan.

The second pattern concerned the interrelated nature of CE and CC, with high quality cognitive engagement proving central to promoting and sustaining the group’s consequential engagement. Group 10 devoted time to developing a task plan that considered the purpose of modeling and the focus of this particular model relevant to the unit problem. This initial plan ensured that group members understood the goal of the concept map, and informed their monitoring throughout the activity. The plans they developed and their monitoring were essential for facilitating consequential engagement. The group showed continued willingness to undertake revisions to the explanation, to be more complete and representative of the evidence and curricular and technological resources accounting for larger problem of fish death, and to explicitly introduce the rationale for an integrated mechanism or connection.

Achievement results

We can also explore whether these between-group differences in described engagement related to differences in learning outcomes. To examine this question, we consider students’ scores on individually completed pre and post-tests (see Method for additional information on the unit tests) (Table 5). While both groups showed similar pre-test scores, members of Group 10 scored highest on the post-test, with a majority of group members obtaining scores of 3 s across item types (M_{Group 10} = 2.89). In contrast, members of Group 6 showed variation in post-test scores, with one group member showing a mode of 3 and the second group member having a mode of 1 (M_{Group 6} = 2.38). Ultimately, there is some evidence for a relationship between collaborative groups’ engagement quality with individual group member’s learning outcomes, demonstrating benefits for high quality engagement.

Table 5 Individual Pre and Post-test Achievement Scores for Case Groups

Discussion

In the current study we examined engagement quality of collaborative groups within a CSCL context using a new observational measure intended to explore the multi-faceted, dynamic, shared, and contextualized nature of the engagement construct. Our methodology combined the use of ratings operationalizing engagement as multi-faceted and collective, with qualitative analyses that provided rich description, and afforded an examination of the dynamic relations among these facets over the course of activity within a technology-mediated collaborative inquiry activity. With these methods, we explored the quality variation of 10 collaborative groups’ behavioral, social, cognitive and conceptual-to-consequential engagement. Our initial exploration of the full sample of 10 groups enabled the consideration of whether there was range in quality variation across groups. By developing cases of two groups characterized by high and low quality engagement we more richly describe the nature of these quality distinctions, showcasing these four dimensions as evolving and interrelated in their nature. This contributes to an engagement in CSCL literature that is limited in scope (Dillenbourg et al. 2009; Järvelä and Hadwin 2013), despite the acknowledged affordances and challenges to fostering engagement within technology-mediated inquiry contexts (e.g., Blumenfeld et al. 1991, 2006; Quintana et al. 2004; Soloway et al. 1992). Further, extant research has operationalized engagement using single dimensions, as stable, and as characteristics of the individual learner, as well as decontextualized from conceptual and disciplinary tasks.

One limitation of the present study is the small sample size. A larger participant pool would likely introduce a wider spectrum of engagement trends. Similarly, we closely investigated two groups for our case analysis, using groups found at two ends of the engagement continuum to ground our conceptualization of engagement. Future research should extend the study of engagement to include groups representative of the full range of quality. Gaining access to the challenges faced by groups demonstrating moderate engagement quality would help to elaborate our characterizations of engagement, as well as the interrelationships among the different facets that may jointly explain the reasons behind these identified challenges, with implications for supporting these groups in more targeted ways (Rogat and Linnenbrink-Garcia 2011, 2013). In addition, our observations were of two technological tools, with our cases focused on the EMT software. It will be important to richly describe variations in engagement across varying types of tools to inform design and gain insight into challenges groups face. A better understanding of these facets should be productive in designing scaffolds to help orchestrate activity and support deep engagement in CSCL environments that make use of multiple technologies.

In applying the observation protocol, we characterized the engagement of groups at 5-minute intervals. These quality ratings afforded capturing the primary engagement quality for that time period and allow us to add to the total sample of observations for each group in the study. This interval is fine enough to allow us to observe overall variations in engagement patterns within and between groups. However, we lose information regarding the moment-to-moment nature and fluctuations in engagement, as well as detail relevant to the contextualized nature of engagement. We tried to partially address this limitation by returning to the videotape for descriptions to explain the observation ratings, however, coupling discourse analysis or thick qualitative analysis in future research would further deepen and contextualize these conceptualizations in disciplinary and technology-mediated activity. Further, our analysis of fluctuation in engagement considered change within a single lesson, with findings indicated evidence for the evolving nature of engagement within activity. However, there is added value to tracing the dynamic nature of engagement over the course of several lessons, the duration of a unit, or across units. Finally, given the nature of observational protocols, we made decisions to narrow our focus but this has meant the exclusion of other relevant constructs. In particular, we evaluated engagement in the collaborative group and the interactions among group members. While we have provided some description of the curricular and task context, our methodological choices meant that we have excluded observation of group members engaging within the larger whole class context and the teacher, as well as the teachers’ use of other instructional practices. We also did not account for group member’s individual activity, given our interest in the engagement at the group level. Future research would benefit from exploration of the reciprocal relations between individual and group engagement (Järvelä et al. 2010; Ryu and Lombardi 2015).

Unpacking group engagement in CSCL contexts

We developed an observational protocol that operationalized engagement using four dimensions. Here, engagement integrates aspects of participation and socio-emotional climate of the group (i.e., behavioral and social engagement) with the regulatory and learning strategies, and means of consequentially engaging with the activity (i.e., cognitive and consequential engagement). One contribution of this framework is that engagement is operationalized with the group as the unit of analysis, promoting a view of engagement as shared among group members. We apply theories of shared activity and knowledge co-construction to advance views of engagement (Roschelle and Teasley 1995; Suthers 2006). Our results highlight the shared nature of engagement given Group 10’s high quality planning, monitoring and connections as best characterized as resulting from joint and mutual negotiation within the group. Here, studying engagement as a group-level phenomenon also means that it is inextricable from the individual, and highlights a role for how interactions within the group context influence its quality (Pintrich et al. 2003; Rogat and Adams-Wiggins 2015; Rogat and Linnenbrink-Garcia 2011, 2013). Participation and the socio-emotional climate either supported or impeded the content connections and solutions to the authentic problem negotiated within the group. In particular, the disjointed and incoherent social interactions impeded CC engagement for the group exhibiting low quality engagement, but the responsiveness and respectful interactions augmented the strategic and consequential engagement of the high group in their consideration of multiple causes that may have led to the fish death.

A second contribution is the extension of Fredricks, Blumenfeld & Paris’ (2004) framework to incorporate conceptual-to-consequential engagement as a higher quality form that supports a contextualization of engagement in authentic activity, as groups work to solve meaningful problems. Our case results for Group 10 characterize CC engagement as improving several explanations for fish death, with a final explanation evolving from the group’s consideration of the available resources, as well as building from one another’s ideas. Further, this group reflected on the problem as a complex web of cause and effect relationships based on observed behaviors of relevant components. In contrast, Group 6 demonstrated low quality consequential engagement, given their final model and discussion focusing on declarative knowledge, disconnected facts explaining levels of oxygen, as well as the inclusion of components not supported by the evidence. Integration of consequential engagement elevates conceptualizations of regulation and connections, to consider group’s reflections on the larger picture and resolving of the driving question. Further, our specifying a continuum from low quality focused on disconnected and declarative facts to higher quality linkages between prior knowledge, experience, resources and a meaningful problem supported the differentiation between the low and high group cases.

Our findings have implications for how we conceptualize the relations among forms of engagement. In particular, our results suggest interrelations among behavioral, social, and cognitive forms of engagement, with subsequent influence for groups’ CC engagement. Here, we provide a review of Group 10’s case that builds toward these points. First, for Group 10, we see on-task participation and a positive climate as setting the stage for higher quality CE and CC engagement. Here, broad participation and sustained on-task engagement ensured mutual attention over the course of activity. Further, positive socio-emotional interactions, reflective of responsive interactions and the equitable solicitation of ideas, ensured that group member’s ideas were taken up and integrated within the group response. It is notable that the resulting positive interactions and inclusiveness required continued effort by Matt to ensure that there was agreement among group members related to the components and relationships integrated into their shared explanatory model. Future research should continue to examine the role of social engagement for engagement quality and more generally for group activity.

Previous CSCL studies have considered the degree to which group members participate and issues related to group dynamics (e.g., Guzdial and Turns 2000; Lipponen et al. 2003; Salomon and Globerson 1989; Stahl 2000; Van den Bossche et al. 2006). As discussed above, these two facets did help to differentiate engagement quality for Group 6 and 10. However, Group 10’s deep-level engagement was more than everyone’s participation and responsive interactions. It was our addition of CE and CC that enriched and elaborated our description of the deep-level engagement showcased by Group 10. Future research should explore the threshold at which behavioral and social engagement must be attained in order to sustain high quality CE and CC engagement.

We find that the observed high quality behavioral and social engagement seemed to facilitate Group 10’s cognitive engagement. The group jointly developed a plan that considered the purpose of the model, and help to maintain a shared goal of developing an explanation for fish death. It was this high quality plan that also ensured that their monitoring went beyond superficial checking, to consider accounting for the set of resources and evidence. The group’s monitoring encouraged multiple rounds of revision and consideration of varying explanations for fish death. Ultimately, it seems that this willingness to revise and evolve their model that promoted the group’s consequential engagement. However, we can also see consequential engagement as related to the synergistic influence of these engagement dimensions, affording a focus on improvement of their shared explanatory model in ways that answered the driving question. Taken together, our findings culminate in a synergistic view of engagement (also see Rogat and Linnenbrink-Garcia 2013). Our results give primacy to the highly interrelated and mutually influencing nature of these four dimensions of engagement.

Our findings provide some initial evidence that a multi-faceted conceptualization of engagement, and its operationalization as shared, dynamic, and contextualized, affected the case group’s understanding of aquatic ecosystems, as demonstrated by their final explanatory models and their individual post-test achievement scores. This contributes to extant research by suggesting a relationship between the quality of collective engagement and individual achievement.

Implications for design and instruction

Based on our findings, we present suggestions for refining the design of these technologies to promote groups’ cognitive and conceptual-to-consequential engagement. For instance, there is potential to redesign the modeling tool to scaffold inquiry-based practices to address the pond problem. Built-in prompts can appear on the screen when groups add new components or write explanations connecting two components. These prompts could sustain cognitive and conceptual-to-consequential engagement by having the groups consider the relevance of the component in the context of the larger problem, guide them to identify and cite the source of evidence that led them to consider a particular factor, and think about their observed behavior and function in the complex system. However, designers need to carefully consider the conditions under which those prompts might appear to create a balance between encouraging thoughtfulness and interfering in the flow of the collaborative work. In addition, the teachers can reinforce the idea that the modeling tool is a medium for the group to evolve and revisit their conceptual understanding.

Given that groups worked on model creation synchronously, it may have been a challenge for the teachers to monitor the conceptual and scientific practice understanding, as well as progress made by each group. For instance, without close monitoring of Group 6’s activity, it may not have been clear that the group was generating a list of reasons for low oxygen levels as the primary cause of fish death. Teachers would benefit from tools and educative materials that would allow access to log data or a means by which they could examine evolving models created by the groups in order to diagnose significant areas of challenge.

Future research

There is a general concern that schools do not give students opportunities to engage with curricular content in conceptually and consequentially meaningful ways (Gresalfi et al. 2009). Designing such rich learning environments is a challenging task. Evaluating student engagement as part of the design process in such complex learning environments may help in overcoming this challenge. This study is a step towards characterizing groups’ engagement in curricula that encourages such high quality engagement. Specifically, our conceptualization of engagement helps to tease apart influences and interactions between various kinds of engagement that have a bearing on uptake of technological affordances. The study identifies factors (such as design of technological tools, curriculum and structure of group interactions) in the CSCL environments that have the potential to promote positive participatory practices.

This study is important to the field of CSCL as it adds to the literature on inter-subjective meaning making (Koschmann et al. 2003, Koshmann et al. 2005, Rochelle 1996; Stahl 2004; Suthers 2006). Specifically our findings show how engagement acts as a lens to highlight aspects of the joint composition of interpretations, in the form of “predictions, commentary, expressions of attitudes, expressed verbally, gesturally, or through manipulations of representations” (Suthers 2006; p. 7).

Conclusions

CSCL environments are complex and attempts at understanding them need complex conceptualizations of how and whether groups take up the technological affordances in productive ways (e.g., Kapur et al. 2011; Teasley 2011). We argue that to richly conceptualize collaborative engagement in computer-supported contexts we need to draw on a multi-faceted, shared, and contextualized operationalization that extends beyond participation and group socio-emotional interactions. Our results show that these forms of engagement are interrelated and that the quality is mutually influential. Moreover, high conceptual-to-consequential (CC) engagement is facilitated by the synergistic influence of behavioral, social, and cognitive engagement dimensions. The CC dimension is especially important in computer-supported inquiry learning because we want learners to use the technology to go beyond building knowledge for its own sake (Chan 2013). Rather, the goal is knowledge building for action in which learners use knowledge as a tool for thinking (Hmelo et al. 1998).

Appendix

Example Application of Scoring Criteria

To illustrate how the coding was applied to student drawings, we examine the pre and post-test drawings of a participating student (See example in figure below). We applied the Macro/Micro code as Level 1 in the pre-test example because all structures (e.g., fish, coral, seaweed) are macroscopic, whereas the posttest example is coded as Level 3 because the student identifies relations between macro and micro levels (e.g., fish and ammonia, algae and oxygen). We applied the Biotic/Abiotic code as Level 1 in the pre-test example because the student drew a largely biotic scene and included only one abiotic structure (ocean floor).

In the posttest example, we coded this as Level 3 because the student included examples of biotic and abiotic structure relations (e.g., algae and sunlight; bacteria and nitrate). In both drawings, no structures were deemed irrelevant so Extraneous Structures was coded as Level 1 for each. For SBF, the pre-test example was coded as Level 2 because the student related components and mechanism relations (e.g., starfish eats the clams; fish lives in the coral). In the posttest example, the student reached Level 3 of the SBF code (e.g., sunlight causes algae to grow links to algae makes oxygen for fish).

* Note: Student’s drawing at pretest (left) and posttest (right) with student’s explanatory labels in red.