Analytic Frameworks for Assessing Dialogic Argumentation in Online Learning Environments

Abstract

Over the last decade, researchers have developed sophisticated online learning environments to support students engaging in dialogic argumentation. This review examines five categories of analytic frameworks for measuring participant interactions within these environments focusing on (1) formal argumentation structure, (2) conceptual quality, (3) nature and function of contributions within the dialogue, (4) epistemic nature of reasoning, and (5) argumentation sequences and interaction patterns. Ultimately, the review underscores the diversity of theoretical perspectives represented within this research, the nature of dialogic interaction within these environments, the importance of clearly specifying theoretical and environmental commitments throughout the process of developing or adopting an analytic framework, and the role of analytic frameworks in the future development of online learning environments for argumentation.

While research on technology-enhanced learning environments often focuses on human–computer interactions, technology-enhanced learning environments also provide powerful affordances for scaffolding human–human interactions. Over the last decade, sophisticated online learning environments have been developed to support students engaging in dialogic argumentation. Dialogic argumentation focuses on the interactions of individuals or groups attempting to convince one another of the acceptability and validity of alternative ideas. Engaging students in dialogic argumentation is considered a powerful mechanism for increasing students’ understanding of challenging concepts (e.g., Andriessen et al. 2003; Hogan et al. 2000; Leitão 2000; Driver et al. 2000) as well as for increasing students’ ability to engage in productive argumentation and reasoning practices (e.g., Baker 2003; Bell 2004; Kuhn et al. 1997; Teasley 1997). This review highlights the foci, affordances, and constraints of several different analytic frameworks for assessing dialogic argumentation in order to help researchers (1) understand the multiple facets of interactivity in online argumentation environments and (2) refine analytic frameworks and approaches for their own research and development.

Harnessing Online Learning Environments to Promote Argumentation

Online argumentation environments include a broad range of specific instructional features to promote productive interactions between participants, including collaborative communication interfaces, co-creation and sharing of intellectual artifacts, enriched access to information, and scripting and awareness functionalities. Methodical integration of these features can potentially facilitate active learning and productive interaction beyond what can be achieved in more traditional learning environments (Fabos and Young 1999; Marttunen and Laurinen 2001; Pea 1994; Roschelle and Pea 1999; Schellens and Valcke 2006).

Collaborative communication interfaces

Online learning environments incorporate specialized asynchronous and synchronous collaborative communication interfaces to support interactions between students. Asynchronous modes of communication (e.g., CSILE, Netmeeting, Allaire Forum) allow learners to participate more equitably and to spend more time constructing well-conceived and elaborate arguments (Joiner and Jones 2003; Marttunen and Laurinen 2001; Scardamalia and Bereiter 1994; Schellens and Valcke 2006; Veerman 2003). Synchronous modes of communication (e.g., CONNECT, TC3) can deliver a higher degree of joint elaboration and construction of arguments as students work on a common shared artifact but these synchronous modes place higher demands on learners’ abilities to interpret challenging conceptual material (e.g., de Vries et al. 2002; Janssen et al. 2006).

Co-creation and sharing of intellectual artifacts

Some online learning environments incorporate tools that enable students to co-create and share intellectual artifacts, such as shared text documents (e.g., de Vries et al. 2002; Erkens et al. 2003) or concept maps that visualize arguments (e.g., Kirschner et al. 2003). These external artifacts guide learners to compare and refine their ideas through a process of dialogic argumentation. For example, the DUNES system (Schwarz and Glassner 2007) encourages students to engage in dialogic argumentation as they co-construct rich argumentation maps in which shapes represent types of contributions (e.g., information, argument, comment, or question) and arrows between shapes show connections.

Enriched access to information

Another function for promoting productive argumentation involves providing students with enriched information, such as visualizations (e.g., TELS and WISE) and knowledge bases (e.g., CSILE) to help students justify, evaluate, and potentially increase the persuasiveness of their arguments (see Oestermeier and Hesse 2000). Knowledge bases can range from glossaries embedded within specific online learning environments to the greater World Wide Web, or specified sections thereof, such as online libraries (e.g., Kolodner et al. 1997).

Scripts and awareness heightening tools

Scripts are tools embedded into technology-enhanced learning environments that enable designers to specify, sequence, and assign roles or activities for students (Fischer et al. 2007; Weinberger 2003) in order to foster productive argumentation. These tools, which are based on O’Donnell’s (1999) scripted cooperation approach, are often used to scaffold learners’ construction of an individual argument or to guide learners through a specific argumentation sequence (e.g., Stegmann et al. 2006). Scripts can also group students with opposing perspectives together into the same discussion forum (e.g., Clark and Sampson 2005, 2007a, b; Jermann and Dillenbourg 2003). Awareness heightening tools, on the other hand, provide feedback to learners about the quality of their interactions in order to foster more productive argumentation (Jermann et al. 2001). These tools can heighten awareness in terms of the number of words contributed or utilize sophisticated computer-based text analysis technology to provide feedback based on automated analysis of students’ argumentation (Dönmez et al. 2005; Erkens and Janssen 2006; Jermann et al. 2001). Participants can use this feedback to modify how they interact with others (Hesse 2007). The data gathered by these awareness heightening tools can also allow the environment to actively modify other structural features to scaffold the learners in terms of script implementation, group organization, or data access.

Integration of multiple features

Researchers can integrate multiple structural features into online learning environments that reflect their pedagogical goals and theoretical perspectives on dialogic argumentation (e.g., Fischer et al. 2007; Kirschner et al. 2003). For example, the DUNES system (1) provides easily accessible databases, (2) provides a dynamic visualization of students’ argumentation, (3) guides students through a number of individual and collaborative phases, (4) engages asynchronous and synchronous functionalities, and (5) makes students aware of their opinions about the theories in question. Thus, in practice, these online learning environments engage complex interrelationships of features reflecting the designers’ pedagogical goals to support productive argumentation and interaction. Clark et al. (2007) provide an in-depth overview of potential features and their integration in various technology-enhanced argumentation environments.

Assessing Argumentation Quality in Online Learning Environments

Interactions in online argumentation environments can be supported in multiple ways for multiple purposes. Researchers are therefore confronted with complexities and challenges as they attempt to measure how students interact and engage in argumentation within these environments. To date, researchers have developed a broad range of methods in service of these goals. These methods reflect specific perspectives on argumentation, pedagogical goals, and environmental structures.

In order to facilitate the comparison of these analytic frameworks, this review focuses on a short segment of argumentation generated by students working in an online learning environment. The students in the example are interacting within a customized asynchronous threaded discussion forum about their interpretations of data collected during an earlier part of an online project designed to help students understand the basic scientific principles of thermodynamics (Clark and Sampson 2005, 2007a, b). They have completed a series of experiments using computer probes and simulations. At the heart of their argument is the principle of thermal equilibrium.

Fran	I think objects in the same room remain different temperatures because some objects are good conductors and some are bad. This determines how much heat energy is allowed in and out of the object.
Amy	I disagree; I think all objects in the same room are the same temperature. Conductivity only determines how quickly an object will reach room temperature.
Fran	No, good conductors let in more heat energy than poor conductors, so objects that let in more heat will get hotter. For example, when I put a piece of metal and a piece of plastic in hot water the metal was a higher temperature after 30 seconds.
Amy	I guess you’re right. Maybe objects are different temperatures.

In the example, “Fran” is incorrectly convinced that objects remain different temperatures. She explains that conductivity determines how much total heat energy is allowed in and out of the object (which is not correct). “Amy” disagrees and says that conductivity only affects the rate of heat transfer until the equilibrium temperature (the room temperature in this example) is reached (which is essentially correct for the purposes of this discussion). Fran then reiterates her point and supports this point with data from an experiment they did. Amy then changes her position and agrees with Fran.

It is important to note that the sample argument is purposefully very short in order to allow us to compare several analytic frameworks. Many of the analytic frameworks discussed in this review were developed to analyze much more detailed arguments and thus will not be shown to their full potential in analyzing such a short sample. We attempt to take the brevity of the example into account in our discussion of each framework.

A second important caveat is that this review focuses on the dialogic interaction of students or groups attempting to convince one another of the acceptability and validity of alternative ideas (e.g., de Vries et al. 2002; Driver et al. 2000; Erduran et al. 2004; Forman et al. 1998; Kuhn and Udell 2003). As a result, this review does not cover the diverse array of excellent analytic frameworks by authors such as Greg Kelly, William Sandoval, and others for analyzing the quality of rhetorical arguments produced by students. See Sampson and Clark (2006) for reviews of rhetorical frameworks.

The analytic frameworks discussed in this review were chosen to represent a range of promising approaches for analyzing dialogic argumentation in online learning environments. The selection process focused on each method’s capabilities for assessing dialogic argumentation within online environments independent of whether or not the method had been originally developed for application in online or offline environments. We categorize our discussion of these frameworks in terms of each framework’s analytic focus: (1) formal argumentation structure, (2) conceptual quality, (3) nature and function of contributions within the dialogue, (4) epistemic nature of reasoning, and (5) argumentation sequences and interaction patterns.

Formal Argumentation Structure

Formal argumentation structure provides a common focus for analytic methods and pedagogical approaches designed to foster argumentation. Toulmin’s Argument Pattern (Toulmin 1958) is probably the most heavily cited framework for the assessment of argument quality (within technology-enhanced learning environments or without), so this review will begin with a brief discussion of this approach. We then discuss a prominent adaptation of this approach (Erduran et al. 2004) that specifically analyzes students’ dialogic argumentation.

Toulmin: A core foundation for argumentation structure

Toulmin’s framework divides the components of an argument into six categories: claims (assertions about what exists or what values people hold), data (statements that are used as evidence to support the claim), warrants (statements that explain the relationship of the data to the claim), qualifiers (special conditions under which the claim holds true), backings (underlying assumptions), and rebuttals (exceptional conditions capable of defeating or rebutting the warranted conclusion). Toulmin indicates that (1) context determines which components are necessary in a given situation and (2) field-dependent criteria determine the quality of each component. Toulmin’s structural model does not specify what these field-dependent criteria are for each field, but helps to analyze and compare the qualities of different arguments based on the presence/absence of the different structural components and their interrelations.

Toulmin’s model has been employed as a prescriptive model in several online learning environments, often with the idea that stronger arguments contain more of these different components than weaker arguments. In Belvédère, for example, students construct scientific explanations for complex problems and represent these explanations in specific Toulmin-inspired concept maps (Suthers et al. 1997). Similarly, Toulmin’s model has been implemented in asynchronous discussion boards to facilitate the construction of arguments (e.g., Stegmann et al. 2004). Learners in Stegmann and colleagues’ environment complete separate text windows dedicated to individual structural components of a simplified Toulmin model that are then integrated into one argument in the main text window.

Although Toulmin did not provide specific criteria for categorizing ongoing dialogue, the application of our student example would focus primarily on the inclusion of the components to support claims. From the perspective of Toulmin’s framework, the students in the example provide data and warrants for their claims to a reasonable degree and therefore engage in high-quality argumentation (Table 1).

Table 1 Application of Toulmin’s (1958) Framework to the Argumentation Example

Erduran et al. (2004): Adapting Toulmin to dialogic argumentation in the classroom

Erduran et al. (2004) provide an authoritative account and analysis of the excellent framework developed through the joint work of Jonathan Osborne, Sibel Erduran, and Shirley Simon (e.g., Erduran et al. 2004; Osborne et al. 2004; Simon et al. 2006). Their framework elaborates on Toulmin to examine the extent to which elements of an argument are present to assess the quality of argumentation during small group and whole class discourse. The group distinguishes between the process of arguing, which they refer to as argumentation, and the content of an argument. These structural argumentation issues are separated from the validity of the content. Therefore, students might engage in high level argumentation with fallacious argument content.

The group’s framework first characterizes the argumentative operations of each conversational turn during episodes of oppositional dialogue. These argumentative operations include: (a) opposing a claim, (b) elaborating on a claim, (c) reinforcing a claim with additional data and/or warrants, (d) advancing claims, and (e) adding qualifications. The framework applies these codes to each conversational turn. After identifying the argumentative operations of each conversational turn, the quality of the overarching oppositional episodes is assessed using the hierarchy outlined in Table 2. An oppositional episode includes the full sequence of connected conversational turns within an oppositional exchange.

Table 2 Dialogic Argumentation Hierarchy Developed by Erduran et al. (2004)

This hierarchy is based on two major assumptions about what counts as quality. First, high quality arguments must contain grounds (i.e., data, warrants, or backing) to substantiate a claim because “developing rational thought is reliant on the ability to justify and defend one’s beliefs” (Erduran et al. 2004, p. 926). Second, arguments that include rebuttals are “of better quality than those without, because oppositional episodes without rebuttals have the potential to continue forever with no change of mind or evaluation of the quality of the substance of an argument” (p. 927). Whereas Toulmin’s (1958) definition of rebuttal (“the exceptional conditions which might be capable of defeating or rebutting the warranted conclusion,” p. 94) includes “circumstances in which the general authority of the warrant would have to be set aside,” (p. 94), Erduran et al. (2004) refine Toulmin’s definition of rebuttal to focus exclusively on challenges to the grounds of a claim.

The framework assesses the individual elements of our student example quite similarly to Toulmin’s approach (Table 1). One difference involves collapsing the data, warrant, and backing categories into a single “grounds” category due to the practical challenges of distinguishing between data, warrants, and backings in student work. The four conversational turns of the student example would be considered one oppositional episode. Overall, the episode would be considered level 4 according to the hierarchy outlined in Table 2 because the episode includes a clearly identifiable rebuttal against the grounds of an opposing claim. The example therefore represents fairly high quality argumentation from the perspective of this framework even though the students arrive at a non-normative conclusion in terms of the argument itself.

Formal argumentation structure: Affordances, constraints, and other considerations

Toulmin’s framework provides a way to examine the structural components of arguments relatively independently of discipline or domain. The versatility of Toulmin’s argument framework, however, also poses a constraint because it provides little specific information about field-dependent features of what counts as an appropriate claim, warrant, backing, or datum for each field. A further constraint lies in the fact that differentiating between data, warrants, and backings often proves difficult in practice (e.g., Forman et al. 1998; Jimenez-Aleixandre et al. 2000). The framework detailed in Erduran et al. (2004) builds on Toulmin’s framework by allowing researchers to focus on the overarching nature of the argumentation rather than focusing only on the individual components.

Overall, these Toulmin-inspired approaches are well-suited for online-environments where the content of students’ argumentation will cover multiple topics because of the field independent nature of the approach. Clearly these frameworks fit best where the pedagogical goal focuses on argumentation structure. Essentially, the field independence becomes an affordance rather than a constraint. Furthermore, this type of framework has often been used for analyzing environments involving ongoing streams of dialogue, whether synchronous or asynchronous, rather than the production of an in-depth product, such as concept map or several paragraphs of text. In addition to analytic affordances, these frameworks provide ample prescriptive affordances. Essentially, these frameworks provide versatile and clear models for scaffolding students and teachers in argumentation. Stegmann et al. (2004), for example, integrate Toulmin’s model directly into their environment’s student interface to help guide students’ response construction.

Conceptual Quality

The conceptual quality represented in students’ argumentation provides another potential focus for pedagogy and analysis. Clark and Sampson’s (2007a) analytic framework, for example, examines the relationship of conceptual quality, grounds quality and structure. Kuhn and Udell (2003) compare the frequencies of different types of epistemic contributions with the conceptual quality of students’ arguments before and after the dialogue.

Clark and Sampson: Integrating content quality and grounds quality

Clark and Sampson’s framework (2005, 2007a)¹ was designed to evaluate the quality of argumentation that is more oppositional in nature, requires consensus building, and focuses on helping students develop a valid explanation for a natural phenomenon (rather than a socio-scientific issue). To encourage students to engage in this type of argumentation, Clark and Sampson developed a customized online asynchronous threaded discussion forum that uses scripting tools to create discussion groups that consist of students that have provided different explanations for the same phenomenon. This type of structure provides students with an opportunity and a reason to critique the explanations of other students and to respond to the critiques of their own explanation which tends to foster dialogical argumentation.

To evaluate argumentation quality using Clark and Sampson’s framework, the conversational turns (or comments) in an asynchronous discussion forum are first given a nature of contribution, grounds quality, and conceptual quality code. The forums are then broken into episodes and these episodes are given an overarching structural quality score based on the nature of their constituent comments (Table 3) in manner similar to Erduran, Simon, and Osborne’s hierarchy outlined in Table 2. In the hierarchy used by Clark and Sampson, however, “rebuttals” directed against the thesis of a comment are also considered important indicators of quality argumentation. This difference and the theoretical rationale for this difference is discussed in greater detail in Clark and Sampson (2007a) and provides another example of how theoretical perspectives on argumentation, pedagogical goals, and the field-dependent features of argumentation influence the development and modification of analytic schemes.

Table 3 Dialogic Argumentation Hierarchy used by Clark and Sampson (2007a)

The student example (Table 4) represents one oppositional episode composed of four conversational turns (or comments) and is considered level 5 argumentation according to Clark and Sampson’s framework. This indicates a high level of structural quality. In terms of conceptual quality, however, the argumentation is not strong. Only one of the comments consists of nuanced normative content. Moreover, this episode illustrates how students can distort evidence to match claims. In this example, Fran convinces Amy to abandon her normative idea that objects sitting in the same room are in thermal equilibrium by providing inappropriate evidence in support of a non-normative idea. Grounds use is also erratic.

Table 4 Application of Clark and Sampson’s (2005) Framework to the Argumentation Example

Overall, this framework is well suited for environments that (a) use asynchronous threaded communication, (b) group students with opposing perspectives, (c) engage students in extended dialogue, and (d) involve well-defined or complex problems. Environmental structures that sort students into discussion groups containing multiple perspectives are critical because the framework is designed to assess argumentation that is more oppositional in nature. The asynchronous threaded structure is also critical because the analysis depends heavily on determining the parent comment for each response in order to investigate the connections between structural moves, conceptual quality, and grounds use. Finally, the focus on a well-defined scientific issue is important because the analysis involves assessing the scientific normativity of the content for each comment.

Kuhn and Udell (2003)

Kuhn and Udell’s (2003)² framework focuses on measuring the development of students’ argumentation skills over time. The framework analyses the conceptual quality of argumentation by classifying the arguments contributed by each individual in a discussion using a hierarchical coding system. The lowest level comprises nonjustificatory arguments, which have little or no argumentative force. The middle tier comprises nonfunctional arguments, which focus on tangential aspects of the problem rather than core issues. At the highest conceptual level, functional arguments address core aspects of the problem. The possible arguments for (pro) and against (con) the topic being debated (which is capitol punishment in their study) in each tier are explicitly listed in order to compare conceptual quality before and after instruction. The goal involves measuring changes in the sophistication of students’ arguments and their understanding of the topic. The approach therefore focuses on the logical coherence and relevance of the arguments generated by students. This type of focus is especially well-suited for online environments where students debate and discuss issues without scientifically normative answers (such as capital punishment).

A rubric of discourse codes (originally developed in Felton and Kuhn 2001) is then used to analyze the nature of a dialogue between two students as they debate an issue on which they hold opposing positions. This component of Kuhn and Udell’s analytic method is actually exemplary of the third category of analytic foci (see “Nature and Functions of Contributions Within the Dialogue”). Codes for 25 distinct discourse moves are included which can be grouped into four broad categories: exposition (e.g., an extension of previous statement or a clarification of speaker’s argument), challenge (e.g., disagreements and counterarguments), requests (questions), and other non-request discourse moves. These codes are applied to each “utterance” from the moment someone starts speaking until their speaking turn ends. Kuhn and Udell expect increased frequency of the challenge codes and decreased frequency of the exposition codes if students are increasing in the sophistication of their argumentation.

An analysis of the student example using Kuhn and Udell’s framework indicates that that the argumentation that takes place between Fran and Amy is of high quality in spite of its brevity (Table 5). The arguments presented by Fran and Amy are functional in terms of their conceptual quality, which indicates that these students address key aspects of the problem. Moreover, the discourse moves used by the students in this example heavily emphasize challenge moves (e.g., questioning the ideas of others) rather than exposition (e.g., proposing or clarifying one’s own ideas).

Table 5 Application of Kuhn and Udell’s (2003) Framework to the Argumentation Example

Overall, this framework is well suited for environments that (a) use synchronous communication, (b) group students with opposing perspectives, and (c) call for students to discuss the merits of alternative viewpoints. This type of online learning environment would require group organization scripting and a way for students to construct their own arguments before and after engaging argumentation to allow for this scoring and sorting. Environment structures including access to large databases of information and enhanced representations of the subject matter would also offer advantages.

Conceptual quality: Affordances, constraints, and other considerations

By examining the content of student ideas and how students interact with each other, researchers can better support students as they attempt to negotiate meaning or validate ideas in online environments. One challenge, however, is that rubrics with a focus on conceptual quality become very topic-specific and thus require significant modification for application across contexts. A focus on conceptual quality fits well with environments that include easily accessible and indexed knowledge bases and enriched representations of focal subject matter because these types of functionalities provide conceptually rich material for consideration and integration within the discussion. Furthermore, if the pedagogical goal of an online environment focuses on helping students learn how to engage in argumentation (e.g., proposing, justifying, and challenging ideas), the analytic framework can focus on the structure of students’ contributions to the discussion and still be sufficient, but if the goal of the online environment involves providing an opportunity for students to learn from argumentation (e.g., develop a more in-depth understanding of the content that is being discussed), the analytic framework’s examining the conceptual quality of students’ ideas become critical.

In choosing an analytic framework, researchers must determine the importance of the relationship between the normativity of a comment and the relative time of its contribution. Non-normative content at the onset of dialogue followed by increasing normativity by the conclusion of the dialogue might represent something entirely different than the reverse trajectory. Clark and Sampson (2007a) and Kuhn and Udell (2003) address the temporal issue by measuring the normativity of students’ arguments before and after the dialogue but do not examine the trajectories within the dialog itself (as discussed in “Argumentation Sequences and Interaction Patterns” below). Both Clark and Sampson (2007a) and Kuhn and Udell (2003) use short ordinal scales of conceptual quality to facilitate analysis in a manner similar to that employed by Zohar and Nemet (2002).

Nature and Functions of Contributions Within the Dialogue

The nature of participants’ contributions forms another category of analytic focus. Whereas approaches emphasizing formal argumentation structure focus specifically on the components of an argument, this type of framework focuses on the types of dialogue as well as the proportions of conceptually and argumentatively productive dialogue in which students engage. Kuhn and Udell (2003) and Clark and Sampson (2005, 2007a) include analysis of the nature and function of contributions, as discussed in the preceding section, but the analytic frameworks of de Vries et al. (2002), Janssen et al. (2006), and Baker et al. (2007) focus specifically on the analysis of the nature and function of these contributions.

de Vries, Lund, and Baker (2002)

De Vries et al. (2002)³ examine ways to promote epistemic dialogue in online learning environments. As defined by de Vries, Lund, and Baker, epistemic dialogue (1) takes place in a collaborative problem-solving situation, (2) can be characterized as argumentation or explanation, and (3) concerns the knowledge and concepts underlying the problem-solving rather than the execution of problem-solving actions.

To foster this type of discourse between students, de Vries, Lund, and Baker integrate synchronous computer-mediated communication, scripting, and awareness heightening tools into the CONNECT environment. In this environment, students work together in order to produce a piece of text that explains a puzzling phenomenon through a process of collaboration and negotiation. The CONNECT environment has a communication interface and a task interface. The communication interface is a text-based chat facility in which two students (and a teacher) can communicate with each other. A free message text box is complemented with a number of pre-defined communication buttons for interaction management (i.e. “Yes”, “No”, “I don’t agree”). The task interface provides a shared text editor and displays individual students’ texts. The interface displays individual student texts as a number of statements that can be rated by each student for agreement. Consensus or discrepancy between ratings is translated into instruction labels for the students: “Discuss” (conflict), “Verify” (agreement), “To be seen” (neither student knows), or “Explain” (one student doesn’t know). The ratings and instruction are meant to focus the student’s discourse (de Vries et al. 2002) by providing socio-cognitive structuring and awareness heightening of the discussion process.

Figure 1 outlines the categories that De Vries, Lund, and Baker use to code this type of discourse. This scheme is typically applied at the phrase level of the discourse. Coders first determine the major discourse category in which the phrase occurred. The coders then assign a sub-type to each of the phrases based on the major category. Analysis involves qualitative and quantitative comparisons of the discourse in each category and subcategory across the phases of the project.

Fig. 1

de Vries, Lund, and Baker’s hierarchy of epistemic categories

Table 6 applies de Vries et al. (2002) framework to our student example. From the perspective of this framework, the example represents desirable epistemic discourse because all four contributions to the discussion can be characterized as either explanation or argumentation. As previously mentioned, de Vries et al. suggest that explanation and argumentation are “potentially powerful mechanisms by which students can collaboratively construct new meaning” (2002, p.64).

Table 6 Application of de Vries et al.’s (2002) Framework to the Argumentation Example

In our opinion, the framework is especially well suited for the analysis of student argumentation in environments that incorporate specialized computer-mediated communication, scripting tools, and awareness heightening tools. For example, if the framework could be applied online, an awareness heightening tool could help students view the kinds of contributions they are making (e.g., procedural, off-task, or explanation) and if they are making too many inappropriate contributions (e.g., off-task). Students might learn to engage in more productive dialogic argumentation through this type of formative feedback.

Janssen, Erkens, Jaspers, & Kanselaar (2006)

Janssen et al. (2006)⁴ framework focuses on argumentative dialogue acts through a computer-automated coding process. The Dialogue Act coding framework identifies the communicative function of each utterance typed by the students during their online collaboration and communication. The five main communicative functions include: argumentative (indicating a line of argumentation or reasoning), responsive (e.g., confirmations, denials, and answers), informative (transfer of information), elicitative (questions or proposals requiring a response), and imperative (commands). The framework specifies twenty-nine different dialogue acts within these five main functions. Seven of the twenty-nine focus on argumentative dialogue (Table 7).

Table 7 Argumentative Dialogue Acts within Janssen et al. (2006) Framework

The multiple episode protocol analysis (MEPA) computer program automatically codes protocols and identifies which dialogue acts are used during collaboration. As part of this process, a production rule system automatically categorizes utterances into dialogue acts (300 production rules). After segmenting the dialogue acts, a set of if-then rules uses pattern matching (1,300 production rules) to look for typical words, phrases, and punctuation that serve as discourse markers signaling the communicative function of a sentence in conversation in natural language (Schiffrin 1987). For example, “because” at the beginning of an utterance usually indicates a reason. The computer-driven nature of the MEPA software results in high reliability and connects directly to the databases of online learning environments. The automated nature provides affordances for research involving large volumes of data.

Janssen and colleagues apply the framework to analyze chat protocols of secondary school students collaborating in the Virtual Collaborative Research Institute environment. The students engage in a four-week project where they collaboratively write an essay based on historical sources. The Virtual Collaborative Research Institute environment offers several shared tools to support the project activities of the students. Although asynchronous communication is possible, most communication involves synchronous chat for coordination and discussion during the investigation of resources and the collaborative writing. These chats are automatically coded.

Based on these codings, MEPA calculates frequencies of sequences of argumentation consisting of two, three, four, or five arguments. These findings are then compared to group performance in terms of the shared texts constructed by each group. Shared texts are scored in terms of use of sources, content and argumentation, and text construction and language. Using this combined approach, the framework allows analysis of various interventions on discourse and artifact quality. The framework and automated coding system are thus intended for much longer interactions that include artifact production. The unit of analysis in the framework, obtained through the segmentation procedure, focuses on the single sentence or phrase. Interpretation of higher levels of argumentation (i.e., argumentative episodes) has to be done bottom-up. The framework focuses on the communicative nature of task-oriented discourse, but is not appropriate for judging the conceptual quality of contributions. From the perspective of the framework our student example represents an “extend” sequence of argumentation and is therefore of high quality (Table 8).

Table 8 Application of Janssen et al.’s (2006) Framework to the Argumentation Example

Baker, Andriessen, Lund, van Amelsvoort, and Quignard (2007)

Baker et al. (2002, 2007)⁵ have created a coherent framework (called Rainbow) for analyzing computer-mediated pedagogical debates. Rainbow includes seven principal analytic categories focusing primarily on the epistemic nature of the contributions that students make during collaboration. The framework was developed to allow the researchers to investigate the processes through which participants achieve conceptually deeper levels of interaction. The Rainbow framework has been used extensively to analyze texts, diagrams, and chat of students working together in the TC3 groupware environment. This environment enables students to write shared texts and to construct shared argumentative diagrams based on information sources. Furthermore, they can chat with each other as they engage in these activities. Analyses typically focus on the processes through which the shared texts are constructed.

At the most basic level, the Rainbow framework distinguishes between activity that is part of the prescribed assignment and activity that is not. From there, Rainbow differentiates activity that is part of the prescribed assignment as either task-focused or non task-focused. Non task-focused activity is categorized as either social relation (interaction that is concerned with managing students’ social relations with respect to the task) or interaction management (interaction concerned with managing the interaction itself). Task-focused activity is categorized as task management (management of the progression of the task itself), opinions (interaction concerned with expressing opinions with regard to the topic under debate), argumentation (expression of arguments and counterarguments directly related to a thesis), and explore and deepen (interaction concerned with arguments and counterarguments linked together, their relations, and the meaning of the arguments themselves). The rationale for using each of these seven categories is grounded in the research on collaborative learning, task-oriented dialogues, verbal interactions, and argumentation theory.

The default level of analysis is the individual student comment because that process allows students’ activity to self-define the unit boundaries. Each student comment is then assigned to the category that best represents its primary nature. However comments are interpreted in the context of the ongoing debate, for example explore and deepen can only occur after an argument or opinion has been given. In this way process characteristics of the discussion are specified in a direct way. Baker and colleagues also outline the potential for subsequent analysis at micro and macro levels. Researchers may apply the seven codes at a smaller grainsize by parsing individual comments into components or apply the codes at a macro level to larger episodes comprising strings of multiple comments focusing on a coherent goal.

The student example represents quality argumentation when assessed using the rainbow framework because Fran and Amy are involved in conceptual deepening and exploration of the topic (Table 9). Table 9 codes the student example at the standard comment level as well as the micro level. Overall, the Rainbow framework might analyze the first three comments as a single macro episode representing “explore and deepen.”

Table 9 Application of Baker et al.’s (2007) Rainbow Framework to the Argumentation Example

Nature and function of contributions: Affordances, constraints, and other considerations

This category of analytic frameworks focuses on ongoing discourse. These frameworks therefore work best with synchronous forums or asynchronous forums rather than environments focusing on the juxtaposition of a small number of crafted responses or the interpretation of dialogic artifacts. That said, however, frameworks such as Rainbow can adapt to other formats as discussed by Baker and colleagues. Each of the frameworks provides valuable affordances. The framework by de Vries et al. (2002) offers detailed consideration of the types of discourse moves that students may make. The Rainbow framework is carefully grounded theoretically and is parsimonious enough to simplify application and analysis. Janssen, Erkens, Jaspers, and Kanselaar’s framework offers potential in terms of its automated capabilities. In sum, these frameworks provide flexible approaches for researchers interested in assessing the nature of student’s contributions and the overall effectiveness of online environments designed to encourage substantive discussions of this type.

Epistemic Nature of Reasoning

A fourth category of analytic focus revolves around the epistemic nature of students’ reasoning. In other words, what types of reasoning do students employ to support their claims or to challenge the claims of others? Both Jimenez-Aleixandre et al. (2000) and Duschl (2007) have developed analytic methods designed to address this question.

Jimenez-Aleixandre, Rodriguez, and Duschl: Adding focus on the nature of students’ reasoning

The Jimenez-Aleixandre et al. (2000)⁶ framework first identifies whether the topic of a conversation is task-related or content-related. The framework then applies a Toulmin model to identify data, warrants, backings, and qualifiers during a discussion. Once these elements are identified, the framework examines how students elaborate, reinforce, or oppose claims by classifying the types of reasons that students use in their arguments with epistemic operations based on the work of Pontecorvo and Girardet (1993) who study argumentation in the social sciences and the work of Sober (1993) and Thorley (1992) who examine reasoning and discourse in science. These epistemic operations include: induction (looking for patterns or regularities), deduction (identifying various instances of rules and laws), causality (relation cause-effect, looking for mechanisms or predictions), definition (stating the meaning of a concept), classification (grouping objects or organisms according to criteria), appeals to analogy, exemplar, instance, attribute, or authority (appealing to analogies, instances, or attributes as a means of explanation), consistency with other knowledge, experience, commitment to consistency, or metaphysical (factors of consistency, particular with experience, or general in need for similar explanations), and plausibility (predication or evaluation of own or others’ knowledge). Analysis then compares the proportions of different epistemic moves within the dialog.

Jimenez-Aleixandre et al. (2000) developed the framework to examine how students construct and assess arguments in the context of small group problem solving. The main task in their study involves solving a problem dealing with genetics (selective breeding of chickens to remove an unwanted trait from a population), comparing solutions developed by different groups, and justifying choices. The study assesses the quality of argumentation within small group discussions and whole class discussions. The framework suggests suitability for a broad range of online environment structures where students (a) work on well-defined or complex problems, (b) use synchronous or frequent asynchronous computer-mediated communication, and (c) strive to reach consensus or solve a problem. When the framework is applied to the student example (Table 10), we see that (1) the discourse is content-related rather than task-related, (2) students justify their ideas, and (3) students’ reasons focus on causality, consistency, and appeals to instances rather than plausibility or appeals to authority. From the perspective of this framework, the student example therefore represents fairly high quality argumentation.

Table 10 Application of Jimenez-Aleixandre et al.’s (2000) Framework to the Argumentation Example

Duschl (2007): Using Walton’s schemes of presumptive reasoning to examine the ways students reason

Duschl’s (2007) framework represents an innovative application of Walton’s (1996) framework to scientific argumentation in the classroom. Walton suggests that dialogic argumentation is grounded in burden of proof, presumption, and plausibility rather than in structural form alone. Walton details twenty-five different argumentation schemes that focus on how presumptions are brought forward in arguments as premises or inferences that link premises to conclusions. The goal involves shifting the weight of presumption from one side of a dialog to the other. An opposing voice can then respond with questions or statements that shift the weight of presumption back upon the original participant. Analysis focuses on categorizing the types of reasoning employed within an argument.

Duschl and colleagues first narrowed Walton’s 25 categories to nine categories relevant to scientific argumentation in the classroom (Table 11). Distinguishing between these nine categories often proved difficult, however, in coding students’ work. Duschl and his group therefore collapsed the nine categories into four categories including requests for information, expert opinion, inference, and analogy. The framework applies these coding categories at the level of the reasoning sequence, which is approximately at the level of each student comment in our example. Analysis focuses on the number and proportion of epistemic discourse types in students’ discussions.

Table 11 Duschl’s (2007) Framework Collapses Nine of Walton’s (1996) Categories into Four Categories for Greater Coding Reliability

This framework was originally developed to examine the nature of argumentation that takes place between students as they reason about explanations, experiments, and models. The framework has been used in a number of contexts including: (a) a unit called vessels, which was designed to encourage students to evaluate causal explanations; (b) a unit called Acids & Bases, which was designed to encourage students to evaluate chemical models; and (c) a unit called Earthquakes & Volcanoes, which was designed to give students an opportunity to evaluate scientific arguments. Data sources typically consist of the interactions that take place between small groups of students (typically three students in a group). This framework seems applicable to most online science learning environments where (a) ongoing dialog is supported, (b) students are working on well-defined or complex problems, and (c) the goal of the discourse involves reaching consensus or developing a solution to a problem.

From the perspective of this framework, our student example involves inferences from evidence to hypothesis (the metal was a higher temperature after 30 s) and inferences from cause to effect (conductivity determines how quickly an object will reach room temperature) in order support or refute ideas (see Table 12). These epistemic moves represent desirable argumentative practices in this context because the lesson involves students reasoning from partial sets of experiences and evidence. Our student example therefore represents fairly high quality argumentation from the perspective of this framework. It is worth noting, however, that these types of epistemic moves might not represent the most productive, or even appropriate, choices in other contexts. For example, relying on an expert opinion to shift presumption onto the other participants might represent effective and appropriate practice in other contexts. It is therefore important to take pedagogical goals, theoretical perspectives, and context-specific (or field-specific) norms into account when using this type of framework to assess argumentation quality.

Table 12 Application of Duschl’s (2007) Framework to the Argumentation Example

Epistemic nature of reasoning: Affordances, constraints, and other considerations

Frameworks that focus on the epistemic nature of reasoning provide valuable information about how students determine ‘what counts’ as warranted knowledge and how students determine which ideas should be accepted, rejected, or modified. Rather than assessing the normative quality of students’ reasoning and contributions, these frameworks revolve around the types of reasoning that students use when they propose, support, evaluate, and challenge ideas. In terms of specific affordances and constraints, Jimenez-Aleixandre, Rodriguez, and Duschl’s framework integrates the assessment of reasoning type with structural quality. Duschl’s framework, in turn, is noteworthy for its distillation and synthesis of Walton’s framework into a manageable discipline-specific coding scheme.

Overall, these frameworks (and this categorical focus for analysis) offer strong affordances for researchers interested in helping students improve their discourse skills, reasoning skills, and argument evaluation skills. Applicable environments might focus on specific discourse goals (e.g., securing commitments from an opponent or undermining the opponent’s argument) and effective strategies for achieving these goals (e.g., justifying claims with evidence or requiring opponents to justify their claims with evidence). Generally speaking, the frameworks focus on frequency counts so they are better suited to environments supporting free flowing dialog, such as asynchronous and synchronous discussions, rather than the micro-analysis of smaller segments. Finally, these frameworks afford relative content independence and therefore require little modification for application across related topic areas because the frameworks focus on a core attribute of all argumentation—the quality of reasons.

Argumentation Sequences and Interaction Patterns

The fifth major category of analytic focus revolves around argumentation sequences and patterns of interaction. Whereas the category “nature and function of contributions within the dialogue” (discussed earlier) focuses on the frequencies of various types of contributions, this category focuses specifically on the sequences of participant interaction and the development of ideas. Frameworks by Leitão (2000), Hogan et al. (2000), Baker (2003), and Weinberger and Fischer (2006) provide interesting examples of this category of analytic focus.

Leitão (2000): Argument’s potential for knowledge building cycles

Leitão (2000, 2007)⁷ envisions argumentation as a process of resolving differences of perspectives. Leitão (2000) considers a specific sequence of argumentation to be particularly fruitful for knowledge building. This process may take place intrapersonally or interpersonally. In what Leitão calls a knowledge building cycle, students (1) construct an argument, which consists of a position and its justification, (2) construct a counterargument in response to the first argument, and (3) create a reply that captures the participants’ immediate and secondary reactions to the counterargument.

Counterarguments may (1) support a different perspective of the debate, (2) challenge the validity of the claim, or (3) question the validity of the warrants of the claim. Similarly, the students’ replies in the third phase may take several forms, which indicate the degree to which a counterargument is accepted or dismissed in favor of the initial argument. Replies in this third phase include (1) dismissal of the information conveyed by the counterargument, (2) local agreement with the counterargument that acknowledges parts of the counterargument but preserves the initial argument, (3) integrative replies that indicate the speaker’s agreement with parts of the counterargument but modify and qualify the initial argument, and (4) a withdrawal of initial view that entails abandoning the first argument in favor of the counterargument. Through these patterns of argumentation, the initial argument may be preserved, revised, or withdrawn. Leitão (2000) proposes that these patterns of argumentation optimally shape the process of social knowledge construction.

Leitão’s framework is well suited for the analysis of arguments that represent more than one perspective (Leitão 2007). Thus, it can be applied in online environments that are designed to promote knowledge reappraisal (i.e., in which an initial perspective is confronted with another perspective). For example, using this framework may prove valuable for examining how students in asynchronous discussion forums respond to the counterarguments raised by other students. The framework has also been used as a prescriptive model for structuring asynchronous discussion (see Stegmann et al. 2004). Stegmann and colleagues’ environment directs learners to (a) create arguments in favor of a specific position whenever starting a new discussion thread (i.e., every message that is not a reply to an already posted message), (b) compose counter-arguments to that specific position when replying to an initial message, and (c) explicitly integrate arguments and counter-arguments in the third phase. This script for the construction of argumentation sequences leads learners to engage more frequently in Leitão’s knowledge building cycles and facilitates the acquisition of argumentative knowledge (Stegmann et al. 2004).

From the perspective of Leitão’s (2000) framework, the student example represents a complete knowledge building cycle (see Table 13). The episode begins with Fran contributing her initial argument. Amy then counters by questioning the claim. Fran replies by dismissing Amy’s counterargument which enables Fran to preserve her initial viewpoint. Amy then accepts Fran’s ideas and withdraws her initial viewpoint. From Leitão’s (2000) perspective, both this type of outcome and outcomes that result in a revised initial claim represent successful outcomes of argumentation.

Table 13 Application of Leitão’s (2000) Framework to the Argumentation Example

Hogan et al. (2000)

Hogan et al.’s (2000) framework examines discourse components, interaction patterns, and reasoning complexity. The framework focuses on (1) how students work to improve weak or incomplete ideas, (2) the patterns of verbal interactions that take place between individuals in scientific sense-making activities, and (3) the relationships between discourse patterns and the sophistication of scientific reasoning in discussions.

Analysis begins with the assignment of macro-codes at the level of conversational turns in terms of the major modes of a group’s discussion. Macro-codes include knowledge construction, logistical, and off-task. Micro-codes are then assigned at the level of statement or phrase including conceptual, metacognitive, question–query, nonsubstantive, and other. Micro-codes include multiple subcategories. Researchers then create discourse maps illustrating the patterns of interactions between students based on these codes. Patterns of interaction include consensual (where a student proposes an idea and another student agrees), responsive (where a student asks a question and another student answers), and elaborative (where students discuss and revise each others ideas). Researchers next assess reasoning complexity (Table 14) and compare this information to the interaction patterns. Note that the assessment of reasoning complexity represents an example of the category of analytic focus discussed in the “Conceptual Quality” section above.

Table 14 Application of Hogan et al.’s Framework to Code the Reasoning Complexity of the Argumentation Example

The Hogan et al. (2000) study does not apply the framework to a technology supported environment. Instead, the study focuses on face to face group discussions in the classroom. The study analyzes the differences in discourse patterns and collaborative scientific reasoning in peer group discussions and teacher guided group discussions. The framework represents a relatively top-down approach to analysis in comparison to several of the other frameworks. In terms of online environments, the framework seems especially suited for normative or qualitative comparisons within synchronous discussions because of the framework’s distinction between multiple levels of depth for the six criteria.

Our student example represents what Hogan and colleagues describe as an elaborative interaction pattern. Hogan and colleagues suggest that elaborative interaction supports quality argumentation because it prolongs discussions and leads to higher levels of reasoning. As presented in Table 14, our student example involves complex reasoning in terms of generativity, logical coherence, and synthesis.

Baker (2003)

Baker’s (2003)⁸ framework examines the positions adopted by individuals during argumentation, the metamorphosis of ideas over time, and the pragmatic function of language. The framework focuses on argumentation as a way to facilitate collaborative learning. According to the framework, argumentation transforms the epistemic status of solutions by (a) establishing relations between the proposed solutions and other knowledge or (b) promoting the negotiation of new meanings. Arguments strengthen the epistemic status of a solution. Counter-arguments weaken the epistemic status of a solution. As a discursive activity, argumentation establishes relations between possible solutions and other sources of knowledge. As a dialogic activity, argumentation incorporates aspects of formal and pragmatic dialectics. Through the analyses, this framework measures the strengthening and weakening of the epistemic status of various claims as well as the progression of dialectic moves. The analysis interprets the epistemic status of solutions from the perspective of each discussant and from a shared perspective once argumentative outcomes (e.g., concession, refutation) are made explicit.

The framework has been used for deep analysis of argumentative conflict in dyadic interactions of students using the CONNECT environment to solve a problem involving the interpretation of sound in physics. In the environment, students first comment and rate individually written texts and later write a text collaboratively. Communication occurs through synchronous chat functionality. The framework seems especially suited for deep analysis of argumentative episodes in discursive settings, especially online chat or other synchronous communication.

Table 15 applies Baker’s (2003) framework to our student example. Although brief, the discourse changes the epistemic status of idea A (objects remain different temperatures) and idea C (objects become the same temperature). The example therefore represents fairly high quality argumentation from the perspective of this framework. This type of analysis provides a method for tracking the number and types of ideas that students propose and challenge when engaging in argumentation. This method also enables researchers to examine how students use language to generate or validate knowledge.

Table 15 Application of Baker’s (2003) Framework to the Argumentation Example

Weinberger and Fischer (2006): A multi-dimensional approach to analyze argumentative knowledge construction

Weinberger and Fischer’s (2006)⁹ framework examines the process through which knowledge is constructed as students engage in argumentation in online environments. Their framework assesses argumentation along four independent dimensions including the participation dimension (e.g., the amount of participation by each student and the heterogeneity of participation within the learning group), the epistemic dimension (e.g., how learners work on the specific learning task and how learners apply theoretical concepts) the formal argumentative dimension, (e.g., how learners construct arguments and respond to the arguments of others) and the dimension of social modes of co-construction (e.g., how students refer to the arguments of their learning partners in relation to knowledge acquisition).

The Weinberger and Fischer (2006) framework has been applied to analyze dialogic argumentation in online learning environments that incorporate asynchronous discussion boards. To reduce work load of analyzing learners’ dialogue on multiple dimensions, the automated TagHelper tool was used to code dialogue on all dimensions of the analytic framework for comparison with human coders (Dönmez et al. 2005). Groups of three students worked together to analyze a complex problem case in the domain of educational psychology. Along the dimensions of the framework, the interaction of learners was structured to facilitate specific participation patterns, specific sequences of epistemic activities, the construction of single arguments, the construction of specific argumentation sequences, and specific transactive social modes of co-construction (e.g., Weinberger et al. 2005; Weinberger et al. 2007).

Table 16 applies the framework to our student example along these four dimensions. With respect to the participation dimension, Amy (34 words) utters less than half the words Fran (81 words) does. With respect to the epistemic dimension, both Fran and Amy engage in on-task talk and construct relations between the target conceptual space (rather than prior knowledge) and the problem space. Fran, however, constructs inadequate relations, whereas Amy applies the theoretical concepts adequately. On the formal argumentative dimension, Amy and Fran build relatively complete arguments and argumentation sequences (see Toulmin and Leitão). Finally, on the social modes of co-construction dimension, Amy and Fran clearly engage in conflict-oriented consensus building as they refer to each other’s contributions and attempt to negotiate meaning.

Table 16 Application of Weinberger and Fischer’s (2006) Framework to the Argumentation Example

Argumentation sequences and interaction patterns: Affordances, constraints, and other considerations

This analytic category increases the unit of analysis from an individual comment or fragment to larger trajectories, patterns, and sequences. As such it allows us to focus on the actual processes of co-construction of knowledge rather than focusing on frequency counts of elements that correlate to desirable interaction. These approaches thus can be applied to argumentative talk between students, in which expressing and resolving differences of perspectives is more likely than in individual or teacher-guided learning (Hogan et al. 2000). These frameworks may be applied across almost all online argumentation environments independent of environment structure or the nature of the artifacts created because these analyses can focus at microgenetic scales as well as broad scales. Increased complexity of application accompanies this increased power, however. The challenge of this analytic category manifests itself in terms of increased amount and complexity of work required to reliably apply these types of analyses across larger samples.

One interesting dichotomy involves the presence or absence of a pedagogical goal state within the frameworks to inform the development of practice. In other words, does the framework provide a road map for instruction in terms of desirable student practice? For example, Baker’s (2003) analytic framework provides ways to track the evolution and status of the ideas discussed by students, including how (or if) they are challenged, but the framework provides less concrete guidance for instruction—What do we want students to know or to be able to do? Other frameworks are more prescriptive in this regard (e.g., Leitão 2000), which may or may not be desirable depending on the goals of the researchers.

Synthesis

In this review we have considered several frameworks for analyzing dialogic interaction in online learning argumentation environments. In closing, we address four issues. The first issue focuses on the variety of verdicts delivered by the individual frameworks regarding the quality of interaction in our student example. The second issue focuses on how researchers might choose between frameworks in analyzing different environments. The third issue focuses on how choice of framework might influence environment design. The fourth, and final, issue focuses on the future promise of automating these frameworks within online learning environments.

Does our student example represent quality argumentation?

Most of the frameworks discussed in this review would assess the student example as representing fairly desirable argumentative discourse, although often for very different reasons (see Table 17). For example, the frameworks that examine formal argumentation structure suggest that the student example is of high quality because Amy and Fran use data, warrants, and rebuttals as they discuss their different perspectives. Alternatively, frameworks that examine argumentation sequences and interaction patterns suggest that the sample argumentation is of high quality because of the ways Amy and Fran interact with each other. As illustrated in this review, the analytic frameworks focus on many different aspects of argumentation including argument structure, epistemic types of reasoning, conceptual normativity, quality of warrants, number of warrants, logical coherence of claims with warrants, argumentation sequences, patterns of participation, conceptual trajectories, and the process of consensus building.

Table 17 Assessment of the Student Example by the Frameworks in each Category

This variety of perspectives leads to the obvious conclusion that it is insufficient for researchers to say “we measured the quality of argumentation” or “we successfully supported argumentation.” Researchers need to specify the theoretical interpretation of argumentation underlying their analytic methods in order to facilitate communication and comparison in relation to other research in the field. Clearly there are multiple theoretical perspectives and aspects of argumentation worth fostering. Clarity of communication regarding theoretical commitments and pedagogical goals is therefore critical in terms of environment design and analytic frameworks.

How environment design informs the choice of frameworks

As discussed above, researchers have a great deal of flexibility in creating environments. Researchers can integrate multiple synergistic features to facilitate specific aspects of argumentation. An environment might, for example, guide learners in specific knowledge building cycles (see Leitão 2000; Stegmann et al. 2004), represent their argumentation in concept maps (see Suthers and Hundhausen 2001), or facilitate their awareness of the quality of their argumentation from the perspective of a specific framework (Erkens and Janssen 2006; Jermann et al. 2001). Environment designs also vary in terms of the subject of the discussion (e.g., problems with a clearly demonstrated solution or wicked problems with no right answer), the goal of the discussion (e.g., reaching consensus or learning more about the topic), and the rules the students will need to use in order to distinguish between ideas.

Environment goals and environment structure should heavily influence a researcher’s choice of analytic frameworks.To facilitate the process of choosing a framework, Table 18 highlights potential areas of strength for each of the frameworks discussed in this review based on (a) the object of the discussion, (b) the purpose for engaging in the discussion, and (c) the norms that will govern how participants should distinguish between ideas.

Table 18 The Suitability of the Different Frameworks for use in Different Contexts

Apart from the contextual aspects of choosing a framework, the theoretical underpinnings of a framework need to fit or align with the researcher’s questions. Many of the frameworks reviewed here are theoretically grounded in domains and research traditions outside of educational research (e.g., linguistics, philosophy, history of science, developmental psychology). Therefore, researchers in this field should always question the extent to which the definitions of argumentation quality from frameworks with origins in other domains fit questions and theories that are important in educational research (De Wever et al. 2006). Depending on this fit, certain definitions of argumentation quality may or may not be connected to other learning outcomes (and specific frameworks may or may not prove compatible as a result). The frameworks discussed in this review provide indicators of our field’s current progress in adapting inspirations from these other domains to educational research.

How framework choices inform environment design

Just as the design and goals of an environment should guide a researcher’s choice of frameworks to analyze that environment, a designer’s theoretical perspectives should guide the design of environments. Throughout this review, we have outlined several examples of online environments whose designs are informed and guided by specific analytic frameworks.

Once a designer has chosen a framework to represent the theoretical commitments and goals for the environment, the designer has great latitude to integrate specific functionalities to foster the commitments specified within a framework. Collaborative communication interfaces (synchronous or asynchronous) provide general support for dialogic argumentation, for example, and may potentially foster higher quality argumentation than face-to-face settings (e.g., Marttunen and Laurinen 2001; Schellens and Valcke 2006). Asynchronous scenarios provide learners with the necessary time to carefully consider and construct arguments. Synchronous environments, on the other hand, enable learners to more fluidly co-construct arguments with others. Sharing artifacts (especially argumentation maps or collaboratively written essays) can support the epistemic and conceptual quality of arguments and visualize the formal structure of arguments (e.g., Kirschner et al. 2003). Enriched access to information can support argumentation by offering resources to validate and justify arguments on acceptable grounds or criteria (Oestermeier and Hesse 2000). Scripts can guide learners to engage in specific knowledge building cycles or set the stage for productive dialogic argumentation by grouping students with opposing perspectives (Fischer et al. 2007). Awareness heightening tools can help learners self-regulate their dialogic argumentation by mirroring specific aspects of argumentation processes back to students (e.g., their participation). In short, environment design offers the opportunity to integrate specific features to reify focal aspects of argumentation as guided by designers' theoretical and analytical frameworks.

The future promise of automating frameworks within environments

Automating frameworks within online environments can potentially provide affordances for increasingly complex research integrating multiple lenses. As outlined in the introduction, the student example provides a complex challenge for analysis because the process of argumentation is strong but the content of the conclusion is fallacious (and a more normative solution gets discarded). Integrating multiple categories of analytic focus within a single framework may provide more leverage in investigating these complex relationships (e.g., van Boxtel and Roelofs 2001). By combining, for example, analysis of formal structure with analysis of the epistemic nature of reasoning, we can build a more accurate discipline-specific understanding of the quality of students’ argumentation skills.

Ultimately, integrating other analyses within the analysis of argumentation sequences and interaction patterns seems particularly promising (e.g., Weinberger and Fischer 2006). Many frameworks currently focus on frequency counts as correlational markers of argumentation quality. Careful tracking of participant interaction and the evolution of ideas would align our analyses more directly, and therefore potentially add more validly, with the processes of argumentation that we wish to foster. The challenge, of course, rests in the accompanying increased complexity of conducting such analyses. Fortunately, online learning environments offer strong affordances for grappling with these challenges because online learning environments incorporate the potential to integrate analytic frameworks to automate the logging and coding of students’ actions and interactions.

Automated logging and coding also offers potential affordances for teachers. Teachers face significant challenges when attempting to support argumentation practices within their classrooms. Online learning environments that integrate automated analytic frameworks could provide teachers with tools to monitor and scaffold multiple small groups of students working simultaneously on projects within their classes. Such environments might also model argumentation practices for the teachers themselves by helping the teachers interpret the argumentation practices of their students within the environment.

Additional opportunities will evolve as we develop these automated technologies to track interaction and quality more accurately in real time. We will, for example, be able to harness tracking capabilities to modify supports for argumentation in real time. Dönmez et al. (2005) have made early progress in this regard by harnessing latent text analysis technology to score the quality of students’ argumentation products. Similarly, the Dialogue Act coding software (Janssen et al. 2007) has been integrated into The Shared Space chat tool to make students aware of agreement and discussion processes during their collaboration.

As we strive to develop more sophisticated frameworks for analyzing argumentation, we should therefore continue to monitor the possibilities for automating these analytic frameworks directly within online learning environments as real time functionality. Improved analytic frameworks could thereby potentially improve our research capabilities, expand teachers’ options for supporting their students, and customize of real-time supports for students in our schools and universities.