Sequentially Analyzing and Modeling Causal Mapping Processes that Support Causal Understanding and Systems Thinking

Abstract

A growing body of research shows that deeper understanding of complex systems can be achieved when students construct causal maps to articulate, explore, and refine their understanding. However, to what extent is students’ maps an actual measure of their causal understanding versus the efficacy of different processes used by students to construct their map? To address this question, a set of methods was developed and implemented in this study to mine data from the causal mapping tool, jMAP; sequentially analyze the mined data; and construct transitional state diagrams to visualize, model, and compare students’ causal mapping behaviors to identify differences in action sequences performed by students that created maps of high versus low accuracy. The causal maps of 17 graduate students enrolled in an online course on collaborative learning at Florida State University (USA) were sequentially analyzed and used to illustrate how the proposed methods can be implemented to capture, analyze, model, and identify causal mapping and reasoning processes that produce deeper causal understanding. Identifying and validating the most effective processes will enable software developers to create causal mapping software that can standardize the map construction process, control for individual differences in mapping skills, enable students to produce causal maps that accurately reflect their causal understanding (and/or help students to produce more accurate causal maps), and thus enable instructors to use such mapping tools to conduct automated large-scale assessments of students’ causal understanding and systems thinking skills.

1 Introduction

Causal maps, a network of nodes and links that define the causal relationships between nodes, have been used in science education as a tool for teaching and assessing learners’ systemic understanding of complex problems and phenomena (Kali & Linn, 2007; Ruiz-Primo & Shavelson, 1996). Given that causal maps in theory represent learner’s cognitive structures, their complex reasoning, and conceptual development (Jonassen & Ionas, 2008), causal maps can be and have been used to elicit, articulate, refine, assess, and improve analysis, identification, and understanding of the causes and causal mechanism underlying complex problems. The greatest improvements in students’ understanding have been observed when students construct (either individually or collaboratively) their own maps as opposed to simply presenting students the instructor or expert maps (Nesbit & Adesope, 2006). Maps can be used to support collaborative learning when students compare their maps to identify, trigger, and focus group discussions around key differences in viewpoints and understanding (Jeong, 2009, 2010a).

To find new ways to help students construct better causal maps and improve learning, a growing number of studies have formulated various metrics to measure the accuracy and structural attributes of students’ maps (parsimony, temporal flow, total links, connectedness). The goal is to identify the metrics and attributes that are correlated with map accuracy and therefore can be emphasized and presented to students as general guidelines to help them create more accurate maps (Ifenthaler, Masdsuki, & Seel, 2011; Jeong, 2009; Plate, 2010; Scavarda, Bouzdine-Chameeva, Goldstein, Hays, & Hill, 2006). For example, studies have been conducted to determine how different constraints imposed on the map construction process affect student’s maps and learning—constraints like imposing hierarchical order by allowing students to move and reposition nodes (Ruiz-Primo, Shavelson, & Schulz, 1997; Wilson, 1994), providing terms for nodes (Barenholz & Tamir, 1992), providing labels for links (McClure & Bell, 1990), and allowing more than one link between nodes (Fisher, 1990). Among these studies, software tools have been developed to automate and reliably measure both the accuracy and the structural attributes of maps. Software programs like HIMATT (Ifenthaler, 2010) and jMAP (Jeong, 2010b) have been developed to address issues of rater reliability and validity by using the software to measure and test the correlation between different structural attributes and accuracy (Ifenthaler et al., 2011), as well as to measure how maps change over time and how these changes over time contribute to convergence in shared understanding between learners (Jeong, 2010).

However, measuring the structural attributes of students’ causal maps does not appear to provide useful guidelines that will help students construct more accurate causal maps because (a) the many if not most of the structural attributes examined in prior research (Ifenthaler et al., 2011) have not been found to correlate with map accuracy and (b) structural attributes may be largely dependent on the nature or complexity of the problem, domain, or system. Furthermore, Jeong and Lee (2012) found that the effects of an attribute can differ depending on how or when they are implemented during the instructional task and what particular outcome one is trying to achieve. For example, the study’s findings suggest that (a) causal understanding during initial map construction can be adversely affected if students are instructed to temporally sequence nodes to flow from left to right and position the outcome node farther away from the left edge of the map relative to other nodes in the map, (b) causal understanding achieved during the later process of map construction following class discussions on proposed causal links can be increased by instructing students to minimize the number of causal links and create a map with left-to-right temporal flow, (c) temporal flow can make the greatest impact among the three attributes on the ratio of root causes correctly/incorrectly identified, and (d) minimizing number of links can make the greatest impact on understanding how root causes directly/indirectly impact outcomes.

At the same time, students’ maps will vary widely in accuracy when maps are compared to expert maps regardless of what interventions are used to help improve the quality and accuracy of students’ maps (Ruiz-Primo & Shavelson, 1996; Scavarda et al., 2006). Based on their review of the empirical research, Ruiz-Primo and Shavelson (1996) concluded that maps should not be used in the classroom for large-scale assessments until students’ facility, prior knowledge, skills, and processes used to create the maps, as well as training techniques, are thoroughly examined. As a result, the alternative approach to finding ways to help students construct more accurate causal map is to first identify the processes or action sequences that produce more accurate maps. These types of action sequences can include actions at both the local level (e.g., move node→link node to affected node→redirect existing link to point to mediating node→specific causal link strength) and global level (work forward versus backward from outcome node using inductive versus deductive approach, work breadth versus depth first to identify multiple contingent causes versus causal chains). However, no prior research has been found at this time that has empirically identified, tested, and validated specific action sequences that lead to increased accuracy despite the variety of guidelines and logic rules that have been developed to support causal mapping, root cause analysis, and systems thinking (Bryson, Ackerman, Eden, & Finn, 2004; Clarkson & Hodgkinson, 2005; Decision Systems & Inc., 2012; Jonassen & Ionas, 2008; Scavarda et al., 2006). The purpose of this study was to develop and apply a set of methods that can address the following questions:

• 1. What patterns exist in the action sequences performed by students that produce causal maps of high versus low in accuracy?

• 2. Which sequences of actions help students create more accurate causal maps?

2 Methodology

2.1 Procedure

To identify the action sequences that produce more accurate causal maps, seventeen graduate students in an online course on collaborative learning at Florida State University, USA were presented with an activity to articulate and formulate their personal theory as to how certain variables affect team effectiveness in collaborative learning projects. Students were instructed to use jMAP (Jeong, 2010) to individually create a causal map containing seven predefined variables believed to influence team effectiveness in collaborative learning groups (Fig. 13.1). Each student’s goal was to construct a map to convey his/her personal theory as to how the seven variables might be causally interrelated and how they either directly or indirectly affect team effectiveness. The maps they produced were included and presented later in a graded written paper worth 20 % of the course grade. To help students produce theories that were not overly complex, easier to interpret, and hence more parsimonious, the total number of outward pointing arrows stemming from any one node in the maps was limited to a maximum of two. When a given node possessed two outward pointing arrows (and was believed to simultaneously affect two other variables), students were instructed to distinguish which of the two affected variables was most impacted by setting one arrow at “high” impact level and the other arrow at “moderate” impact level. Students were presented a video and practice activity to learn how to move and reposition nodes, insert links between nodes and affected nodes, change the density of the link to convey relative level of impact (high, moderate, low), and change direction of the causal relationship (positive = black, inverse = red). Students were required to log into a designated video conferencing room in Elluminate™ to video record the entire map construction session (limited to 45 min maximum). At the same time, the jMAP software logged each action students performed on their map from start to finish.

Fig. 13.1

Instructor’s map used as criterion for assessing the accuracy of students’ maps

2.2 Data Analysis

A total of 14 causal maps were submitted by the students. The maps were imported into jMAP and assessed in relation to the instructor’s map (Fig. 13.2). The instructor’s map was based on an empirical study that used path analysis to determine the relationships between the variables (Bossche, Gijselaers, Mien, & Kirschner, 2006). Scores from six criterion measures were each multiplied by 10 and summed to determine each students’ total score (Fig. 13.3). The maps produced by students that achieved the top five and bottom five scores were selected for analysis. The jMAP software captured 26 mechanical actions that students performed while constructing their maps (Table 13.1). These 26 actions were collapsed and reduced to six final codes to facilitate the process of identifying overall patterns in students’ action sequences.

Fig. 13.2

jMAP’s visual and quantitative assessment of student1’s map

Fig. 13.3

jMAP’s ranking of students based on scores across six criterion measures. Note: Names have been blurred to maintain anonymity. First order = link stemming directly out of root cause (node with no inward-pointing links). Second order = link from node one link removed from root cause. Third order = link from node two links removed from root cause

Table 13.1 Codes assigned to each action students perform in jMAP software

Next, the code sequences produced by low performers were compiled into a single column in an Excel spreadsheet. The numeral 1 was placed in the cell immediately to the right of the first action performed by a given student to mark the first of a sequence of actions performed by each individual student. The six final codes and the sequence data were entered into the Discussion Analysis Tool (DAT) software (Jeong, 2012) to (a) compute the transitional probabilities between action pairs (Fig. 13.4), (b) compute z-scores at p < 0.01 to identify behavior “patterns”—transitional probabilities found to be significantly higher than expected (bottom of Fig. 13.4), and (c) generate a transitional state diagram to reveal the behavioral patterns exhibited by low scorers (left diagram in Fig. 13.5). This procedure was repeated using code sequences produced by the high performers to produce the transitional state diagram to the right in Fig. 13.5.

Fig. 13.4

Screenshot of the DAT software output presenting the frequency, transitional probability, and z-score matrices. Note: Bold values = transitions significantly higher than expected (p < 0.01). Bold underlined values = transitions significantly below expected value

Fig. 13.5

Transitional state diagrams of action sequences performed by low versus high performers. Notes: Black and gray arrows identify probabilities that are and are not significantly greater than expected. Arrows are weighted in direct proportion to the observed transitional probability. The first and second numerical value displayed in each node identifies the number of times the given action was performed and the number of events that followed the given action. The size of the glow emanating from each node conveys the number of times the given action was performed

The DAT software (Jeong, 2012) is a Microsoft™ Excel-based application that sequentially analyzes temporal events to identify sequential patterns between events using techniques developed by Bakeman and Gottman (1997). The software was used to tally the number of times a particular action was performed, the total number of actions that immediately followed the given action, and the number of times each type of action immediately followed any one given action. For example, the frequency matrix in Fig. 13.4 shows that students in the high-scoring group moved or repositioned the nodes in their maps a total of 321 times (listed in MOVE row and GIVEN column). The number of actions that immediately followed each MOVE was 321 (listed under REPLIES column). When these students moved a node, this action was followed with more moving of nodes by a total of 259 times and adding of links by a total of 32 times. Next, the DAT software and spreadsheet converted the observed frequencies into transitional probabilities to convey how likely one event was to follow another given event (as displayed in the transitional probability matrix in Fig. 13.5). For example, the transitional probability between MOVE→MOVE was 81 % (or 259 divided by 321) given that 259 out of the 321 actions following a MOVE was another MOVE. In contrast, the transitional probability between MOVE→LINK was only 10 % (32 divided by 321).

To determine whether or not the transitional probability between any two actions was a behavioral pattern, z-scores were computed for each transitional probability and reported in the z-score matrix presented in Fig. 13.4. Each z-score determined whether a given transitional probability was significantly higher or lower than the expected transitional probability based on chance alone. A z-score that was significantly less than the critical z-score of −2.32 at alpha <0.01 identified a transitional probability that was significantly lower than the expected probability. These z-scores (and their associated frequencies and transitional probabilities presented in the frequency and transitional probability matrices) were underlined and highlighted in bold red-colored font. A z-score that was significantly greater than the critical z-score of 2.32 at alpha <0.01 identified a transitional probability that was significantly higher than the expected probability (values highlighted in bold green-colored font). The z-score statistic (Bakeman & Gottman, 1997) takes into account not only the observed total number of responses following each given behavior but also the marginal totals of each response observed across all behaviors. One important note is that paired actions with significant z-scores but with observed frequencies of less than 4 should not be identified as a behavioral pattern because the low cell frequencies are not sufficient in size to compute a reliable z-score.

Finally, DAT software was used to convert the observed transitional probabilities into two transitional state diagrams (a network of nodes used to represent each action linked or interconnected by lines/arrows that depict their sequential order) to provide a graphical and Gestalt view of the behavioral patterns produced collectively by the high- versus the low-scoring students (Fig. 13.5). The density or thicknesses of the arrows are directly proportional to the observed transitional probabilities between given nodes/behaviors so that high probabilities are identified with large thick arrows and low probabilities are identified with thin arrows. Arrows were either presented in solid black, to convey a transitional probability with a significant z-score, or in light gray, to present a transitional probability with a nonsignificant z-score. The relative size of the glow emanating from each node in the state diagrams conveys the relative frequencies in which each particular action was observed. For example, the left state diagram in Fig. 13.5 shows that students in the low-scoring group moved nodes a total of 101 times but commented on the links only 37 times. As a result, the size of the glow emanating from the MOVE node is 2.7 times larger (101 divided by 37) than the glow emanating from the COMM node. Finally, the first and second numerical value displayed in each node identifies the number of times the given action was performed and the number of events that followed the given action. These features of the transitional diagrams enable one to visually compare the state diagrams of the low- versus high-scoring students when placed side by side and, by doing so, identify the similarities and differences in behavior patterns produced by each group. The patterns that distinguish the high- from the low-scoring students help to identify processes that produce and/or explain how and when students produce more accurate causal maps.

3 Discussion

3.1 Low Performers

The transitional state diagram for low performers (Fig. 13.5) reveals four behavioral patterns. The patterns reveal their tendency to construct each component of their causal maps in separate discrete stages. These students tended to start first by moving multiple nodes into their respective positions. Once multiple nodes were positioned, they inserted links between the nodes one after another. Once links were added, these students specified the attributes of each causal link. In addition, once links were inserted, these students tended to insert their comments into one link (one link after another) to explain the causal mechanism underlying the inserted link.

3.2 High Performers

The state diagram for high performers in Fig. 13.5 also revealed the same four behavioral patterns exhibited by low performers. However, high performers exhibited four additional patterns. These additional patterns were attributed in part to how the higher performers relinked and deleted links more often than low performers. These patterns may help to explain how students can produce more accurate maps. The high performers exhibited the tendency to (1) specify the attributes of a causal link that is inserted between two nodes, (2) add a new link immediately after they redirect an existing link, (3) add a new link immediately after they deleted a link, and (4) delete another link immediately after deleting a link. These patterns serve as possible indicators that show that high-scoring students produced more accurate maps because they engaged more frequently in the process of iteratively revising their causal maps (DEL→AddLink; Relink→AddLink; DeleteLink→DeleteAnotherLink). The observed patterns might also serve to demonstrate that high-scoring students applied rules of logic or reasoning like those embedded in the REASON software (Decision Systems & Inc., 2012). For example, a logical reasoning process can lead a student to realize that both A and B do not exert direct effects on C (when realizing that C can occur without A), but instead, A directly effects B, which in turn directly effects C. This line of reasoning involves the process of inserting links between A and C, and B and C, then deleting the link between A and C followed by adding the link from A to B or redirecting the existing A→C link that points to C so that it now points instead to node B.

The unique behavioral patterns observed among the high performers suggest that redirecting and deleting links are a critical part of the mapping process. To determine what prior actions tend to trigger Relink and Delete actions, the historical state diagram (Fig. 13.6) shows that (a) the event that was most likely to precede the Relink action was the Move action (62 % of all preceding events) and Link action (100 %) among the high and low performers, respectively, and (b) the event most likely to precede the Delete action was the Link action (45 % of all preceding events) and Move action (37 %) among the high and low performers, respectively. In general, the historical diagrams like the ones presented in Fig. 13.6 can be used to identify the triggers and/or prior actions that are most likely to initiate specific action sequences that are of key interest.

Fig. 13.6

Historical state diagrams of actions most likely to elicit given action performed by low versus high performers

4 Conclusion

The findings from this study illustrate how the presented methods can be used to identify specific action sequences that can be prescribed to help students produce more accurate causal maps and to achieve deeper causal understanding of complex systems. One way to improve on the methodology presented above is to include a pretest that measures students’ prior understanding of the causal relationships to tease out the effects of prior understanding from the effects of individual differences in processes used by students on students’ causal understanding. Because the number of causal maps examined in this study was too small in number to produce any conclusive findings, larger samples are needed to determine if the reported findings can be replicated. In order to identify and examine the logic rules and reasoning processes/strategies used by high-scoring students, the coding scheme will need to be refined in a way that can verify, for example, when a Relink is or is not redirected to a mediating node (where A→C and B→C is changed to A→B→C).

Furthermore, a more detailed breakdown and analysis of the mined data will be necessary to determine to what extent students use backward/deductive versus forward/inductive approach and/or depth versus breadth-first approach (work in progress), and ultimately to determine which approach is most effective. For example, the segment of data presented in Fig. 13.7 shows that the student linked node 7 to node 4. Immediately following this action, the student linked node 1 to node 7 to perform a backward or deductive reasoning process to produce the 1→7→4 causal chain. Immediately after linking 1→7, the student linked 7 to 5 to produce the 1→7→5 causal chain—an action sequence that illustrates a forward/inductive reasoning process. Both of these two behaviors serve to illustrate a depth-first approach to identifying the causal chains. The data in Fig. 13.7 also shows that the student linked 7→5 and then linked 7→4 to produce two parallel causal chains that stem from node 7. These behaviors are indications that the student also used a breadth-first reasoning process.

Fig. 13.7

A segment of the mined data from jMAP used to measure deductive versus inductive reasoning processes

Once the described method and tools have been refined and validated over time, further research and developments in the instructional application of knowledge mapping can be conducted by (a) integrating the target action sequences directly into causal mapping software interface to conduct controlled experimental tests to determine if the target processes increases map accuracy regardless of students’ prior knowledge/understanding, (b) examining to what extent the target processes are effective across different topics that are and are not temporal in nature, and (c) revisiting and retesting criterion measures identified in prior research for assessing the accuracy or quality of knowledge maps while controlling for the specific processes students use while constructing their causal maps.