Introduction

Achieving a coherent understanding of chemical reactions is difficult for students using curriculum featuring static pictures in textbooks (Gabel 1999; Johnstone 1991), as these pictures do not convey the dynamic interactions of the molecules and atoms (Pedrosa and Dias 2000). Conservation of mass and chemical reactions are central to chemistry and serve as the foundation for many topics such as stoichiometry, limiting reactants, chemical equilibrium, and acid/base reactions (National Research Council 2011). Furthermore, many students struggle with a molecular understanding of conservation of mass or chemical reactions (Boo and Watson 2001) and have difficulty understanding the dynamic nature of chemical reactions at the molecular level (Ben-Zvi et al. 1987).

In this paper, we explore the value of adding dynamic visualizations to instruction. Visualizations enable students to interact with chemical reactions on a molecular level. Students can manipulate heat, pressure, or even bond types and energies and see how these variables impact chemical reactions (Xie and Tinker 2006). Dynamic visualizations can help students form a molecular understanding of these concepts by providing representations of processes that are normally invisible to the naked eye. Meta-analyses suggest an overall value of dynamic visualizations for science learning, but many individual studies show little to no benefit for learning from visualizations (Hoffler and Leutner 2007; Honey and Hilton 2011; Smetana and Bell 2012; Wu and Shah 2004). To benefit from visualizations, students need to understand and interpret them and make connections to underlying scientific concepts (Ainsworth 2006). Novice students studying chemical reactions may only pay attention to surface features of dynamic visualizations (e.g., Lowe 2004) or fail to make inferences about how the visualization connects to conservation of mass or chemical reactions.

Recent studies suggest that a combination of visualizations and instructional support can help students link molecular interactions to relevant chemistry concepts (Ardac and Sezen 2002; Kozma 2003; Plass et al. 2009; Stieff and Wilensky 2003; Wu et al. 2001). Research in science instruction shows that students succeed when they are guided to make predictions, add missing ideas, distinguish among the mix of ideas, and reflect on their progress (Bransford et al. 2000; Kali 2006; Williams et al. 2012). Successful visualization-enhanced chemistry instruction should help students build and reflect upon connections among visualizations and chemistry concepts (Smetana and Bell 2012). In this study, we explore ways to combine rich, dynamic visualizations with guidance designed to promote coherent links among ideas. We investigate ways to support high school students studying chemical reactions to develop these connections as they carry out an online inquiry unit designed to promote integrated understanding. We compare the visualization-enhanced unit to typical instruction and look at immediate as well as course-long impacts. In particular, this paper investigates the following questions:

  • How can inquiry instruction be designed following research-based guidelines to take advantage of molecular visualizations and help students learn about chemical reactions?

  • How does a visualization-enhanced unit, compared to typical instruction, impact the coherence of student ideas immediately after instruction and after study of additional topics during a year-long chemistry course?

  • What links and connections among ideas do students develop while studying the visualization-enhanced unit and how do these develop during subsequent course experiences?

Integrating Ideas in Chemistry

Chemistry students at all levels have diverse and often contradictory ideas about conservation of mass and chemical reactions (Ben-Zvi et al. 1987; Calik and Ayas 2005; Haidar 1997; Mulford and Robinson 2002; Ozmen and Ayas 2003; Yarroch 1985). For instance, students often think of chemical reactions as an additive process rather than an interactive process of bonds breaking and forming (Ben-Zvi et al. 1987). Students may be able to accurately predict products of reactions, but neglect ideas about energy and bonding (Boo 1998). Similarly, students may be able to balance chemical equations mathematically, but have little to no understanding of what the equations represent on a molecular level (Yarroch 1985). Students also fail to connect conceptual explanations to symbolic representations and algorithmic procedures used to describe chemical change processes (Agung and Schwartz 2007). Inaccurate or incomplete ideas about chemical reactions persist after explicit instruction (Hinton and Nakhleh 1999) and even after completing undergraduate degrees in chemistry (Bodner 1991).

Alternative student ideas about chemical reactions can come from multiple sources. Learners have intuitions about observable phenomena from nature and daily life, such as everyday observations of carbonated beverages, metals rusting, or common combustion reactions involving gasoline or candles (Bransford et al. 2000). Students may have existing ideas from other domains like mathematics about representations used to describe chemical reactions like subscripts and coefficients (Krajcik 1991). Students can also acquire non-normative ideas from static images of chemical reactions in textbooks (e.g., Pedrosa and Dias 2000; Theile and Treagust 1995).

Students not only have a large, incoherent repertoire of ideas about chemical reactions but also have difficulty connecting different levels of representation in chemistry: the observable realm of visible phenomena; the molecular and atomic level; and symbolic chemistry: the equations, mathematics, and stoichiometry-describing phenomena (Gabel 1999; Johnstone 1991). Connecting these levels is fundamental to understanding modern chemistry (National Research Council 2011). Students observe chemical reactions in labs, see molecular pictures of chemical reactions in textbooks, and use symbols in chemistry to solve math-like problems. Experts easily connect and traverse these different levels, but students often have isolated or partially connected ideas (Kozma 2003). For instance, ideas about subscripts and coefficients from mathematics can hinder connections to the macroscopic or molecular representations of chemical phenomena. Studies show that students need guidance to reflect upon and sort out their ideas at various levels in chemistry (Tien et al. 2007). Developing this kind of representational competence is of great importance to current chemistry education efforts (e.g., Grove et al. 2012; Kozma and Russell 2005; Levy and Wilensky 2009; Stieff 2011; Talanquer 2011).

Dynamic visualizations can help students understand chemical reactions by making typically unobservable molecular levels of chemical processes available to the naked eye (Stieff and Wilensky 2003). Dynamic visualizations differ from static visualizations in textbooks (such as graphics, models, and diagrams) in that they display processes of scientific phenomena that change over time (Ainsworth and VanLabeke 2004; Tversky et al. 2002). Sophisticated dynamic visualizations like computational models and simulations allow students to interact and experiment on very small or large scales with phenomena such as molecular dynamics (Pallant and Tinker 2004), genetics (Buckley et al. 2004), and electricity (Finkelstein et al. 2005). Students can use dynamic visualizations to build understanding by generating hypotheses, testing their hypotheses by interacting with the dynamic visualization, and refining hypotheses and understanding by reflecting upon the dynamic visualization (Linn et al. 2010).

Visualization-based instruction has been shown to be particularly effective for learning chemistry (Chang and Quintana 2006; Levy and Wilensky 2009; Pallant and Tinker 2004; Williamson and Abraham 1995). Students can use dynamic visualizations to develop improved understanding of chemical reactions on a molecular level (Ardac and Akaygun 2005; Levy 2012). For instance, Ardac and Akaygun (2004) compared students using a multimedia environment that included visualizations, videos, drawings, and interactive assessments to students receiving textbook-based instruction. Students using the visualization-based 2-week curriculum outperformed students with regular instruction from pretest to posttest. Fifteen months later, students in the treatment group used molecular representations more often and accurately than students with regular instruction.

Although many studies report positive effects of dynamic visualizations for chemistry, the effect of visualizations on science learning is contested (Mayer et al. 2005; Tversky et al. 2002). Visualizations can overload learners, and students can overestimate their understanding of visualizations, fail to monitor their progress, and acquire superficial understanding (Betrancourt 2005; Chiu and Linn 2012; Cook 2006; Hegarty 2004; Lowe 2004; Moreno and Valdez 2005; Zhang and Linn 2011). Students using visualizations not only need to attend and make sense of the visualization itself but also connect the visualization to the underlying scientific phenomena or learning objectives (Plass et al. 2009). Although some visualization-enhanced curricula can increase student understanding of chemistry, researchers warn that these representations can confuse students (Boo and Watson 2001). Students can passively observe or superficially interact with dynamic visualizations similar to passively listening to a lecture or watching a video (Lowe 2004). For instance, students using a molecular visualization may focus on how fast the molecules move, manipulate settings to make the molecules move as fast as possible, and think they understand the visualization and move on without making connections to underlying concepts such as energy. Learners can struggle to learn productively with visualizations when used in isolation or without additional instructional support (de Jong and van Joolingen 1998; Rieber et al. 2004).

Providing effective guidance with visualization-based instruction can have a large impact on how students interact with and how much students learn from dynamic visualizations (Honey and Hilton 2011; Plass et al. 2009). Research suggests that prompting students to explain connections across representations and making sure that students reflect upon their understanding can help students make connections among ideas (Ainsworth et al. 2002; Ardac and Akaygun 2004; Bodemer et al. 2004). Successful studies using molecular visualizations guide students to make links among symbolic, molecular, and observable phenomena and help students sort out the varied and contradictory ideas they bring to class (Ardac and Sezen 2002; Levy and Wilensky 2009; Kozma 2003). For instance, eChem (Wu et al. 2001) guided students through comparisons of molecular and macroscopic representations, construction of molecular models, and 3D visualizations in a 6-week chemical toxin curriculum. Transcripts revealed that eChem visualizations helped students recognize conflicting ideas about chemical structure and bonding. Students improved on measures that assessed conceptual understanding of molecular and observable levels as well as the ability to connect levels of representation.

Determining how to design guidance for visualization-enhanced instruction requires research. To synthesize chemistry research findings, Wu and Shah (2004) put forth design principles to help students develop visualization skill in chemistry. Principles included (1) providing multiple representations and descriptions, (2) making linked referential connections visible, (3) communicating the dynamic and interactive nature of chemistry, and (4) making information explicit and integrated for students. Developing students’ ability to link representations including molecular visualizations has shown promise in some research (Levy and Wilensky 2009; Stieff 2011). Instruction that guides students to link multiple, coordinated representations of phenomena has improved learning outcomes (Kozma 2000, 2003) in some studies but not in others (Hegarty 2004). Other studies suggest that providing explicit support for students to make connections among levels can result in long-term impacts on student learning (Ardac and Akaygun 2004).

Knowledge Integration

The knowledge integration (KI) perspective can be particularly beneficial for providing guidance with dynamic visualizations as it focuses on helping students build and retain connections among scientifically relevant ideas and existing knowledge (Kali 2006; Linn 1995; Linn et al. 2004; Williams et al. 2012). KI draws on research showing that learning is a process of integrating ideas—adding, sorting, evaluating, distinguishing, and refining accounts of experiences and phenomena (Bransford et al. 2000). Developmental, sociocultural, cognitive, and constructivist research illustrates the importance of connecting new ideas to prior knowledge, as well as promoting deliberate, reflective learning (Linn and Eylon 2006). KI takes advantage of the rich, diverse, and conflicting ideas that students bring to class from various contexts and experiences about scientific phenomena like chemical reactions (Clark 2006). Instead of viewing non-normative ideas as something to be replaced, KI views alternative existing ideas as resources that careful instruction can use to promote more integrated and durable understanding of science (Linn and Eylon 2011). Since students tend to isolate ideas in chemistry and fail to distinguish ideas that they gain through dynamic visualizations, instruction guided by the KI perspective can be especially beneficial for learning with dynamic visualizations (Clark et al. 2008; Kali and Linn 2008).

This paper explores how a visualization-enhanced inquiry project designed to promote coherent links among ideas can help high school chemistry students develop connections among ideas and representations of chemical reactions. We discuss the design and refinement of a visualization-enhanced inquiry chemistry project using KI processes and principles. We compared students using the visualization-enhanced inquiry project to students receiving traditional textbook-based instruction and investigated immediate as well as long-term impacts. We aimed for results of this study to contribute to the ongoing dialogue of how to provide effective guidance for visualization-enhanced chemistry instruction.

Curriculum Design

Partnership Design Process

A partnership of teachers, researchers, technologists, and discipline experts participated in the development and refinement of the visualization-enhanced chemistry unit, Chemistry Scene Investigators (hereafter CSI, Appendix 1). Teachers, including those involved in the studies reported here, participated in the iterative design of CSI, giving ideas and feedback about what would work in their classrooms.

Chemistry scene investigators (CSI) was implemented in the Web-based Inquiry Science Environment (WISE). WISE is a free, open-source, online science inquiry environment based on over 20 years of research and refinement that offers pedagogical tools such as an inquiry map, drawing tools, concept mapping, and embedded assessments (Fig. 1). WISE logs student work and engages students in KI processes (Slotta and Linn 2009).

Fig. 1
figure 1

Screenshot of the CSI curriculum unit in WISE

Full size image

We used design-based research to understand how dynamic visualizations combined with research-based guidance can help students learn about chemical reactions in classroom environments (Brown 1992; Collins 1992; The Design-Based Research Collective 2003). Iterative cycles of design, implementation, analysis, and refinement of CSI were used to understand how guidance and visualizations impacted learning.

Creating CSI Guidance

To create CSI guidance, we combined a KI perspective and related visualization research. The KI framework includes curriculum design meta-principles and patterns that focus on helping students build on their productive ideas, make links among their ideas, and sort out connections between new ideas and existing networks (Linn and Eylon 2011).

KI Meta-principles in CSI

The KI meta-principles emerged in longitudinal and experimental studies (Linn et al. 2004; Linn and Hsi 2000). We employed KI design meta-principles in the design of CSI to help chemistry students learn from dynamic visualizations and make connections among ideas. The four meta-principles and their relevance to CSI are:

Make science accessible calls for encouraging students to connect new knowledge to preexisting knowledge and appreciate the relevance of science to their lives. CSI made science accessible by guiding students to investigate how chemical reactions relate to climate change, with specific focus on how combustion reactions contribute to greenhouse gases. Students in CSI were charged with making a recommendation about allocating research dollars to different programs for mitigating greenhouse gas production.

Make thinking visible refers to the process of modeling and critiquing how ideas are connected and organized in normative understanding as well as in students’ repertoires. Molecular visualizations in CSI aimed to make the workings of chemical reactions visible for students who may not have a clear mental picture of chemical reactions on a molecular scale.

Help students learn from others calls for encouraging the use of ideas and beliefs of others to develop criteria and refine one’s own understanding. In CSI, students worked in pairs to collaborate and share ideas.

Promoting autonomy and lifelong learning involves helping students refine their knowledge by monitoring and reflecting upon their ideas throughout their studies. In CSI, students followed an inquiry map and made decisions about the pacing of their investigation; the exploration of links between observable, symbolic, and molecular representations; the use of hints; and the implications of the insights they gained.

KI Instructional Pattern

The KI instructional pattern synthesizes results from numerous design studies and provides guidance to promote KI processes of eliciting existing ideas, adding new normative ideas, distinguishing among existing and new ideas, and reflect upon new connections among ideas (Linn and Eylon 2006). Since students tend to isolate ideas in chemistry, instruction guided by the KI perspective can be especially beneficial for learning with dynamic visualizations (McElhaney and Linn 2011).

Eliciting existing ideas acknowledges the individual backgrounds and experiences that students bring to learning contexts. Eliciting ideas insures that learners consider the ideas they have developed over many years in interpreting new ideas and enables teachers to appreciate the ideas their students bring to science class.

Adding normative ideas is often the sole goal of science instruction. From the KI perspective, the goal is to design instruction so that new ideas provoke students to reconsider their initial ideas (Linn 2005). Presenting new ideas carefully can also stimulate students to make connections across contexts. Designing visualizations so that they add normative ideas often requires iterative refinement because initial versions are too complex, overwhelming, or confusing.

Distinguishing among alternative ideas is often neglected in science instruction because designers assume that the normative ideas will be compelling. However, students rarely embrace new ideas, especially if they have evidence to support their existing ideas (Zhang and Linn 2011). For example, students may neglect bond breaking and formation because they believe that matter is composed of small pieces of the observable material. To distinguish ideas, students need to develop criteria for scientific evidence that can be cultivated individually by deliberate and intentional learners or socially constructed by groups and communities of learners.

Reflecting on connections among ideas is a well-established way to develop robust understanding by monitoring progress and applying ideas to novel situations (Bransford et al. 2000). Students reflect on their views when they synthesize their knowledge in essays and reports.

Taken together, the meta-principles and KI design pattern combined with related chemistry education research informed decision making for the unit (e.g., Quintana et al. 2005; Wu and Shah 2004). Instead of simply adding ideas, CSI explicitly encouraged students to distinguish among new and existing ideas, to develop criteria for selecting among competing accounts of scientific phenomena and to deliberately reflect upon and monitor connections among their ideas with the goal of developing lifelong learners.

CSI Visualizations

Chemistry scene investigators (CSI) used Molecular Workbench simulations that visualize atomic motion based on real-time estimations of classical dynamics and applicable forces (Pallant and Tinker 2004; Xie and Tinker 2006; Xie et al. 2011). CSI designers created interactive dynamic visualizations that enabled students to manipulate chemical reactions on a molecular scale and change variables or settings of the model and see different outcomes. We used dynamic molecular visualizations of chemical reactions to help students add ideas about the dynamic and interactive nature of chemical reactions, as well as connections between energy and molecular speed.

Chemistry scene investigators (CSI) provided multiple, linked interactive dynamic visualizations to help students integrate ideas at different levels in chemistry (Stieff and Wilensky 2003). Research suggests that linked representations, such as dynamic real-time graphs and molecular animations, can benefit students (Kozma 2003). In CSI, dynamic molecular visualizations were connected to real-time concentration graphs and the symbolic chemical formulae to help students add and refine ideas about connections among the molecular and symbolic levels of chemical reactions, as well as intermediate and limiting reagents.

Other research suggests that student construction of molecular models and 3D visualizations help students make connections among ideas (Chang and Quintana 2006; Schank and Kozma 2002; Wu et al. 2001). We designed an activity in CSI to feature interactive molecular visualizations that guided students to manipulate chemical species to form product molecules from reactants. In the visualizations, students broke bonds among reactant molecules, moved atoms around, and created different numbers of product molecules. This activity targeted understanding of conservation of mass and limiting reagents on a molecular level.

Combining Visualizations and Guidance

Since many students fail to distinguish or reflect upon ideas from dynamic visualizations (Chiu and Linn 2012; Lowe 2004), we used the KI patterns to structure instructional supports to visualizations in CSI. For the visualization design, CSI guided students through repeated experiences with complex visualizations and guided students to revisit the same visualization with different goals. In this way, we aimed for students to become familiar with and be able to interpret the visualizations by addressing certain pieces of the visualization at different times, with specific foci.

For the design of the instructional supports surrounding visualizations, CSI used embedded explanation prompts that encouraged students to explain connections among levels and ideas. To reveal students’ existing ideas, CSI prompted students to explain their initial ideas. For example, before a dynamic visualization of a combustion reaction, students were prompted to explain differences between chemical formulae and chemical reactions. Students could then add normative ideas about the meaning of coefficients and subscripts on a molecular level through the dynamic visualization. Eliciting students’ existing ideas ideally leads students to identify how their ideas may conflict. Many students believe that coefficients relate only to the first atom in a molecule (Ben-Zvi et al. 1987). CSI aimed to help students distinguish ideas by explicitly prompting students to explain their observations of the simulation and connections to representations (e.g., “How did the molecular simulation relate to the chemical equation?”). CSI guided students to monitor and reflect upon their knowledge to find gaps or discrepancies in their understanding. For instance, students were guided to apply newly developed criteria of coefficients to other symbolic representations of molecules, like 2H2O or 4NH3. In this research, we refined the unit by implementing the KI pattern and testing to see whether student responses to embedded assessments showed that they were integrating their ideas.

We hypothesized that taking a KI-based approach for instruction with dynamic visualizations would be particularly beneficial for learning about chemical reactions because dynamic visualizations would help students “see” dynamic and interactive aspects of chemical reactions, and KI-based instructional support would help students make connections among their ideas. Additionally, we hypothesized that there may be long-term impacts on students using CSI since chemical reactions relate to many subsequent topics in chemistry.

Methods

The three studies presented in this paper describe iterative design-based research on learning with CSI. The first study reports on a pilot of the CSI curriculum, results, and following refinements. The second study compared CSI to typical textbook-based instruction to investigate how CSI could help students connect ideas in chemistry compared to typical approaches. In the third study, we investigated course-long effects of CSI by administering delayed posttests at the end of the year, months after students studied with CSI.

KI Pretests and Posttests

Pretests and posttests assessed the students’ knowledge of the learning goals of the unit using the validated KI assessment construct and framework (Liu et al. 2011). Both pretests and posttests consisted of 7 open response KI items and one typical balancing equations item. The KI items asked students to relate molecular and symbolic representations of balanced equations (Items 1–4), critique an answer to balance an equation without conserving mass (Item 5), draw the products of a limited reaction on a molecular scale (Item 7; adapted from Mulford and Robinson 2002), and explain connections between temperature and molecular speed (Item 8). Items 1–2 and 7–8 also addressed the dynamic and interactive nature of chemical reactions. Several items (1–4, 7) required students to generate molecular representations from symbolic representations of balanced chemical equations and vice versa to target connections among symbolic and molecular levels of chemical reactions. One non-KI item on balancing equations was included to assess traditional understanding of numerically balancing equations (Question 6). Appendix 2 contains pretest and posttest items.

Scoring Rubric

All KI assessments were scored using a KI rubric (Appendix 3), which identifies the numbers of connections students make among ideas (Linn et al. 2006; Liu et al. 2011). Higher numbers of connections among scientifically relevant ideas resulted in higher KI scores. For example, question 2 on the pretest asked students to draw a molecular representation of the chemical equation 2H2 + O2 → 2H2O (Fig. 2). The rubric coded for both understanding of and connections among ideas such as coefficients and subscripts on a molecular scale and conservation of mass. A response with connections among ideas would receive a score of 4. In contrast, a response demonstrating no integration of ideas would receive a score of 1. A student demonstrates a connection by consistently using ideas throughout the answer. A partial link is coded if the student showed inconsistent use of ideas throughout the answer. For instance, the student’s response that earned a score of 3 in Fig. 2 drew the correct molecular structure for 2H2 but instead of 2H2O drew a molecule of H4O2. This student demonstrated a full link to conservation of mass on a molecular scale, but a partial connection to coefficients and subscripts on a molecular scale.

Fig. 2
figure 2

Knowledge integration scoring guide for pretest and posttest Item 2

Full size image

The scoring guide did not penalize students for aspects of chemical reactions and representations not addressed in the curriculum. For instance, if students demonstrated understanding that coefficients relate to the number of molecules and subscripts refer to the number of atoms within molecules, then they received full credit for their response. Other structural concepts such as bond angles and number of bonds did not affect the overall scoring of the item. Likewise, if students did make connections to other scientifically relevant ideas not in CSI, then the response would receive credit for a link to other relevant and normative ideas. For all KI assessments, two assessment specialists as well as two researchers helped to confirm the trustworthiness of the scoring rubrics through coding representative samples of responses. All assessments for all studies were coded blind to group membership. Any disagreements between raters were discussed and resolved so that reported pretest and posttest scores reached 100 % agreement.

Study 1: Pilot Study

We pilot tested the CSI curriculum to determine its strengths and limitations. The design partnership used evidence from embedded assessments, pretests, and posttests to refine the curriculum for the comparison studies.

Pilot Study Participants

Twenty-one honors chemistry students from an urban high school participated in the pilot implementation of the CSI. In the pilot run, students had not yet covered topics of balancing equations. Students attended the 10th and 11th grades and came from a variety of ethnic and economic backgrounds. The school’s population consists of 87 % African–American, 8 % hispanic, 2 % white, 2 % multi-racial, and 1 % asian, with 62 % of students on free or reduced meals. The teacher and students had no previous experience using WISE. The teacher selected pairs of students within each class to work through the entire unit together, as typical in WISE units.

Pilot Study Methods

The pretests were administered before instruction began to individual students. During instruction, students worked in pairs. After instruction, the posttest was administered to individuals.

Pilot Study Results

Technical difficulties caused CSI to stretch over 5 weeks. The teacher interrupted use of CSI and went on to the next subject with the class for 3 weeks while the technology department fixed bandwidth problems. Once the problems were fixed, the teacher resumed CSI. This interruption may have impacted the pretest to posttest gain.

To understand how dynamic visualizations and prompts can help students form more integrated views of chemical reactions, effect sizes from pretests to posttests were calculated using the pooled standard deviation (Cohen 1988; Rosnow and Rosenthal 1996). Students learning these topics for the first time made major gains while studying CSI (Table 1). The pilot group significantly improved from pretest to posttest (Pilot: t = 5.44, df = 20, p < 0.05). Item analysis revealed that students gained insights for all items and the most progress occurred on items concerning balancing equations and heat and molecular motion.

Table 1 Average knowledge integration scores, standard deviations, and effect sizes of pretests and posttests for Pilot, CSI, and comparison groups
Full size table

Pilot Curriculum Revisions

The partnership reviewed the embedded assessments, pretests, and posttests and revised the CSI unit including the visualizations used with CSI as well as the surrounding instruction. First, we simplified the Molecular Workbench visualizations. In spite of the documented success of the unit, classroom observations revealed that students had difficulty sorting out information presented in the visualizations, consistent with other research (Lowe 2004; Tversky et al. 2002). Students asked their partners or their teacher what purpose the visualizations served, what they were supposed to do with the visualizations, and how the visualizations related to the unit. Because the visualizations loaded slowly, we put instructions for students to read before interacting with the visualizations. However, many students forgot or had trouble remembering the instructions once the visualization loaded in a separate window. Based on these findings, we decreased the file size of visualizations to decrease loading times and reduce technical difficulties. The supporting text was moved to appear with the visualizations. In addition, we revised the wording and text of surrounding curriculum and embedded explanation prompts to make links between the visualizations and the overall inquiry more explicit.

Study 2: Comparison Study

We conducted a comparison study to determine the effect of the CSI unit compared to typical instruction using the revised version of CSI. All students took the pretests and posttests.

Comparison Study Methods

Study 2 students came from the same urban high school taught by the same teacher as those who participated in the pilot study. Two general chemistry classes participated in the comparison study. Classes were assigned to the CSI and comparison condition based on the logistics of distribution of the computers. We use the pretest to adjust for any differences between the classes.

The CSI group of 24 general chemistry students studied CSI. The comparison group of 25 general chemistry students took the pretests and posttests. These students had covered most topics of chemical reactions, balancing equations, and limiting reactants. The students had no previous experience using WISE. The teacher had taught the pilot version of the curriculum earlier in the year. As in the pilot study, the teacher selected pairs of students within each class to work through the entire unit together. The same pretests and posttests and scoring rubrics as used in the pilot study were used for the comparison study. The teacher gave identical pretests and posttests to both classes on the same days. Students in this comparison group received lecture and text-based chemistry instruction instead of the CSI curriculum.

Comparison Study Results and Discussion

Pretest and Posttest Results

Mean scores, standard deviations, effect sizes and average KI level for the CSI and comparison groups are presented in Table 1. There were significant differences between pretest scores for the two groups (p < .05). Students in the CSI group on average started between partial and full links on the pretest, whereas students in the comparison group on average began around the partial link level on the pretest. Posttest scores were regressed with pretest scores and group as explanatory variables (r 2 = 0.77, F(2, 46) = 78.59, p < 0.001). Pretest score and group had significant effects on posttest scores (Pretest: b = 0.81, p < 0.001; Group b = 2.29, p < .05). After controlling for pretest scores, CSI and comparison groups significantly differed at the .05 level.

The CSI group gained from pre to post, with a medium effect size. Although the comparison group started with somewhat lower pretest scores than CSI students, only the CSI students improved. This suggests that CSI helped students make connections among ideas and levels of representations of chemical reactions.

Comparing the CSI group to the pilot group shows that CSI students made progress in linking and connecting ideas whether they started with minimal knowledge or with reasonably sophisticated understanding. The pilot group made large gains but had posttest scores that were lower than the pretest scores for the CSI group. The greater gains of the pilot group reflect their greater opportunity to learn. These gains may also have resulted because the pilot students were in high school honors chemistry, while the CSI students were in general high school chemistry or because the CSI instruction was interrupted and the duration of the unit was lengthened.

Individual Item Performance

To analyze the impact of CSI, we analyzed the pretest posttest gains by item for the pilot, CSI, and comparison group (distinguishing between the one non-KI and other KI items). We calculated item-level effect sizes by group (Table 2).

Table 2 Average knowledge integration scores and standard deviations for individual pretest and posttest items by group
Full size table
Balancing an Equation

On the non-KI balancing equations question (Item 6), students balanced a combustion reaction but were not asked to explain their reasoning. The question was scored for correctness out of 3 points. Pilot students started with many blank or irrelevant responses and made progress toward correct responses, with most still having incorrect balanced equations or numbers of atoms [Pretest M(SD) = 0.52(0.51); Posttest M(SD) = 1.05(0.50)]. The CSI group started with many correct answers, and scores declined on the posttest [Pretest M(SD) = 2.46(1.25); Posttest M(SD) = 1.96(.95)]. Comparison groups started with many correct answers and also declined on the posttest [Pretest M(SD) = 2.12(.97); Posttest M(SD) = 2.00(1.04)].

The CSI group had pretest scores that were considerably higher than the posttest scores for the pilot group (KI levels on average of 2.4 vs. 1), suggesting a ceiling effect for the CSI students on the pretest. Additionally, students could have confused products and reactants. For this typical item, classroom instruction was effective and CSI did not add value for the procedure of balancing equations. These results are similar to other results comparing typical assessment items to KI-based items (Linn et al. 2006).

Chemical Representations

Items 1–4 assessed students’ ability to translate symbolic and molecular representations. Pilot group pretest scores started on average with partial links and progressed to close to full links on the posttest [Pretest: M(SD) = 2.22(1.48); Posttest M(SD) = 2.91(1.43)]. Scores reflect gains in understanding of conservation of matter at the molecular level. The CSI group started above the simple link level on the pretest and made progress toward complex links on the posttest [Pretest M(SD) = 3.21(1.31); Posttest M(SD) = 3.61(.97)]. CSI scores reflect an understanding of conservation of matter along with insights into bond breaking and formation at the molecular level. The Comparison group started between the partial and simple connection level and remained at that level [Pretest M(SD) = 2.55(1.24); Posttest M(SD) = 2.71(1.26)]. They did not make progress in linking symbolic and molecular representations.

Results suggest that KI-guided support to link molecular visualizations to the symbolic equation may have helped students articulate and distinguish connections among symbolic and molecular representations. Students in the CSI group gained in ability to link the symbolic and molecular representations when asked to draw molecular representations of chemical equations (Item 2). The comparison group made smaller gains than CSI or pilot students, possibly because their primary source of information was the textbook. These questions did not require students to explain their answers or articulate specific differences among concepts or representations such as coefficients and subscripts.

Students involved in CSI made moderate progress in articulating and distinguishing symbolic and molecular representations for single chemical formulae (Items 3 and 4). In contrast, the comparison group made little progress. For example, item three asked students to explain the differences between molecular representations of 2NO and NO2 using symbolic representations. Both Pilot and CSI students improved their scores for this item, while comparison group students did not. On the pretest, some students were able to identify correct symbolic formulae, but many students were not able to explain the connections among the symbolic and molecular representations. On the posttest, Pilot and CSI groups were able to articulate connections between the two representations.

In CSI, students were prompted to explain connections among the visualization and the symbolic representation after interacting with visualizations. For instance, students interacted with a visualization of a chemical reaction and then were prompted to explain the connections between the coefficients of the balanced equation and the molecules within the visualization. Scaffolding visualizations by explicitly prompting students to explain connections among representations may have helped students develop the ability to articulate and distinguish links among the symbolic and molecular representations.

Limiting Reagents

Item 7 was an adapted Chemical Concept Inventory question that asked students to draw the contents of a closed container after a certain reaction occurred (Mulford and Robinson 2002). In general, students from both Pilot and CSI groups progressed from giving irrelevant, non-normative ideas to responses with normative ideas. The comparison group students stayed at the irrelevant, non-normative level.

Specifically, Pilot group pretest scores started on average close to no link, indicating very low understanding of limiting reagents. Pilot students made progress toward partial links on the posttest [Pretest M(SD) = 0.67(0.91); Posttest M(SD) = 1.29(0.97)]. The CSI group started near the posttest levels of the Pilot group and made more progress toward simple links on the posttest [Pretest M(SD) = 1.38(0.35); Posttest M(SD) = 2.25(1.68)]. The comparison group started at the no-link level and made little progress [Pretest M(SD) = 1.08(0.64); Posttest M(SD) = 1.24(0.66)].

Results suggest that using KI support with visualization may have helped students understand and connect ideas about limiting reagents on symbolic and molecular levels. For example, on the pretest, many students drew molecular representations of some part of the balanced equation, failing to integrate concepts of conservation of mass, limiting reagents, or the dynamic nature of reactions. On the posttest, more students made connections to these concepts. In CSI, students interpreted visualizations that started with different numbers of molecular reactants and then manipulated the molecules to create as many product molecules as possible. Prompts in WISE guided students to explain how and what they created in the visualization related to the balanced equation. These KI-supported visualizations may have helped students form an integrated understanding of limiting reagents on a molecular and symbolic level.

Heat and Molecular Motion

On Item 8, students in the Pilot group scored close to irrelevant/blank on the pretest, progressing to over no-link levels on the posttest [Pretest M(SD) = 0.33(0.66); Posttest M(SD) = 1.33(0.73)]. The CSI group started near posttest levels of the Pilot group and made slight progress toward partial links on the posttest [Pretest M(SD) = 1.5(0.89); Posttest M(SD) = 1.83(1.00)]. Comparison groups started at the no-link level and made little progress toward partial links on the posttest [Pretest M(SD) = 1.04(0.68); Posttest M(SD) = 1.28(0.74)].

Knowledge integration (KI) guidance to interpret the visualizations may have helped Pilot students integrate ideas about heat and molecular motion. CSI guided students to manipulate heat and observe effects on a chemical reaction with Molecular Workbench visualizations. Students were then prompted to explain how heat relates to molecular motion. Having students not only interact with the visualization but also sort out ideas about heat and molecular motion may have contributed to learning gains. On the pretest, many students made connections to other knowledge not relevant to the question. On the posttest, more pilot students identified ideas such as increasing molecular motion or the interactive nature of chemical reactions forming and breaking bonds. Although the CSI group started with more knowledge and made less progress than the Pilot group, they appeared to need to sort out their prior knowledge and benefitted from prompts to note how heat connects to molecular motion. These findings guided the revisions to the visualizations and surrounding instruction.

Critiquing Balancing Equations

A critique item required students to critique a fictional student’s balanced equation and explain whether or not the student’s response was correct. On the critique item, the Pilot group scored close to no link on the pretest, progressing to over no-link levels on the posttest [Pretest M(SD) = 0.71(0.85); Posttest M(SD) = 1.10(0.83)]. The CSI group started between partial and simple links on the pretest and made a little progress toward simple links on the posttest [Pretest M(SD) = 2.46(1.25); Posttest M(SD) = 2.71(1.68)]. The comparison group started near partial levels and stayed at similar levels on the posttest [Pretest M(SD) = 1.88(1.13); Posttest M(SD) = 1.96(1.21)]. Pilot group students had not covered balancing equations in class, had very low initial pretest scores, and made substantial progress on the critique item. CSI students started at a level that was higher than the Pilot group posttest and were able to make slight progress in articulating their understanding of balanced equations.

Summary

In summary, the Pilot students started with much lower scores than the CSI and comparison group, consistent with their lack of classroom instruction on these topics. Both Pilot and CSI groups made substantial progress on the KI items associated with connecting representations and limiting reagents. Thus, CSI can increase integration of chemistry ideas for students across a wide range of pretest levels. Specifically, the CSI group had pretest scores that were equal to or higher than the posttest scores for the Pilot group on all but one item. In all cases, the pretest scores for the CSI group were much higher than the pretest scores for the Pilot group. This finding suggests that the CSI unit can benefit students both before classroom instruction on these topics and after the topics have been introduced.

Prior to classroom coverage but not subsequent to classroom coverage, CSI impacted the typical balancing equations item as well as the heat and molecular motion item. Both prior to and subsequent to classroom coverage, students made progress on the KI items concerning links between symbolic and molecular representations and concerning limiting reagents. Thus, KI instruction is robust for student prior knowledge in these areas and can add value to classroom instruction at multiple points in the course.

Study 3: Course-Long Study

The results of both study 1 and 2 suggest the value of CSI for promoting coherent understanding of chemical reactions whether the unit occurs early in the course or after some classroom instruction. To assess the course-long impact of CSI, we conducted another study with students similar to those in study 2 and added an end-of-course assessment in addition to pretest and posttest assessments. We investigated how these students preformed on the end-of-course assessment. Without further intervention, we expected that performance would decline between the posttest and the end-of-course assessment. We also compared end-of-course performance of CSI students to students who studied a textbook-based curriculum.

Course-Long Study Methods

The CSI course-long participants (n = 53) included high school classes taught by one teacher who used the same version of CSI, pretest, and posttest used in study 2. Students took an end-of-course test two months after the posttest. Students came from a similarly diverse school population as study 2. A total 71 % of the school population is classified as socioeconomically disadvantaged, with 75 % of students from underrepresented populations in science (58 % Hispanic or Latino, 11 % black, and 6 % Filipino).

The typical comparison group students (n = 502) came from five high schools and eight teachers similar to the CSI school in demographics and teacher characteristics. Five of the teachers came from schools in the same district as the CSI school. All of the typical course-long teachers taught standards-based chemistry classes and administered the end-of-course test at a similar time.

Item 8 was included in pretests, posttests, and the end-of-course test. All assessments were scored with the same KI rubric used in study 2. Paired t tests were used to investigate pre-post differences for CSI students. Wilcoxon–Mann–Whitney tests were used to investigate differences from posttest to end-of-year tests and to compare CSI to typical course-long students. Missing data for three students was not included for the end-of-year comparisons. The pretest and posttest scores of the dropped students fell within a standard deviation of the average class score.

Course-Long Study Results and Discussion

CSI Course-Long Study Results

Students using CSI made significant gains from pretest to posttest, replicating results from studies 1 and 2 (Pretest M(SD) = 2.11(.73); Posttest = 2.52(.57); d = 0.63; t = 4.75, p < 0.01). Pretest to posttest gains of d = .63 are comparable to d = .53 for study 2 and lower than the d = .99 for study 1. The CSI course-long sample group [Pretest M(SD) = 1.56(1.23); Posttest = 1.94(.89)] performed similarly to the CSI group [Pretest M(SD) = 1.50(0.89); Posttest = 1.83(1.00)] on Item 8. Thus, the impact of CSI for this group is similar to previous results.

End-of-Course Results

Chemistry scene investigators (CSI) students significantly increased from the posttest to the delayed test (Fig. 3; Posttest M(SD) = 1.94(.89); Delayed = 2.58(.84); z = 3.76, p < 0.01; d = .74) on Item 8. This is impressive since often students forget instructed material from posttest to delayed test (Bjork 1994). On average, students were between no link and partial understanding on the pretest, and close to partial links on the posttest. On the end-of-course tests, students made progress toward a simple link between two scientifically relevant ideas. Thus, rather than forgetting, students were able to link more ideas on the end-of-course test. These gains are consistent with the idea that CSI provides a firm foundation for subsequent instruction. After learning about chemical reactions in the unit, students went on to study stoichiometry, equilibrium, and acid–base reactions with typical instruction. Since future topics related to chemical reactions, results suggest that students were able to add connections to their understanding and integrate more ideas.

Fig. 3
figure 3

Average KI score and standard error bars for pretests, posttests, and end-of-year tests for CSI course-long students and end-of-year tests for typical course-long students in study 3

Full size image

Interviews with the teacher pointed to the importance of visualizations for students’ understanding of subsequent concepts. The teacher reported that visualizing the molecular level was particularly powerful for students who had limited exposure or understanding of chemical reactions. The teacher reported that the visualizations and surrounding instruction helped students make connections among symbolic, molecular, and macroscopic levels. The teacher also reported referring back to the molecular visualizations when talking about subsequent concepts such as chemical equilibrium. The visualizations allowed the teacher and students to build on prior knowledge using a common reference point or model.

Course-Long Comparison Results

Typical course-long students resemble the comparison group from study 2. Typical students had an average KI score consistent with the average posttest score from the comparison group in study 2 [M(SD) = 2.21(1.1)].

Comparing Course-Long Students

Chemistry scene investigators (CSI) students outperformed typical course-long students on the end-of-course item (z = 2.73, p < .01; Fig. 3). These results are informative but not conclusive since the schools were different. The results for the end-of-course assessment suggest that CSI, compared to typical instruction, helped students form durable, connected knowledge. Students in the typical course, on average, were not as successful as those in the CSI-enhanced course. These findings are consistent with other comparisons between WISE and typical instruction (Linn et al. 2006).

Discussion and Implications

This study investigated how a visualization-enhanced unit designed following KI meta-principles and instructional patterns could help students form integrated understandings of chemical reactions. Results indicate that students using CSI made connections among representations and concepts such as conservation of mass and limiting reagents. Students taking the same pretests and posttests in the comparison group did not make similar progress. These findings strengthen other findings that visualization-enhanced instruction helps students integrate representations in chemistry (Ardac and Akaygun 2004; Yang et al. 2004; Wu et al. 2001). Studies not showing gains from dynamic visualizations may fail to support the process of KI, primarily delivered through activities, embedded explanation prompts, and opportunities to reflect in the supporting instruction. Results suggest that visualizations with KI support can contribute to understanding connections among symbolic, molecular, and observable representations.

Students with a range of prior knowledge were able to make significant improvement in numbers of relevant ideas and connections. Results replicated across the three studies. Students using CSI with widely different pretest scores and from different schools all gained on the representation and limiting reagent items. Overall, these findings suggest that design following the KI pattern combined with powerful visualizations enables students across the performance spectrum to make progress.

Students’ scores from the course-long study not only increased from pretests to posttests, but also significantly increased from posttest to end-of-course test months after the implementation of the unit. These findings suggest that the CSI unit facilitated the incorporation of subsequent instruction after the unit ended.

Performance of the comparison groups in study 2 and 3 suggest that typical instruction is less effective for achieving the goals of CSI, compared to using the unit. This is especially true for students who already have some understanding of the topics in CSI such as those in study 2. For students with minimal prior knowledge, CSI was very effective (study 1), but students using typical textbook and lecture instruction largely caught up with this group by the second semester. However, for students in study 2, CSI added insights and promoted KI beyond what they had achieved in typical instruction.

In addition, students in the course-long study who made similar gains to those in study 2 outperformed a control group on the end-of-course assessment. This supports the idea that CSI has a lasting impact, compared to typical instruction, months after implementation.

Knowledge Integration and Chemistry Instruction

The studies suggest that the KI framework may be particularly beneficial to chemistry instruction as it provides specific guidance to help students take advantage of molecular visualizations and make connections among ideas. The curriculum here builds upon successful efforts such as Connected Chemistry (Levy and Wilensky 2009) and 4M:Chem (Kozma 2003) that specifically focus on students’ integrating levels in chemistry through the use of visualizations. Combining KI with dynamic visualizations offers specific guidance and assessment frameworks to help develop and measure student understanding of chemistry. CSI used the KI pattern to encourage students to elicit ideas, add ideas, distinguish ideas, and reflect. These scaffolds appear to help learners to build on the ideas in the visualization.

Student Use of Molecular Visualizations

These findings suggest the need for further exploration of how the design of visualizations and supporting instruction impacts student learning. Although students were able to use KI supports and prompts to make relevant connections between representations, students still had more progress to make. This points to the difficulty of making these kinds of connections, even with instructional guidance. Next steps include the need for more precise investigations to clarify the specific instructional supports that promote KI.

Limitations

The results of these design studies reveal areas for unit revision and demonstrate the advantages of the CSI unit for student learning. Results may apply to similar populations that include a large proportion of economically disadvantaged students as well as students from ethnic groups that are underrepresented in science.

We refer to this as design research because the results help validate and refine the design of the instruction. We control for differences in groups assigned to the comparison condition in study 2 by regressing posttest scores on pretest scores. In study 3, we validate the progress of the CSI class across the course and use a convenient comparison group from a set of schools that are comparable to the school implementing the CSI instruction. Thus, the comparison gives an indication of progress of comparable students but is not a randomized comparison.

Conclusion

This research describes how the design decisions following KI principles and practices lead to benefits from a visualization-enhanced chemistry curriculum unit. Contrary to many studies showing that dynamic visualizations may not benefit learning (Betrancourt 2005; Hegarty 2004; Lowe 2004; Mayer et al. 2005), the findings in this study demonstrate ways to design combinations of visualizations and instruction that improve understanding. The KI principles informed design of guidance that helped students connect ideas and representations of chemical reactions. Our findings demonstrate that dynamic visualizations embedded within tested instructional patterns resulted in increased connections among students’ ideas about chemical reactions at the beginning of chemistry and after typical instruction on chemical reactions. This instruction formed a foundation that led to further increases in performance on the end-of-course assessment. Specifically, CSI helped students form connections among the molecular levels depicted in the visualizations and the symbolic levels of chemical reactions. The end-of-course assessment results suggest that the curriculum helped students form robust and durable networks of ideas that were strengthened by subsequent instruction.